Fix missing ifdefs in voice assistant

[jesserockz]

What does this implement/fix?

Missed some ifdefs when not using speaker

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Other

Related issue or feature (if applicable): fixes

Pull request in esphome-docs with documentation (if applicable): esphome/esphome-docs#

Test Environment

• ESP32
• ESP32 IDF
• ESP8266
• RP2040
• BK72xx
• RTL87xx

Example entry for config.yaml:

# Example config.yaml

Checklist:

• The code change is tested and works locally.
• Tests have been added to verify that the new code works (under tests/ folder).

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated in esphome-docs.