feat(apollo-react): add STT dictate and voice interaction buttons [JAR-9545]#571
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Dependency License Review
License distribution
Excluded packages
|
There was a problem hiding this comment.
Pull request overview
Ports the apollo-design-system voice UX into packages/apollo-react ap-chat by replacing the legacy push-to-talk audio controls with two independently gated voice affordances (STT dictate + always-on voice interaction) and wiring the corresponding service events, docs, and playground showcase.
Changes:
- Added STT dictate state/events to
AutopilotChatService(SpeechToTextToggle,SetSpeechToTextState) anddisabledFeatures.audioStreaming. - Updated chat input UI to (a) add an STT mic button, (b) swap send↔voice-interaction when input is empty, and (c) collapse input chrome while voice interaction is active.
- Added docs + i18n strings + a react-playground demo consumer to exercise both flows.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/apollo-react/src/material/components/ap-chat/service/ChatService.ts | Adds STT state management and publishes new STT-related events; adds default audioStreaming: true in disabled features. |
| packages/apollo-react/src/material/components/ap-chat/service/ChatModel.ts | Extends event enum and disabled-features type with STT + audioStreaming. |
| packages/apollo-react/src/material/components/ap-chat/locales/en.json | Adds English strings for “Dictate” and “Voice interaction”. |
| packages/apollo-react/src/material/components/ap-chat/components/input/chat-input.tsx | Collapses/hides input chrome and error while voice interaction is active; passes voice-active state down. |
| packages/apollo-react/src/material/components/ap-chat/components/input/chat-input-actions.tsx | Replaces legacy audio UI with STT mic + always-on voice button; adds STT auto-stop-on-submit. |
| packages/apollo-react/src/material/components/ap-chat/components/input/always-on-voice-button.tsx | Introduces the always-on voice interaction button using existing audio input/output hooks. |
| packages/apollo-react/src/material/components/ap-chat/components/common/action-button.tsx | Removes press/release union props tied to the deleted push-to-talk UI. |
| packages/apollo-react/src/material/components/ap-chat/components/audio/chat-audio.tsx | Deletes the legacy push-to-talk + speaker combo component. |
| packages/apollo-react/src/material/components/ap-chat/DOCS.md | Documents STT + voice interaction behaviors and consumer wiring patterns. |
| apps/react-playground/src/pages/components/ApChatShowcase.tsx | Adds toggles + fake STT/voice consumers so both flows can be exercised without external services. |
| // Show voice interaction button when audioStreaming is enabled and either | ||
| // voice is already active or the input is empty (no text to submit). | ||
| const showVoice = | ||
| disabledFeatures.audioStreaming === false && | ||
| (isVoiceInteractionActive || (disableSubmit && !waitingResponse)); | ||
|
|
There was a problem hiding this comment.
showVoice is derived from disableSubmit, but disableSubmit can be true even when the input is non-empty (e.g. attachments are still loading, skeletonLoader). In those cases the send button would incorrectly swap to the voice-interaction button even though the user has text/attachments to submit. Consider passing an explicit isInputEmpty/hasSubmittableContent prop from chat-input.tsx (based on message + attachments) and use that instead of disableSubmit for the send↔voice swap condition.
| const [isSpeechToTextActive, setIsSpeechToTextActive] = React.useState(false); | ||
| React.useEffect(() => { | ||
| if (!chatService) { | ||
| return; | ||
| } | ||
| return chatService.on(AutopilotChatEvent.SetSpeechToTextState, (isActive: boolean) => { | ||
| setIsSpeechToTextActive(isActive); | ||
| }); | ||
| }, [chatService]); |
There was a problem hiding this comment.
isSpeechToTextActive local state is always initialized to false and only updates after SetSpeechToTextState fires. If the input actions component mounts while STT is already active (e.g. the service instance persists across remounts), the mic button styling will be out of sync until the next event. Initialize state from chatService.isSpeechToTextActive (and/or set it once inside the effect before subscribing).
| React.useEffect(() => { | ||
| if (!chatService || !isActive) { | ||
| return; | ||
| } | ||
| return chatService.on( | ||
| AutopilotChatEvent.OutputStream, | ||
| (event: AutopilotChatOutputStreamEvent) => { | ||
| if (event.mediaChunks) { | ||
| for (const chunk of event.mediaChunks) { | ||
| if (chunk.mimeType.startsWith('audio/pcm;')) { | ||
| queueOutputAudio(chunk.mimeType, chunk.data, chunk.sequenceNumber); | ||
| } | ||
| } | ||
| } | ||
| if (event.interrupted) { | ||
| clearOutputAudioQueue(); | ||
| } | ||
| } | ||
| ); | ||
| }, [chatService, isActive, queueOutputAudio, clearOutputAudioQueue]); |
There was a problem hiding this comment.
The OutputStream subscription is only attached while isActive is true. Because isActive is set via React state inside handleAudioInputStart, any sendOutputStreamEvent(...) calls triggered synchronously by consumers in response to the activityStart InputStream event can occur before this effect re-runs, causing the first audio chunks to be missed. Prefer subscribing on mount (or whenever chatService exists) and gate playback via an isActiveRef/state check instead of conditionally subscribing.
| const { startAudioInput, stopAudioInput } = useAudioInput({ | ||
| handleAudioInputData, | ||
| handleAudioInputStart, | ||
| handleAudioInputEnd, | ||
| }); | ||
|
|
||
| const handleClick = useCallback(async () => { | ||
| if (disabled) { | ||
| return; | ||
| } | ||
| if (isActive) { | ||
| stopAudioInput(); | ||
| } else { | ||
| await startAudioInput(true); | ||
| } | ||
| }, [disabled, isActive, startAudioInput, stopAudioInput]); |
There was a problem hiding this comment.
AlwaysOnVoiceButton does not stop microphone capture or clear queued audio on unmount. If the chat closes or audioStreaming is disabled while voice mode is active, the component can unmount without calling stopAudioInput(), leaving media tracks / AudioContext running. Add a useEffect cleanup that stops audio input and clears output (and, if needed, notifies onActiveChange(false)), ensuring resources are released reliably.
| tooltip={_(msg({ id: 'autopilot-chat.input.actions.dictate', message: `Dictate` }))} | ||
| overrideColor={isSpeechToTextActive ? 'var(--color-background)' : undefined} | ||
| data-testid="autopilot-chat-stt-button" | ||
| ariaLabel={_(msg({ id: 'autopilot-chat.input.actions.dictate', message: `Dictate` }))} |
There was a problem hiding this comment.
The dictate mic button is a toggle, but it doesn’t expose its pressed/toggled state to assistive tech. Consider adding aria-pressed={isSpeechToTextActive} (and similarly for the voice interaction toggle if applicable) so screen readers can announce the current state, not just the tooltip/label.
| ariaLabel={_(msg({ id: 'autopilot-chat.input.actions.dictate', message: `Dictate` }))} | |
| ariaLabel={_(msg({ id: 'autopilot-chat.input.actions.dictate', message: `Dictate` }))} | |
| aria-pressed={isSpeechToTextActive} |
| const { startAudioInput, stopAudioInput } = useAudioInput({ | ||
| handleAudioInputData, | ||
| handleAudioInputStart, | ||
| handleAudioInputEnd, | ||
| }); |
There was a problem hiding this comment.
useAudioInput does not auto-clean up on component unmount, so if the chat (or this button) unmounts while voice mode is active, the microphone stream/AudioContext/worker can remain alive. Add a cleanup effect in this component that calls stopAudioInput() (and clears output audio) on unmount / when disabling voice mode, and guard against state updates after unmount if needed.
| preventHover={true} | ||
| disabled={disableSubmit} | ||
| onClick={handleSubmitWithSttStop} | ||
| data-testid="autopilot-chat-submit-button" | ||
| ariaLabel={ | ||
| waitingResponse | ||
| ? _(msg({ id: 'autopilot-chat.input.actions.stop', message: `Stop` })) | ||
| : _(msg({ id: 'autopilot-chat.input.actions.send', message: `Send` })) | ||
| } | ||
| /> |
There was a problem hiding this comment.
The STT auto-stop logic is only applied to the submit button’s onClick, but keyboard submit (Enter in chat-input.tsx) still calls handleSubmit() directly, so dictation can continue after the prompt clears and re-populate the input. Also, when waitingResponse is true this same button acts as “Stop response”, but the wrapper still toggles STT, which is unexpected. Consider moving the “stop STT when sending a request” logic into a single shared submit path (e.g., inside handleSubmit when it actually sends) and skipping it for the stop-response path.
…R-9545] Replaces the push-to-talk audio combo with two independently-gated voice affordances: - STT dictate button (mic icon, disabledFeatures.audio) — emits SpeechToTextToggle; consumers own the recognizer and stream transcripts back via setPrompt. - Always-on voice interaction button (waveform icon, disabledFeatures.audioStreaming) — replaces send when input is empty, streams mic audio via InputStream / plays model audio via OutputStream. Input chrome collapses while active. STT auto-stops on submit so the cleared input doesn't re-populate with the prior transcript. The now-dead onPress/onRelease plumbing in action-button is removed along with the deleted push-to-talk component. ApChatShowcase gets toggles and fake consumers that demonstrate both flows without needing a real recognizer or model. Ports UiPath/apollo-design-system#5094 to the open-source chat package. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes the Format check on PR 571. Apply `biome format --write` to the showcase additions so tabs/spaces match the repo style. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixes the Format check on PR 571. The previous format pass missed apollo-react — only react-playground was covered. Apply `biome format --write` to the two files touched in this port: always-on-voice-button.tsx and chat-input-actions.tsx. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Seven review comments rolled into one commit: - chat-input.tsx / chat-input-actions.tsx: move STT auto-stop into the shared `handleSubmit` so the Enter-key path gets the same auto-stop as the click path, and skip it entirely on the stop-response branch so hitting "Stop" doesn't also kill active dictation. - chat-input.tsx / chat-input-actions.tsx: new `isInputEmpty` prop drives the send↔voice-interaction swap. Fixes an incorrect swap during attachment-loading / skeleton, where `disableSubmit` was true even though the user had real content to submit. - chat-input-actions.tsx: seed `isSpeechToTextActive` from `chatService.isSpeechToTextActive` so a remount while STT is already active shows the active visual without waiting for the next event. - always-on-voice-button.tsx: subscribe to OutputStream on mount and gate playback via an `isActiveRef` so audio chunks that arrive synchronously from the consumer's `activityStart` handler are not dropped. Add an unmount cleanup that stops mic capture and clears the output queue so the MediaStream / AudioContext is not leaked when the chat closes mid-voice. - action-button.tsx + both toggles: thread an `ariaPressed` prop so screen readers announce the toggled state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dsuresh-ap
force-pushed
the
feature/stt-voice-interaction-JAR-9545
branch
from
f544097 to
ddb5dd4
Compare
| * @property audioStreaming - Whether the chat has the always-on voice interaction button | ||
| * (requires the consumer to handle InputStream/OutputStream audio events) |
There was a problem hiding this comment.
Same issue for audioStreaming in this docs excerpt: it’s part of AutopilotChatDisabledFeatures, so audioStreaming: true disables the voice interaction button. Please adjust the description so consumers don't invert the boolean when configuring features.
| * @property audioStreaming - Whether the chat has the always-on voice interaction button | |
| * (requires the consumer to handle InputStream/OutputStream audio events) | |
| * @property audioStreaming - Whether to disable the always-on voice interaction button | |
| * (the feature requires the consumer to handle InputStream/OutputStream audio events) |
| * @property newChat - Whether the chat has the new chat button | ||
| * @property settings - Whether the chat has the settings button | ||
| * @property audio - Whether the chat has realtime audio enabled | ||
| * @property audio - Whether the chat has the STT dictate (mic) button |
There was a problem hiding this comment.
This JSDoc reads as if audio means the feature is present/enabled, but AutopilotChatDisabledFeatures is interpreted inversely throughout the UI (!disabledFeatures.audio). Please adjust the wording to reflect that this flag disables the STT dictate button to avoid consumer confusion.
| * @property audio - Whether the chat has the STT dictate (mic) button | |
| * @property audio - Whether to disable the STT dictate (mic) button |
| * @property audioStreaming - Whether the chat has the always-on voice interaction button | ||
| * (requires the consumer to handle InputStream/OutputStream audio events) |
There was a problem hiding this comment.
Similar to audio, this doc says "Whether the chat has" the voice interaction button, but the property lives in AutopilotChatDisabledFeatures and is treated as a disable flag. Please update the wording to make it explicit that audioStreaming: true disables the voice interaction affordance.
| * @property audioStreaming - Whether the chat has the always-on voice interaction button | |
| * (requires the consumer to handle InputStream/OutputStream audio events) | |
| * @property audioStreaming - Whether to disable the always-on voice interaction button | |
| * (when enabled, it requires the consumer to handle InputStream/OutputStream audio events) |
| // Mirror the publish back via setSpeechToTextState so the mic button stays in sync | ||
| // (publishSpeechToTextToggle already did this, but a real consumer would do it | ||
| // here after its recognizer actually started/failed). | ||
| if (isActive) { |
There was a problem hiding this comment.
The comment says the showcase "Mirror[s] the publish back via setSpeechToTextState" so the mic button stays in sync, but the handler never calls service.setSpeechToTextState(...). Either call setSpeechToTextState after the fake recognizer actually starts/stops, or update the comment to match the current behavior.
| * @property newChat - Whether the chat has the new chat button | ||
| * @property settings - Whether the chat has the settings button | ||
| * @property audio - Whether the chat has realtime audio enabled | ||
| * @property audio - Whether the chat has the STT dictate (mic) button |
There was a problem hiding this comment.
In this AutopilotChatDisabledFeatures excerpt, the audio field description reads like an enable flag ("Whether the chat has...") but in practice audio: true disables the dictate button. Updating the wording here would make the docs match the API semantics.
| * @property audio - Whether the chat has the STT dictate (mic) button | |
| * @property audio - Whether to disable the STT dictate (mic) button |
…ming Addresses follow-up Copilot review comments: the `@property` JSDoc for `audio` and `audioStreaming` read as "Whether the chat has..." but the interface is `AutopilotChatDisabledFeatures` and the UI treats these inversely (`!disabledFeatures.audio`). Reword to "Whether to disable..." so consumers don't invert the boolean when configuring features. Also fix the stale showcase comment claiming the fake STT consumer mirrors the toggle via `setSpeechToTextState` — it doesn't; describe the actual behavior and the failure-revert pattern a real consumer follows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Ports apollo-design-system#5094 into the open-source
apollo-reactchat package. The old push-to-talkAutopilotChatAudiocombo (mic + speaker, gated bydisabledFeatures.audio) is replaced with two explicit, independently-gated voice affordances:disabledFeatures.audio) — UI state only. EmitsSpeechToTextToggle; consumer apps own the recognizer (e.g. Azure Speech SDK) and stream transcripts back viasetPrompt().disabledFeatures.audioStreaming) — replaces the send button when the input is empty. Streams mic audio viaInputStreamevents and plays model audio viaOutputStreamevents. Input chrome collapses while active (textarea, error, left actions hidden; border/shadow removed).Jira: JAR-9545
Changes
publishSpeechToTextToggle(),setSpeechToTextState(),isSpeechToTextActivegetter. NewaudioStreaming: truedefault.SpeechToTextToggle+SetSpeechToTextStateevents;audioStreaming?: booleanonAutopilotChatDisabledFeatures.useAudioInput/useAudioOutputhooks; exportsVoiceButtonContainerso the STT mic button can share its active-state styling.components/audio/chat-audio.tsx. Thechat-audio-input.ts/chat-audio-output.tshooks stay.onPress/onReleasediscriminated union removed fromaction-button.tsx— the deleted push-to-talk button was its only caller.autopilot-chat.input.actions.dictate,autopilot-chat.input.actions.voice-interaction) added toen.json.activityStart), so both flows are exercisable without real services.Why now
Aligns the open-source chat with the internal apollo-design-system behavior so consumers that adopt
@uipath/apollo-reactget the same voice affordances without needing to port this manually.Reviewer notes
recordingButton/publishRecordingButtonClick/isRecordingAPI — the upstream delete-half of the diff is a no-op here.t()helper for i18n, matching every other string inap-chat.pnpm i18n:extractis broken onmain(pre-existing duplicate idautopilot-chat.error.multiple-filesacrossattachements-provider.tsx:112anddropzone.tsx:70). I hand-added the two new English strings toen.jsonas a workaround — once the extract bug is fixed, translators pick the keys up in the normal cycle. Flagged as follow-up.Test plan
pnpm turbo run build --filter=@uipath/apollo-react --filter=react-playgroundsucceedstsc --noEmitclean in both apollo-react and react-playgroundbiome lintclean on all changed filespnpm dev:react-playground, Components → Chat:[voice] activityStart, canned assistant reply streams in; click stop → chrome returns, console logs[voice] activityEndFollow-ups (not part of this PR)
pnpm i18n:extract— resolve the duplicateautopilot-chat.error.multiple-filesmessage id, then regenerate catalogs so translators can localize the two new English strings.🤖 Generated with Claude Code