echo cancellation when running locally - the output audio is looping back into the input

[cgreening]

Running on a Mac using the hosted version. It seems to get very confused by its own voice coming out of the speaker and being picked up by the microphone.

Using headphones seems to work much better.

not-working.mp4

[codewithcheese]

This is an issue for me also, the mic is picking up the assistants voice, which is then being transcribed and replied to, resulting in awkward conversation with it self.

[remykarem]

Same here.

Is there a way we can temporarily pause the input stream when the output stream is being played?

[ajar98]

hey folks!! Indeed there is no echo cancellation enabled in system audio conversation, few things:

• This won't happen for turn-based conversation: https://github.com/vocodedev/vocode-python/blob/main/vocode/turn_based/turn_based_conversation.py
• I'd love to hear if folks have any ideas for fixing this (that don't involve pausing the input stream, since that precludes the user from interrupting the bot!). Other surfaces like web / phone do this out-of-the-box

[as14g18]

I think this is a hard problem. Amazon has a few articles on the topic. [1]

Two ideas:

• Have a "wake word". For example, Google's "Hey Google".
• Have the system learn your voice so it only responds to your voice

[1] https://www.amazon.science/tag/echo-cancellation

[rjheeta]

Headphones helped me with the choppy audio too.

Interested in how to handle simultaneous conversation. I want to interrupt the assistant as it's talking.

[kevcmk]

This is definitely a challenging implementation detail but I think it might be worth emphasizing this early on in the Quickstart documentation. It's not something that I could've intuitively troubleshot without coming to the Github Issues for the repo.

[ajar98]

good idea @kevcmk, just added this to the docs: https://docs.vocode.dev/python-quickstart#a-note-on-echo-cancellation

I'll leave this issue open as we brainstorm more options here.

[boorich]

Push to talk would be a gain already.

[ajar98]

in the new version, you can have the transcriber stop listening while the agent is speaking. here's a code snippet:

async def main():
    (
        microphone_input,
        speaker_output,
    ) = create_streaming_microphone_input_and_speaker_output(
        use_default_devices=False, use_blocking_speaker_output=False
    )

    conversation = StreamingConversation(
        output_device=speaker_output,
        transcriber=DeepgramTranscriber(
            DeepgramTranscriberConfig.from_input_device(
                microphone_input,
                endpointing_config=PunctuationEndpointingConfig(),
                mute_during_speech=True,
            )
        ),
        agent=ChatGPTAgent(
            ChatGPTAgentConfig(
                initial_message=BaseMessage(text="What up"),
                prompt_preamble="""The AI is having a pleasant conversation about life""",
            )
        ),
        synthesizer=AzureSynthesizer(
            AzureSynthesizerConfig.from_output_device(speaker_output)
        ),
        logger=logger,
    )
    await conversation.start()
    print("Conversation started, press Ctrl+C to end")
    signal.signal(signal.SIGINT, lambda _0, _1: conversation.terminate())
    while conversation.is_active():
        chunk = microphone_input.get_audio()
        if chunk:
            conversation.receive_audio(chunk)
        await asyncio.sleep(0)


if __name__ == "__main__":
    asyncio.run(main())

[ajar98]

another fix for this! If you install Krisp.AI[0] (which is already built into some meeting platforms like Discord), you can select the krisp virtual microphone and speaker on startup and feedback issue is solved[1]!

[0] https://krisp.ai/
[1] at reasonable levels of volume - if you blast it, the feedback won't work

[jul3x]

Fix proposed by @ajar98 here: #16 (comment) did not work for me so I also needed to change use_blocking_speaker_output flag from False to True and it started working correctly for my application.

[mananmani1]

guys ,can we connect vocode with sqlite to handle db data in realtime with fastapi?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Thank you for your contributions.