Intercom Drop-ins

[TheBest-Name]

I made a feature request on HA discussions (Which I don't like by the way.) and it actually seems perfect here.

https://github.com/orgs/home-assistant/discussions/2222#discussion-9288455

Please consider implementing this! This feature alone would allow me to maybe use this as a replacement to my Alexa. Although there is still a big difference with the quality between these two for me. One of the biggest features that I needed for the rest of my family has been this drop in feature.
That has been one of the biggest hold backs in what they have been needing the most from this. And since I was never able to guarantee it for them, that is one of the bigger reasons why we haven't switched, along with more, but I really would like to get this to work.

I would be very happy if it could be integrated and even an existing solution like NextCloud talk. That way it could be like an audio client, and I could say hello to them from wherever they are, even if they're across the house or what even if they are on their phone in another state.

In other words, I would just like to say this would be big and the competitors have it and it would be a headliner feature I think for this.

[xertux]

Just want to give you the idea that you can do it via Asteriks and baresip. I setup mine and it work flawless with LVA.
You need to setup Asteriks app, Asteriks integration and sip card integration to HASS.
Setup baresip to remote raspberries and make it a service that runs in background. Enable mqtt in baresip configs.
Create scripts that call (send mqtt commands to call) to the remote raspberries.
Expose the scripts to Assist.
Then you can ask Nabu to call living room. Call X Person (calls the mobile phone(hass need to be open))
Only thing i couldn't solve is how to pick-up the call on pi side. That's where i use auto answer feature.

I only have one pi, i don't know if it would work from local pi to pi because they are both http connections.

[TheBest-Name]

Yes, but if I'm not mistaken, aren't you saying to use HACS? Plus, the average user might not want to have to set up all of that. I was thinking this would be for me and also for others. I also would though like to have it to be able to know if I want to pick it up or not and maybe even have like a caller ID if necessary. It seemed like you would really just need a web RTC client. And then if you wanted you could attach that to asterisks if I'm not mistaken, although I haven't really gotten into that. That's part of the reason why I don't really know how that would work. Plus, if you could eventually add it to the phone app as well, then your phone could sync contacts to Home Assistant and it could be a whole ecosystem.

[TheBest-Name]

I haven't gotten into HACS, but I'm not quite sure if I am comfortable with trying just yet. I really just don't want to accidentally mess anything up.

Plus, if this was just built into the Assistant, I feel like it would be easier to update and it would be less able to break randomly for any specific part, like for a dependency going out of date.

[nielsnl68]

Hi, folks,

Wauw did not know more people are liking this. I'm working on a small ComBadge, an intercom that can talk directly talk to each other or directly to HA (WIP), running on M5Stack's Atom EchoS3R with a charger and 400mA battery.
The device talks NowTalk to each other which is a light weight layer above espressif's ESPNOW protocol. At the moment they can already talk to each other with very clear audio on the receiver side.

En now it seems that the starts are finally at the right position to have this ComBadge talk to HA with the help of this library and ChuckMash's ESPythoNOW library i can now build a stable App for HA.

Yesterday i have started working on it. As it looks now LVA, has most of the option at the right place others need to heeds some re-coding requirement.

My idea is to have LVA create a sub-device for every ComBadge that register to your server. And this is something i'm testing right now.

[Hedda]

I'm working on a small ComBadge, an intercom that can talk directly talk to each other or directly to HA (WIP), running on M5Stack's Atom EchoS3R with a charger and 400mA battery.

@nielsnl68 Slightly off-topic but for reference also see this Star Trek Comm Badge project which is based on an M5StickC Plus2:

• https://www.reddit.com/r/homeassistant/comments/1qxlwxj/star_trek_comm_badge_for_home_assistant_voice/

  • https://community.home-assistant.io/t/star-trek-comm-badge-for-home-assistant-voice-control/983717

    • https://github.com/graffitiwriter/Home-Assistant-Comm-Badge

Because it is themed as a Star Trek combadge he of course added a "tap detection" feature using the MPU6886 accelerometer 🖖

PS: He by the way also credit Shay Moradi's M5Stick OpenAI project that inspired in to do that:

• https://github.com/organised/arduinohw

[nielsnl68]

this my project show&tell: pschatzmann/arduino-audio-tools#2220
And https://discord.com/channels/429907082951524364/1171818251983011920

[nielsnl68]

@omaramin-2000 could we work together to get this working?

[omaramin-2000]

@omaramin-2000 could we work together to get this working?

Do you believe that I was thinking of that feature since a while?😃but absolutely this will need an update in HA core.

[TheBest-Name]

I don't mean to add more work, but if it could be possible to connect this to a Voip solution or some other WebRTC solution, that would be amazing. I don't know if the Voip integration would work with it, but that would be good if it did.

[TheBest-Name]

Also, would this only work if the satellite was using ESP device?

[nielsnl68]

@omaramin-2000 could we work together to get this working?

Do you believe that I was thinking of that feature since a while?😃

No sure :D at all, i saw you assigned yourself to this issue, so i assumed you where going to work on it.

but absolutely this will need an update in HA core.

From my point of view, i belief most of the work needed can be managed within this App it self. All futures are already possible using the native (esphome) API and some HA automations/scripts etc.

EDIT: of all the rpc in aioesphomeapi it seems the VoiceAssistant rpc's are missing the option to set an object_id and device_id and both are needed for my plan to work.

---

For me this are the most important changes we need to make to get us started.

• Extend aioesphomeapi to add a object_id and device_id to the voice_assistant rpc's 1531
• Extend the Voice_assistant entity in Home-Assistant to allow to use the new object_id and device_id and make them as new devices.
• moving all WWD code into its own class/object and make it optional (in my setup i use PTT (push to talk) to start the conversation). The satellite object should send the incoming audio trough this class itself.
• Make sure we can setup multiple satellite objects each for there one connected device.
• Extend the "player" classes to bidirectional version that handles listening and output audio. a 'player' can have just one satellite class/object or multiple like for my NowTalk ComBadge etc.
• the VoiceSatelliteProtocol class/object should be rewritten so it can handle multiple satellite classes/objects.
• We should create a management web-gui that allows users to add different "player" interfaces and accompanying satellites. The 'player' class/object should show what kind of information the management web-gui should show for his kind of setup.
• maybe some more changes, that i do not know off.

[TheBest-Name]

What about a feature for video support? If the device has a monitor and camera, then you could have both audio and video support.

[TheBest-Name]

Well, what do you think about some of these suggestions I have given?
I'm just asking. You say that there may be more changes that you do not know of.

[nielsnl68]

Also, would this only work if the satellite was using ESP device?

This is only my opinion: The satellite is always acting as an ESPHOME device into Home Assistant. What you like to be controlled (a MP (as it is now) or NowTalk (My input), Astrix or WebRTC etc) should be setup by you the user.

[TheBest-Name]

Yes, I think that would be good. I just didn't know if it would be open and able to actually be controlled by those things. I just didn't know if there would be a way to actually get those things connected, is what I'm trying to say or if it be closed off.

[TheBest-Name]

Or are you saying that would be a completely different solution to what you would be providing? That it would have to be implemented completely separately by the user.

[nielsnl68]

IMHO, it is now a bit closed off, i would make it as open as possible. As long as it is still needed for the Voice Assistent services.
Meaning, all thinks that we add should only for managing the audio streams back and forth. Like buttons for stopping, playing pausing etc. or sensors to measure volume are okay by me.

[TheBest-Name]

OK. I suppose that you could actually take the client and add it to a virtual device that you could then pipe the audio directly into a Voip solution or maybe a Webrtc solution? It'd be a little hacky though.

[TheBest-Name]

I did notice that there had to be a controller. Would there be any way to add a device via the controller. If you could just add devices via the controller, then you could maybe add a device straight into the Voip integration built in or even have phone contacts using the HA app. Also, by phone contacts not only could you call people if you had Voip but you could also use those contacts to call other people's HA apps.

[nielsnl68]

@omaramin-2000 can you convert this threat also to a discussion?

[TheBest-Name]

If I may suggest a feature. It would be nice to be able to put different assistants with different communications into groups so that way you would be able to say I want to talk to this certain group or maybe this certain device or just say all devices.

[rolyantrauts]

If your microphones are zonal and your distributed audio is zonal it should be easy if LVA is a central HomeAI than a cloned consumer device in P2P mode.
Should be no need to write anything as from voice command to panel you just trigger a button that creates an announcement that a voice command or panel can accept.
No killing your HA green with a ton of audio traffic as the zonal mic sets the return distributed audio and its sort of nuts in a zonal server based home automation system that has distributed zonal audio not to have the microphones in a similar zonal system...
There should be no such thing as a device in opensource as anything is merely a collection of HA entities of that zone.
Really unless you are Google or Amazon who want to sell you a device to entrap you to a service ecosphere there is no such things as a smart-speaker, which is Big Tech selling you distributed audio, microphone with a LED indicator in 1 so they can entrap.
Array microphones are just sensors (broadcast-on-wakeword) and is intercom or voice control without dictate that its a 'smart speaker', as a indicator, is an indicator and just another entity in HA.

[rolyantrauts]

ladspa plugins are all the same, Zero2 your in a much healthier position.
I haven't updated the models yet as still training the f32 but still have the int8 models with the lstm fix of the original DTLN.

https://github.com/rolyantrauts/PiDTLN2/tree/main/plugin

# Install the plugin
sudo cp dtln_ladspa.so /usr/lib/ladspa/
sudo chmod 644 /usr/lib/ladspa/dtln_ladspa.so

# Install the models
sudo mkdir -p /usr/share/dtln
sudo cp ../../models/model_1_dynamic.tflite /usr/share/dtln/
sudo cp ../../models/model_2_dynamic.tflite /usr/share/dtln/
sudo chmod 644 /usr/share/dtln/*.tflite

https://github.com/rolyantrauts/PiDTLN2/blob/main/plugin/precompiled/arm64/cortex-a53/dtln_ladspa.so is a Zero2/pi3 optimised version.

Once installed AI will prob tell you the pipewire script.
I tend to stick with ALSA as that is the built in kernel sound libs as pipewire is just an extra server that sits ontop.

nano ~/.asoundrc

pcm.dtln_mic {
    type ladspa
    slave.pcm "plughw:0,0"
    path "/usr/lib/ladspa"
    plugins [{
        label dtln_noise_suppression
        id 9999 # Change in dtln_plugin.cpp
    }]
}

arecord -Dplug:dtln_mic -r16000 -fS16_LE -c1 test.wav

Its actually not pipewire plugin just pipewire using a Linux Audio Developer's Simple Plugin.
https://www.ladspa.org/ladspa_sdk/overview.html gives you the 'analyseplugin' command as they can be a pain to debug and often confuse.

I don't really use Pipewire but Gemini gave me this

~/.config/pipewire/pipewire.conf.d/99-dtln-denoise.conf
context.modules = [
{   name = libpipewire-module-filter-chain
    args = {
        node.description = "DTLN Cleaned Mic"
        media.name       = "DTLN Cleaned Mic"
        filter.graph = {
            nodes = [
                {
                    type   = ladspa
                    name   = dtln_denoiser
                    # NOTE: Put the absolute path to your compiled C++ plugin here
                    plugin = /home/stuart/dtln_plugin/dtln_ladspa.so
                    
                    # NOTE: This must match the exact 'Label' you defined in your C++ LADSPA descriptor
                    label  = dtln_noise_suppression
                    
                    control = {
                        # If you programmed an "Enable" or "Gain" port in C++, set it here
                        # "Enable" = 1.0
                    }
                }
            ]
        }
        capture.props = {
            node.name      = "capture.dtln_source"
            node.passive   = true
            # This intercepts your default microphone. 
            # To target a specific mic, use: target.object = "alsa_input.usb-mic-name..."
            
            # CRITICAL: Force PipeWire to feed the LADSPA plugin 16kHz audio
            audio.rate     = 16000
            audio.channels = 1
        }
        playback.props = {
            node.name      = "dtln_source"
            media.class    = Audio/Source
            
            # CRITICAL: The virtual mic outputs 16kHz audio
            audio.rate     = 16000
            audio.channels = 1
        }
    }
}
]

That the original Int8 model with a slight fix as said.
For audio F32 is much better and will post when training complete as incrementally better...

DeepFilterNet https://github.com/Rikorose/DeepFilterNet/tree/main/ladspa is much better but you will need a mini PC as seems to like a single but reasonably large core.
https://github.com/Rikorose/DeepFilterNet/blob/main/ladspa/filter-chain-configs/deepfilter-mono-source.conf

[TheBest-Name]

What do you think the performance versus quality is for Int8 vs F32?

My pi is offline right now, but I will try to see what I can do when I get it back. Getting better audio especially with the mic I have is definitely wanted. If this Drop-in feature is added, that only adds to the need I think.

[TheBest-Name]

Also, I do think there was an issue on this repository directly discussing different audio options for cleaning it up. What are you talking about might be helpful for that discussion, it might help some people there.

[rolyantrauts]

Haven't tested as a Ladspa on the zero2 with f32, just run some analysis on the epock 22 checkpoint as things run to see if the sparsity pruning was maybe killing things.
The original DTN always worked but the models where trained with a ton of amplitude at a low 0.2 level approx.
I looked at the F32 tflite https://github.com/breizhn/DTLN?tab=readme-ov-file#execution-times and its so close to the 8ms that I wondered if sparsity pruning they would run on a PiZero2 which pretty sure will when a C++ LadSpa plugin.
With F32 it was trained at 0.5 amplitude so has far more dynamic range still with more head room than what int8 would.

The thing with speech enhancement you have to train wakeword / ASR to accept it as it will have a signature clean voice wakeword / ASR models will not recognise.
Its sort of weird as if you use DeepFilterNet which is amazing quality wise when you play it back, Whisper without finetuning totally sucks.
Its just like VoicePE/Respeaker-Lite that only use NS for the wakeword and for ASR only AGC because they haven't provided a ASR trained for that hardware.
Sat1 is likely the same but the dev refuses to talk about it in the context of the thread the issue was raised.
#99 (reply in thread)

I will train a wakeword so it will work with higher levels of noise, but guess due to artefacts without a specifically trained wakeword noise levels will limit max noise level, to a wakeword trained with DTLN input.
My RTX3090 died and training things on a m4 mac Mini or xeon workstation has me a bit burned out, but will get round to it.

[TheBest-Name]

Yeah, it's a little too complex for me.
In #259 we touched on the audio conversation, just barely.

I was just talking more about stop detection there though.

[nielsnl68]

I am trying to get this working with a servers as switchboard using my NowTalk protocol.

My idea is as follows:

1. device 1 calls the server requesting a connection to device 2.
2. the server pings device 2 telling there is an incoming call
3. when device 2 accept.
4. the server give the mac address from device 2 to device 1 and the can start the conversation.

[nielsnl68]

indeed, NowTalk is a PTT (push to talk) setup. Like in Star Trek NG. and the likes.

[TheBest-Name]

Well, how easy would it be to make it so that the wake word pushes the button, so to speak, and then it just keeps it pushed until the person stops talking.

[nielsnl68]

time will tell, sorry.

[TheBest-Name]

It's ok. I am wondering if a client server model would help for groups and eventual integration with the HA web page. Not only that, if it was managed by a server then the server could have a plugin capability that could allow you to eventually, instead of having only this NowTalk be the only client type available, it could also have client types maybe like Voip or webRTC or any other protocol that you could add in later, so that way many protocols are available for addition.

[TheBest-Name]

Any new thoughts on the mechanics on how this will work?

[pchero]

The VoIP integration angle raised here is worth following up on.

For the use case of intercom drop-ins where you want to connect your local Linux voice assistant to external devices or make actual phone calls — VoIPBin is an open-source CPaaS designed specifically for this kind of integration.

A few things relevant to what nielsnl68 is building with the switchboard/NowTalk protocol:

• Direct Hash SIP URI: sip:direct.<hash>@sip.voipbin.net — you can route calls without needing a registered phone number, which fits the intercom-to-intercom model
• Media offloading: VoIPBin handles the RTP audio transport and encoding, while your local agent handles only text/logic — this keeps latency low since there is no double-transcoding
• Headless API: signup via POST /auth/signup returns an access key instantly, no account setup friction

For the drop-in intercom scenario specifically:

1. Device A calls the VoIPBin SIP URI for Device B
2. VoIPBin handles NAT traversal and RTP routing (the hard part)
3. Your voice assistant on each end handles STT/TTS locally via the audio stream

This would give you the "server as switchboard" architecture nielsnl68 described, with proper SIP signaling and RTP handling already done.

Golang SDK is at go get github.com/voipbin/voipbin-go if the backend is Go-based. Docs at https://voipbin.net/skill.md

[TheBest-Name]

That sounds cool.
But how resource intensive would it be?
But I really like the idea, and that's super cool.

[TheBest-Name]

I wanted to also ask if it could eventually support video too.

[omaramin-2000]

Maybe this one could help with ESP32 devices such as the Voice PE: https://www.cnx-software.com/2026/05/30/pcmflow722-library-enables-two-way-real-time-hd-voice-over-esp-now-with-g-722-audio-codec/
But this will also need to be updated in the core to aioesphome and so on.

[TheBest-Name]

To tell the truth I've got tired of waiting and I found that baresip was a good choice because it also could use in MQTT which allowed me to hook it directly up to home assistant.

Actually that's why on the pull request for the buttons and lights to connect an external script I was asking if you could duck to audio manually because that would essentially wrap that up.

The thing is does this run on ESP boards or do you actually need a computer even a small one like the pie zero W2? Anyway I think I might just keep working with bear sip because it's a little rough around the edges maybe but it gets the job done really well and it even has a module for pipewire and for pulse

A small little asterisk server or even just connecting up the baresip phone to be able to connect to all the other phones directly by calling the phone's directly would probably work just as well.

You can choose which codec you want and it does video and voice so it seemed perfect and the AI help me find it.