Feature Type
Nice to have
Feature Description
I built an example agent that watches the participant's camera while the agent is talking and adapts to what it sees. If the listener looks lost, it re-explains the last point more simply once the current sentence finishes. If their attention drifts, it wraps up and hands the turn back. If they start to say something and stop themselves, it yields the floor. It never comments on the listener's face, it just adjusts, the way a person presenting to you would.
The reason I think this belongs in the framework: the default video sampler attends at 1 fps while the user speaks and 0.3 fps otherwise. So the system pays the least visual attention at the moment the listener's face carries the most information, which is while the agent is talking. This example flips that, perception peaks while the agent holds the floor.
How it works, in three files:
reaction_awareness.py (Tier 1): an independent rtc.VideoStream tap feeds MediaPipe Face Landmarker on CPU. Signals are deltas against a per-user neutral baseline, and a state machine with debounce, hysteresis, and refractory windows emits an occasional cue: confused, disengaged, wants_to_speak, engaged.- Tier 2: a fired cue, one keyframe, and the last agent utterance go to a VLM for a conservative read or a veto.
reaction_policy.py: decides what to do about it. A clarifying reply queued for when the current speech ends, a hard yield via session.interrupt() on a high-confidence wants_to_speak, or a silent system note on the next user turn. Adaptation is budgeted at 3 per minute with a 10 second minimum gap so the agent never thrashes.
The perception and policy modules are framework-independent and covered by 22 offline tests (synthetic frames, no camera or keys needed). Only new dependency is mediapipe.
Is there interest in this as an example? Happy to restructure, rename, trim scope, or swap the Tier 2 default to whatever the team prefers. Code and tests are ready to PR.
Workarounds / Alternatives
You can approximate this today by letting the video sampler feed frames into the LLM context, but that samples mostly while the user is talking, and the model only sees the frames on the next turn. It cannot notice a reaction mid-speech, finish the sentence, and pivot. I did not find an existing example that adapts while the agent holds the floor, which is why I am proposing one.
Additional Context
What testing this on my own face taught me, because the limits are the interesting part: my confusion produces a brow-down delta of about 0.007, which Tier 1 simply cannot see, and my skepticism shows up as a smirk, which naively scores as a smile. Tuning helped (squint turned out to carry the confusion signal for me, and wants_to_speak works much better as an aborted speech onset than as an open mouth), but the honest conclusion is that the heuristics need per-user calibration and the subtle cases need the second tier with a decent vision model behind it. I also ran the agent end to end as a real voice agent on a fully local stack (silero VAD, local whisper STT, piper TTS, ollama LLM and VLM), and both adaptation paths fired in a live conversation: a soft re-explain on confused and a yield on wants_to_speak.
On scope: these are engagement cues, not emotion detection, and the README says so with the Barrett et al. 2019 citation. The example adapts pacing and depth only, discloses that it pays attention to how explanations land, keeps frames in memory without storing anything, and explicitly scopes out hiring, proctoring, workplace monitoring, and education assessment (EU AI Act Art. 5(1)(f)).
Feature Type
Nice to have
Feature Description
I built an example agent that watches the participant's camera while the agent is talking and adapts to what it sees. If the listener looks lost, it re-explains the last point more simply once the current sentence finishes. If their attention drifts, it wraps up and hands the turn back. If they start to say something and stop themselves, it yields the floor. It never comments on the listener's face, it just adjusts, the way a person presenting to you would.
The reason I think this belongs in the framework: the default video sampler attends at 1 fps while the user speaks and 0.3 fps otherwise. So the system pays the least visual attention at the moment the listener's face carries the most information, which is while the agent is talking. This example flips that, perception peaks while the agent holds the floor.
How it works, in three files:
reaction_awareness.py(Tier 1): an independentrtc.VideoStreamtap feeds MediaPipe Face Landmarker on CPU. Signals are deltas against a per-user neutral baseline, and a state machine with debounce, hysteresis, and refractory windows emits an occasional cue: confused, disengaged, wants_to_speak, engaged.reaction_policy.py: decides what to do about it. A clarifying reply queued for when the current speech ends, a hard yield viasession.interrupt()on a high-confidence wants_to_speak, or a silent system note on the next user turn. Adaptation is budgeted at 3 per minute with a 10 second minimum gap so the agent never thrashes.The perception and policy modules are framework-independent and covered by 22 offline tests (synthetic frames, no camera or keys needed). Only new dependency is mediapipe.
Is there interest in this as an example? Happy to restructure, rename, trim scope, or swap the Tier 2 default to whatever the team prefers. Code and tests are ready to PR.
Workarounds / Alternatives
You can approximate this today by letting the video sampler feed frames into the LLM context, but that samples mostly while the user is talking, and the model only sees the frames on the next turn. It cannot notice a reaction mid-speech, finish the sentence, and pivot. I did not find an existing example that adapts while the agent holds the floor, which is why I am proposing one.
Additional Context
What testing this on my own face taught me, because the limits are the interesting part: my confusion produces a brow-down delta of about 0.007, which Tier 1 simply cannot see, and my skepticism shows up as a smirk, which naively scores as a smile. Tuning helped (squint turned out to carry the confusion signal for me, and wants_to_speak works much better as an aborted speech onset than as an open mouth), but the honest conclusion is that the heuristics need per-user calibration and the subtle cases need the second tier with a decent vision model behind it. I also ran the agent end to end as a real voice agent on a fully local stack (silero VAD, local whisper STT, piper TTS, ollama LLM and VLM), and both adaptation paths fired in a live conversation: a soft re-explain on confused and a yield on wants_to_speak.
On scope: these are engagement cues, not emotion detection, and the README says so with the Barrett et al. 2019 citation. The example adapts pacing and depth only, discloses that it pays attention to how explanations land, keeps frames in memory without storing anything, and explicitly scopes out hiring, proctoring, workplace monitoring, and education assessment (EU AI Act Art. 5(1)(f)).