Skip to content

AlphaAvatar v0.6.5

Choose a tag to compare

View all tags
@github-actions github-actions released this 25 Jul 10:53
· 5 commits to main since this release
v0.6.5
d5aa346

AlphaAvatar v0.6.5 Release Notes

Highlights

AlphaAvatar v0.6.5 introduces a new shared audio perception, interaction routing, and isolated inference architecture.

This release extends the PerceptionRuntime foundation introduced in v0.6.4 from visual perception into the realtime voice pipeline. Audio is now published once, segmented into derived speech observations by the Interaction Router, and consumed independently by transcription, speaker recognition, Persona, and future audio processors.

v0.6.5 also replaces the shared AlphaAvatar workloads previously running through LiveKit’s serialized inference executor with an AlphaAvatar-owned runtime. VAD, speaker recognition, face analysis, and all AlphaAvatar VDB backends now run in dedicated persistent processes.

The result is a more modular full-duplex architecture where raw perception, derived perception, local inference, and interaction orchestration have clearly separated responsibilities.

What's New

  • Added the Interaction Router architecture:

    • Introduced InteractionRouterRuntime as the processing layer between raw perception streams and derived interaction streams.
    • Added RouterProcessorBase for independently managed perception processors.
    • Added AudioActivityProcessor to consume raw audio, run VAD, and publish accepted speech observations.
    • Added SpeechTranscriptionProcessor to consume routed speech and publish normalized transcription events.
    • Router processors start in registration order and stop in reverse order.
    • VAD and STT remain Voice plugin capabilities, while the Router owns their orchestration.

  • Added shared audio ingestion through PerceptionRuntime:

    • LiveKitAudioInputRuntime publishes normalized audio frames into PerceptionRuntime.audio.
    • Raw RTC audio is published once and shared across independent consumers.
    • Audio consumers no longer need to subscribe directly to LiveKit tracks.
    • Each consumer maintains its own perception cursor.
    • Derived speech frames and completed speech segments are published into PerceptionRuntime.speech.
    • Duplicate pre-roll frames can be identified and ignored by downstream consumers.

  • Added an AlphaAvatar-native VAD abstraction:

    • Added asynchronous audio-frame submission and voice-activity event streaming.
    • Added configurable activation and deactivation thresholds.
    • Added minimum speech and silence durations.
    • Added pre-roll preservation around detected speech.
    • Added bounded input queues and inference timeout protection.
    • Added an AlphaAvatar Silero VAD implementation.
    • Router speech segmentation now uses the AlphaAvatar VAD path instead of depending on LiveKit Agent’s internal VAD node.

  • Added an AlphaAvatar-native STT abstraction:

    • Added normalized interim and final transcription events.
    • Added openai_realtime for long-lived streaming transcription connections.
    • Added openai_segment for transcription of completed routed speech segments.
    • Added non-blocking frame and segment submission.
    • Added provider-local audio resampling.
    • Added connection recovery and graceful shutdown behavior.
    • Transcription sources are created independently for each audio source.
    • AlphaAvatar transcription events are temporarily bridged into LiveKit speech events for the existing turn pipeline.

  • Added the AlphaAvatar inference runtime:

    • Added InferenceRunner as the registration and implementation contract for local inference workloads.
    • Added InferenceRuntime as the Worker-owned runtime responsible for managing runner processes.
    • Added InferenceExecutor as the session-scoped asynchronous inference client.
    • Each inference runner is initialized once in its own persistent process.
    • Job processes communicate with the Worker inference runtime through Unix domain sockets.
    • Persistent method connections are reused throughout the session.
    • Requests to the same runner remain serialized, while different runners execute independently.
    • Slow speaker, face, or VDB requests no longer block latency-sensitive VAD inference.
    • Runner startup, health checking, timeout handling, shutdown, and temporary socket cleanup are managed by AlphaAvatar.

  • Migrated AlphaAvatar inference workloads away from the shared LiveKit inference executor:

    • Migrated Silero VAD inference.
    • Migrated speaker-vector inference.
    • Migrated speaker-attribute inference.
    • Migrated Persona face analysis.
    • Migrated Persona LanceDB and Qdrant runners.
    • Migrated Memory LanceDB and Qdrant runners.
    • Migrated MCP LanceDB and Qdrant runners.
    • All AlphaAvatar VDB workloads now run through the AlphaAvatar inference runtime.
    • LiveKit’s inference executor remains only for LiveKit-owned compatibility components such as the current turn detector.

  • Updated Persona speaker recognition:

    • Speaker recognition now consumes routed observations from PerceptionRuntime.speech.
    • Speaker processing no longer depends on LiveKit Agent VAD segmentation.
    • Incoming 20 ms speech frames are accumulated into rolling multi-second windows.
    • Speaker-vector inference runs at a configurable step interval rather than once per audio frame.
    • Speaker attributes are evaluated less frequently than speaker identity.
    • The pending inference queue favors the latest available speaker window.
    • Stale windows are dropped instead of allowing speaker inference to fall progressively behind realtime input.
    • Speaker-vector and speaker-attribute inference run in independent processes.
    • Speaker embeddings are matched against the session cache before querying persistent Persona storage.
    • Recognized persistent identities are loaded and merged with unresolved session profiles.

  • Updated Persona face recognition:

    • Face observations are consumed through the shared PerceptionRuntime.
    • Perception consumption is separated from face-model inference.
    • Frames are transferred into a bounded latest-frame queue.
    • Consumer cursors are committed before slow face inference begins.
    • Face analysis now runs in its own persistent inference process.
    • Face vectors and attributes are published back into the Persona cache.
    • Face annotations are attached to the original observation.
    • Annotated payload views are rendered separately without modifying raw media representations.

  • Updated Persona profile persistence:

    • ProfilerRuntime now receives the session AvatarRuntime.
    • Persona profile load, search, and save operations use AvatarRuntime.inference.
    • The configured Persona VDB runner is selected after configuration resolution.
    • LanceDB and Qdrant backends are registered dynamically before InferenceRuntime starts.
    • Profile details, runtime state, speaker vectors, and face vectors continue to share the same persistent Persona identity.
    • Session-collected speaker or face evidence can be merged into a resolved persistent profile when modalities do not conflict.

  • Updated Sampled Frame Vision:

    • Vision consumes video and screen observations independently.
    • Observations are transferred into a bounded local snapshot buffer.
    • Perception cursors are committed immediately after local ownership is established.
    • Snapshots retain references to shared observations rather than eagerly copying resolved media.
    • Late face or visual annotations can therefore be available when context is injected into the LLM.
    • Annotated payloads are resolved at model-injection time.
    • Slow LLM execution no longer holds shared perception cursors.

  • Reworked online ENV Memory scheduling:

    • Observation capture is separated from slow multimodal extraction.
    • Captured observations are transferred into a local pending batch.
    • Perception cursors are committed before model inference begins.
    • Periodic ENV updates and user-turn-triggered updates share the same scheduler.
    • Successful triggered captures refresh the periodic timer.
    • Pending batches are merged while extraction is already running.
    • ENV extraction remains serialized to avoid overlapping state updates.
    • Retry behavior is bounded.
    • Pending ENV work is drained during session shutdown.
    • ENV Memory is generated only when configured visual or audio evidence is available.
    • User text can provide context for an observation but does not independently create ENV memory.

  • Strengthened the shared perception consumer pattern:

    • Consumers wait for their own pending observations.
    • Required data is transferred into local state or a bounded local queue.
    • The consumer cursor is committed immediately after ownership transfer.
    • Slow model inference runs independently after the shared cursor has advanced.
    • Each module can process the same observation without blocking other consumers.
    • Bounded queues explicitly discard stale work when downstream processing cannot keep up.
    • Perception gaps clear local rolling state where continuity can no longer be guaranteed.

  • Updated runtime composition and lifecycle:

    • AvatarRuntime now aggregates SessionRuntime, ContextRuntime, PerceptionRuntime, and InferenceExecutor.
    • A single session-scoped InferenceExecutor is shared by Router, Persona, Face, Speaker, Profiler, Memory, and MCP consumers.
    • Runtime plugins stop before the session inference executor is closed.
    • RTC sources stop before Router processors and downstream perception consumers.
    • Worker-level inference runner processes remain alive across individual sessions.
    • Worker shutdown closes all runner processes and removes the inference socket endpoint.

  • Added clearer source, processor, and sink startup ordering:

    • Persona, Memory, and Vision consumers start first.
    • Interaction Router processors start after downstream consumers are ready.
    • LiveKit audio and video sources start last.
    • Shutdown runs in the opposite direction.
    • This prevents observations from being published before consumers are ready.
    • It also prevents new observations from entering the system while processors and sinks are shutting down.

Notes

  • LiveKit remains the default RTC backend in v0.6.5.
  • LiveKit continues to manage RTC transport, user interruption handling, turn detection, user-turn commitment, LLM execution, and TTS playback.
  • AlphaAvatar runs its own Silero VAD for Router speech segmentation, while LiveKit VAD is temporarily retained for the current interruption and turn-detection pipeline.
  • AlphaAvatar STT results are temporarily converted into LiveKit-compatible speech events through the STT bridge.
  • All AlphaAvatar VDB runners have migrated to the new AlphaAvatar inference runtime.
  • The remaining LiveKit inference process is used only by LiveKit-owned compatibility components and no longer hosts AlphaAvatar VAD, Persona, Memory, MCP, face, speaker, or VDB workloads.
  • Local speaker and face models may still require hundreds of milliseconds on CPU. This is no longer considered a realtime failure unless processing exceeds the configured inference cadence and causes pending work to accumulate.
  • Existing integrations that instantiate Persona Profiler, Face, Speaker, VAD, or VDB components directly must provide the shared AvatarRuntime or its session inference executor through the updated plugin path.
  • Existing Voice configurations should migrate STT plugin names to openai_segment or openai_realtime.

Internal Direction

v0.6.5 extends the v0.6.4 shared perception architecture into audio processing, interaction routing, and local inference isolation:

RTC / Device Adapters
        ↓
EnvObservation + MediaPayload
        ↓
PerceptionRuntime
  ├── audio
  ├── speech
  ├── video
  ├── screen
  ├── events
  └── annotations
        ↓
Interaction Router
  ├── Audio Activity
  └── Speech Transcription
        ↓
Persona / ENV Memory / Sampled Frame Vision / Agent Turn Pipeline

Local inference now follows a separate Worker-owned path:

Session / Job Process
        ↓
AvatarRuntime.inference
        ↓
InferenceExecutor
        ↓
Unix Domain Socket
        ↓
InferenceRuntime
  ├── Voice VAD Runner
  ├── Speaker Vector Runner
  ├── Speaker Attribute Runner
  ├── Face Analysis Runner
  ├── Persona VDB Runner
  ├── Memory VDB Runner
  └── MCP VDB Runner

The architectural responsibility is now clearer:

  • RTC adapters normalize and publish external media once.
  • avatar-core owns transport-agnostic observations, payloads, streams, annotations, timelines, and consumer cursors.
  • The Interaction Router transforms raw perception into derived speech and transcription events.
  • avatar-agents owns runtime composition, lifecycle ordering, interaction orchestration, and the inference client.
  • InferenceRuntime owns persistent local model and VDB processes.
  • Plugins consume shared perception streams and shared inference services without depending directly on LiveKit Agent internals.
  • Slow model inference is isolated from shared perception cursor advancement.
  • LiveKit currently remains the compatibility layer for turn decisions, LLM execution, and TTS.
  • Future releases will move turn management into the Router, remove the temporary STT bridge and duplicate VAD path, and publish normalized interaction events through an AlphaAvatar-owned interaction runtime.

Full Changelog: v0.6.4...v0.6.5

Full Changelog: v0.6.4...v0.6.5

Assets 2
Loading