Skip to content

feat: wake word detection with openWakeWord#18

Merged
0xharkirat merged 14 commits into
mainfrom
feat/wake-word
Apr 11, 2026
Merged

feat: wake word detection with openWakeWord#18
0xharkirat merged 14 commits into
mainfrom
feat/wake-word

Conversation

@0xharkirat
Copy link
Copy Markdown
Contributor

@0xharkirat 0xharkirat commented Apr 11, 2026

Summary

  • "Hey Hark" wake word detection using openWakeWord (Apache 2.0) + ONNX Runtime
  • Custom trained model (201KB) for "Hey Hark" phrase via openWakeWord Colab notebook
  • Full Pigeon integration: startWakeWordService/stopWakeWordService/isWakeWordRunning/setWakeWordPaused
  • Auto-starts mic on detection, pauses during STT, resumes after
  • Works even when app is backgrounded (engine persists in HarkApplication)

How it works

AudioRecord (16kHz) -> openWakeWord Engine -> "Hey Hark" detected
    -> HarkResultFlutterApi.onWakeWordDetected()
        -> ChatNotifier.onMicPressed() (auto-start STT)

Mutual exclusion with STT: wake word engine fully stops during STT (AudioRecord conflict on Moto G56), restarts after command completes.

Key files

File Purpose
WakeWordDetector.kt Wraps openWakeWord engine with start/stop/pause/resume
HarkPlatformPlugin.kt Implements Pigeon stubs, wires detector to FlutterApi
messages.dart Added wake word APIs to HarkCommonApi + HarkResultFlutterApi
oacp_result_service.dart Exposes wakeWordDetections stream
chat_notifier.dart Subscribes to wake word stream, auto-mic, pause/resume around STT
hey_harkh.onnx Custom trained wake word model (201KB)
wake-word-phases.md Phased plan: Phase 1 (in-app), Phase 2 (background service), Phase 3 (continuous session + AEC)

Dependencies added

  • xyz.rementia:openwakeword:0.1.4 (Maven Central, Apache 2.0)
  • com.github.gkonovalov.android-vad:silero:2.0.10 (JitPack, MIT)
  • JitPack repo added to both plugin and app build.gradle

Known limitations

  • ~25 second buffer rebuild after STT (engine restart clears audio context)
  • Auto-starts on main engine only (no background foreground service yet)
  • No settings screen / toggle (auto-starts after init)
  • Green mic indicator when active (Android privacy feature, unavoidable)

Test plan

  • App loads, wake word auto-starts after splash
  • Say "Hey Hark" -> mic auto-starts listening
  • Say a command ("turn on the flashlight") -> dispatches correctly
  • After command, say "Hey Hark" again -> re-triggers (with buffer rebuild delay)
  • Normal long-press home overlay still works alongside wake word

🤖 Generated with Claude Code

0xharkirat and others added 14 commits April 11, 2026 15:27
@0xharkirat @claude
feat(wake-word): openWakeWord + Pigeon integration
6d5dab2 
Wire end-to-end wake word detection pipeline:

Kotlin side:
- WakeWordDetector.kt: wraps openWakeWord's WakeWordEngine with
  start/stop/pause/resume lifecycle
- HarkPlatformPlugin: implements startWakeWordService/stopWakeWordService/
  isWakeWordRunning/setWakeWordPaused, fires onWakeWordDetected callback

Pigeon schema:
- Added startWakeWordService, stopWakeWordService, isWakeWordRunning,
  setWakeWordPaused to HarkCommonApi
- Added onWakeWordDetected to HarkResultFlutterApi

Dart side:
- OacpResultService: exposes wakeWordDetections stream
- ChatNotifier: subscribes to wake word stream, auto-starts mic on
  detection, pauses wake word during STT, resumes after

Dependencies:
- xyz.rementia:openwakeword:0.1.4 (Apache 2.0)
- com.github.gkonovalov.android-vad:silero:2.0.10 (MIT)
- ONNX Runtime forced to 1.23.0 across all deps

Assets:
- wakeword/hello_world.onnx (placeholder until hey_hark.onnx is trained)
- wakeword/melspectrogram.onnx + embedding_model.onnx (preprocessing)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
fix(wake-word): add JitPack repo for android-vad dependency
c27464a 
android-vad (Silero VAD wrapper) is hosted on JitPack, not Maven Central.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
fix(wake-word): add JitPack repo to app-level build.gradle
46ff36d 
android-vad is on JitPack. The plugin's allprojects block doesn't
propagate to the app module's dependency resolution, so JitPack must
be added to the app's android/build.gradle.kts as well.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
feat(wake-word): auto-start wake word after chat init
8391d85 
Start wake word detection automatically once ChatNotifier finishes
initialization. This is temporary until a proper settings screen
with toggle is added.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
fix(wake-word): move preprocessing models to assets root
8633e2d 
openWakeWord's MelSpectrogram and EmbeddingModel classes hardcode
asset paths to root-level melspectrogram.onnx and embedding_model.onnx.
Moved preprocessing models to assets root, kept wake word model in
wakeword/ subfolder.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
fix(wake-word): resume detection after STT onDone completes
2274d4b 
Wake word was only resumed in cancelListening() but not in the STT
onDone callback path. When STT timed out naturally, wake word stayed
paused permanently. Now resumes in both paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
fix(wake-word): fully stop engine on pause to release AudioRecord
21d3c1a 
Android only allows one AudioRecord at a time. The previous pause()
just suppressed callbacks but kept the engine's AudioRecord open,
preventing STT from acquiring the mic. Now pause() fully stops the
engine and releases the mic, resume() restarts it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
fix(wake-word): lower threshold to 0.3 and cooldown to 1500ms
976b185 
Detection scores range 0.515-0.726 for 'hello world'. Lowering
threshold from 0.5 to 0.3 for faster detection. Reducing cooldown
from 3s to 1.5s for quicker re-trigger after commands.

The main delay is the feature buffer rebuild after engine restart
(~10s of audio needed). This is inherent to the openWakeWord
architecture and will improve with the trained hey_hark model.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
perf(wake-word): keep engine running during STT, suppress detections …
8e0fbdd 
…only

Instead of full stop/restart on pause/resume (which loses the 10-second
audio buffer and causes detection delay), keep the engine and AudioRecord
running continuously. Only suppress detection callbacks during STT.

Android's SpeechRecognizer manages its own audio session and can coexist
with AudioRecord on most devices. This eliminates the buffer rebuild
delay, making wake word detection resume instantly after commands.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat
Revert "perf(wake-word): keep engine running during STT, suppress det…
e5a701e 
…ections only"

This reverts commit 8e0fbdd.
@0xharkirat @claude
docs: wake word phased plan with AEC + barge-in research topics
602902d 
Document three phases of wake word implementation:
- Phase 1 (current): in-app detection with openWakeWord
- Phase 2: background foreground service with VAD gating
- Phase 3: continuous listening session with echo cancellation,
  barge-in, and no repeated wake word needed

Phase 3 research covers AcousticEchoCanceler, VOICE_COMMUNICATION
audio source, WebRTC AEC, and ChatGPT voice mode observations.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
feat(wake-word): add trained hey_harkh.onnx model
78e99fd 
Replace hello_world.onnx placeholder with custom-trained "Hey Hark"
model (hey_harkh.onnx, 201KB). Trained via openWakeWord Colab notebook
with phonetic spelling "hey harkh" for better pronunciation matching.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
docs: update AGENTS.md, README.md, ROADMAP.md for wake word
e90aef0 
- AGENTS.md: added WakeWordDetector.kt to file tree, wake word assets,
  openWakeWord + android-vad to deps, updated API table
- README.md: added Wake Word section, updated roadmap bullets
- ROADMAP.md: wake word status changed to in-progress with 7 shipped
  and 6 remaining sub-items

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat @claude
chore: remove hello_world.onnx placeholder, add review TODOs
19ba34a 
- Removed hello_world.onnx (202KB) since hey_harkh.onnx is shipped
- Added TODO for silent failure on null context in startWakeWordService
- Added TODO for isRunning returning false during pause
- Added TODO for wake word resuming before TTS completes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@0xharkirat 0xharkirat merged commit 95db6a8 into main Apr 11, 2026
@0xharkirat 0xharkirat mentioned this pull request Apr 12, 2026
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@0xharkirat