Skip to content

feat(mobile): add on-device Pocket TTS - #3290

Draft
johnmatthewtennant wants to merge 18 commits into
jtennant/pocket-tts-april-runtimefrom
jtennant/mobile-pocket-tts
Draft

feat(mobile): add on-device Pocket TTS#3290
johnmatthewtennant wants to merge 18 commits into
jtennant/pocket-tts-april-runtimefrom
jtennant/mobile-pocket-tts

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Context

Desktop Buzz can read conversations aloud with an on-device Pocket TTS model. Mobile did not have the same private, offline voice path.

Summary

Adds on-device Pocket TTS to Buzz Mobile using the shared April 2026 runtime. The model is downloaded after install, synthesized on a warm worker isolate, and played through native iOS and Android audio.

Changes

  • Adds a minimal buzz-voice-mobile C ABI with persistent engine reuse, FP32 model selection, cancellation, and reset support.
  • Shares desktop text preparation and PCM shaping, including the resident April runtime's exact 50-token chunking behavior.
  • Pins official sherpa-onnx 1.13.4 mobile artifacts and checksums separately from Pocket model revision 58a6d00cf13d239b6748cb0769f35c580a8f606c.
  • Downloads and verifies 439,555,904 bytes of core model artifacts, 440,213,643 bytes including voice and license files, into a versioned v4 cache with atomic promotion, retries, cancellation, and disk-space checks.
  • Integrates native iOS audio session and Android audio focus, routing, and playback. The controller enters speaking state only after native playback actually starts.
  • Preserves message ordering, steering, cancellation, and no-repeat behavior while excluding forum channels.
  • Exposes structured model-load, synthesis, and playback failures for future native-system fallback PRs.

Related issue

N/A

Testing

  • just ci
  • Shared voice crate: 45 passed, 6 ignored; all 5 real-model ignored gates also passed with the April artifacts.
  • Mobile ABI crate: 3 passed, 1 ignored; the real FP32 ABI test passed.
  • Flutter: 845 passed, 1 skipped.
  • iOS Runner tests: 13 passed, 1 model probe skipped. Native audio-session coverage passed.
  • Built official sherpa-onnx 1.13.4 for iOS device arm64, simulator arm64/x86_64, and Android arm64/x86_64.
  • Verified Android arm64/x86_64 libraries export create, cancel, and reset-cancel symbols.
  • Ran the production Android integration on an isolated arm64 emulator with the real downloader, model, native playback, queued cancellation, and in-flight cancellation.
  • Affected re-reviews from Builderbot, Kalvin, Dead Code, and Timeless Comments found no remaining issues.

Screenshots

The current Pocket Voice settings screenshot is attached in a PR comment.

Reviewer-reproducible examples

The production Android integration completed a full 440,213,643-byte download and reported:

BUZZ_POCKET_METRICS first_pcm_ms=951 ttfap_ms=3137 synthesis_ms=937 audio_s=2.820 rtf=0.333 playback_ms=2869 warm_pcm_ms=951 queued_cancel_ms=5 cancel_ms=241 download_ms=75838 rss_bytes=1612513280 peak_rss_bytes=1768235008

The integration exercises cold download, warm synthesis, native playback, immediate queued cancellation, and cancellation during synthesis.

Notes

  • The shared runtime preserves the current stochastic seed behavior because sherpa-onnx does not expose an equivalent seed control.
  • Model-size selection remains a separate follow-up.
  • Lowest-memory physical-device validation remains a final hardware gate.
  • iOS and Android native system TTS fallbacks remain separate follow-up PRs.

Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
* origin/main:
  Publish symbol-bearing debug relay images (#3250)
  feat(tracing): add datastore tracing plumbing (#2760)
  fix(buzz-acp): accept id-keyed config options when resolving model switch (#2795)
  fix(desktop): probe legacy Goose install dir on Windows (#3248)
  refactor(desktop): extract install command execution into install_exec (#3251)

Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
Signed-off-by: John Tennant <jtennant@squareup.com>
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/mobile-pocket-tts branch from 199f931 to c0e3080 Compare July 28, 2026 13:17
@johnmatthewtennant
johnmatthewtennant changed the base branch from jt/buzz-voice-refactor to jtennant/pocket-tts-april-runtime July 28, 2026 13:18
johnmatthewtennant added a commit that referenced this pull request Jul 28, 2026
johnmatthewtennant added a commit that referenced this pull request Jul 28, 2026
@johnmatthewtennant

Copy link
Copy Markdown
Contributor Author

🤖 ### Pocket Voice settings

The mobile settings surface now describes the April 2026 on-device model and its approximately 440 MB post-install download.

pocket_voice_settings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant