Add experimental native speech-to-text support

[leehack]

Summary

• add an experimental typed SpeechToTextEngine for whole-file Qwen3-ASR transcription over native llama.cpp
• add task-scoped events, completion state, cancellation, explicit model capability discovery, typed failures, and transcript metadata placeholders
• add a native-desktop Qwen3-ASR 0.6B chat-app preset with separate speech-to-text UX, mandatory projector handling, immutable artifact URLs, and SHA-256 verification
• harden concurrent startup, Stop/Clear/conversation-switch/unload lifecycle behavior, catalog migration, and exact-transcript test routing

The root problem is that generic LlamaAudioContent generation had no stable transcription contract and could not distinguish audio-understanding chat from an ASR workflow.

Related to #325. This PR intentionally does not close the issue.

Production-readiness scope

• User-facing scope: Native users can transcribe one complete WAV, MP3, or FLAC input through a typed API and the Flutter chat example using the Qwen3-ASR profile.
• Supported platforms/paths: Native llama.cpp with an explicitly selected Qwen3-ASR profile and an audio-capable matching projector. Real-model validation currently covers macOS arm64 CPU; a separate chat-app/manual smoke covered Metal.
• Unsupported or intentionally unavailable paths: Typed STT on WebGPU, native LiteRT-LM, and LiteRT-LM Web fails or is disabled explicitly. Live microphone/streaming audio, partial transcripts, timestamps, confidence, language claims, diarization, raw PCM, and TTS are unavailable.
• Out of scope / follow-ups: Keep Add a typed SpeechToTextEngine for native Qwen3-ASR #325 open for a redistributable fixture, Linux x64 CPU validation, malformed/silence/stereo/resampling/30-second boundary cases, and real cancellation/cleanup/memory gates. TTS remains tracked by Design experimental native speech synthesis API for Qwen3-TTS #322.

Completeness checklist

• Declared scope is fully implemented, or explicitly reduced above.
• Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
• Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant.
• Full model-backed acceptance coverage from Add a typed SpeechToTextEngine for native Qwen3-ASR #325 is pending as listed above.
• Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
• Follow-up work that is useful but not required for this PR is tracked in GitHub Issues.

PR type guidance

• Feature PR: includes API/docs/example updates, exact platform scope, unsupported-path behavior, and happy/negative-path tests.

Test Plan

• dart format --output=none --set-exit-if-changed lib test tool example/chat_app/lib example/chat_app/test
• dart analyze lib test tool
• flutter analyze in example/chat_app
• dart test -p vm -j 1 --exclude-tags local-only — 1,347 passed
• dart test -p chrome --exclude-tags local-only — 742 passed
• flutter test in example/chat_app — 215 passed
• ./tool/docs/validate_links.sh
• ./tool/docs/build_site.sh
• Coverage gate — 75.02% line coverage, above the 70% threshold
• macOS arm64 Qwen3-ASR 0.6B CPU real-model smoke; separate chat-app/manual Metal smoke

Matrix Evidence

Matrix row	Scope covered	Platform / model / backend	Result	Evidence / notes
speech-to-text-smoke	typed API, projector capability, marker normalization, exact transcript	macOS arm64 / Qwen3-ASR 0.6B Q8_0 / llama.cpp CPU	PASS	Local real-model run; checked-in redistributable fixture remains #325 follow-up
examples-tests	chat transcription UX, lifecycle, catalog, download integrity	Flutter VM / mock engine and filesystem	PASS	215 chat-app tests
Web unsupported contract	typed Web STT rejection	Chrome / Web platform stub	PASS	Included in 742 Chrome tests
docs validation	guide, example, matrix, changelog	Docusaurus	PASS	Links and production build green

Review Notes

• Independent review status: three focused read-only reviews covered the public API, chat/catalog/download lifecycle, and docs/test claims; all concrete findings were addressed. No external human approval yet.
• CI status / head SHA: all 12 required PR checks passed on 6d837317b411071732e383acdb5b6f12876d9585; the separate Copilot reviewer check also passed.

[github-actions]

Chat app preview removed for leehack/llamadart-chat-pr-326.

[Copilot]

Pull request overview

Adds an experimental, typed speech-to-text surface to llamadart, centered on native llama.cpp whole-file transcription via a Qwen3-ASR profile, and wires it through docs, tests, tooling, and the Flutter chat example (including a verified-download preset).

Changes:

• Introduces SpeechToTextEngine / request + task/event model with capability discovery, cancellation, and transcript normalization.
• Updates the chat app to expose a dedicated “Transcribe Audio” flow and adds a pinned Qwen3-ASR 0.6B preset with SHA-256 verification.
• Extends local E2E tooling/test matrix and updates documentation + changelogs to reflect the new experimental scope.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
website/sidebars.ts	Adds the new Speech-to-Text guide to docs navigation.
website/docs/platforms/support-matrix.md	Documents the narrower STT support scope vs general runtime availability.
website/docs/guides/speech-to-text.md	New guide describing the experimental typed STT API and limitations.
website/docs/guides/multimodal.md	Clarifies generic LlamaAudioContent vs typed STT and links to the new guide.
website/docs/examples/chat-app.md	Documents the chat app’s separate transcription workflow and preset sourcing.
website/docs/changelog/recent-releases.md	Adds an Unreleased entry for the STT feature/preset.
tool/testing/test_matrix.dart	Adds a targeted local-only speech-to-text-smoke row.
tool/testing/run_local_e2e.dart	Adds --audio-path and scenario wiring/validation for the STT smoke.
test/unit/tooling/run_local_e2e_test.dart	Tests the new STT local E2E scenario dry-run and argument validation.
test/unit/core/speech/speech_to_text_web_test.dart	Ensures typed STT is explicitly unsupported on Web.
test/unit/core/speech/speech_to_text_test.dart	Unit tests for STT capability probing, normalization, cancellation, and errors.
test/unit/core/speech/speech_platform_web_test.dart	Verifies the web platform stub reports unsupported STT.
test/unit/core/speech/speech_platform_stub_test.dart	Verifies the native stub reports STT as supported.
test/unit/core/exceptions_test.dart	Adds coverage for new speech-related exception types.
test/e2e/backends/speech_to_text_e2e_test.dart	Adds a local-only real-model E2E STT validation harness.
README.md	Mentions the new experimental typed STT capability and links to docs.
lib/src/core/speech/speech_to_text.dart	Implements the typed STT engine/task/events/capabilities + transcript normalization.
lib/src/core/speech/speech_platform_web.dart	Web implementation for platform-level STT unavailability.
lib/src/core/speech/speech_platform_stub.dart	Non-web implementation enabling STT platform support.
lib/src/core/models/chat/content_part.dart	Updates LlamaAudioContent docs to clarify it’s generic multimodal input.
lib/src/core/exceptions.dart	Adds LlamaSpeechException and LlamaAudioFormatException.
lib/llamadart.dart	Exports the new speech-to-text API surface.
example/chat_app/test/unit_test.dart	Validates catalog ordering/source expectations and the pinned ASR preset metadata.
example/chat_app/test/settings_service_test.dart	Tests settings persistence/migration for the STT capability flag.
example/chat_app/test/model_service_test.dart	Adds tests for SHA-256 verified downloads and cleanup on checksum failure.
example/chat_app/test/model_download_controller_adapter_test.dart	Ensures download cache entries surface verified size/SHA metadata.
example/chat_app/test/model_card_test.dart	Widget coverage for STT capability display + projector requirement UX.
example/chat_app/test/manage_models_screen_download_test.dart	Tests STT search keywords and stale custom-entry replacement behavior.
example/chat_app/test/chat_input_test.dart	Covers the “Transcribe Audio” action availability and chat transcription lifecycle.
example/chat_app/README.md	Documents the chat app’s STT flow, limitations, and integrity verification.
example/chat_app/lib/widgets/model_card.dart	Requires projector availability for STT models and displays STT capability chip.
example/chat_app/lib/widgets/chat_input.dart	Adds the “Transcribe Audio” attachment action.
example/chat_app/lib/services/settings_service.dart	Persists model_supports_speech_to_text and migrates Qwen3-ASR selection.
example/chat_app/lib/services/model_service_io.dart	Adds SHA-256 verification, caching, and discard-on-mismatch behavior.
example/chat_app/lib/services/model_download_controller_adapter.dart	Populates cache entries with bound remote asset size/SHA; clears STT when omitting projector.
example/chat_app/lib/screens/manage_models_screen.dart	Adds STT search terms and persists STT capability in custom models.
example/chat_app/lib/providers/chat_provider.dart	Implements dedicated STT picking/transcription flow and hardens lifecycle cancellation/unload.
example/chat_app/lib/models/downloadable_model.dart	Adds STT capability fields and a pinned Qwen3-ASR preset with integrity metadata.
example/chat_app/lib/models/chat_settings.dart	Adds persisted STT capability flag in settings.
example/chat_app/lib/models/chat_message.dart	Adds isTranscription derived flag to suppress regeneration for STT outputs.
doc/testing_matrix.md	Documents the new STT smoke row and its invocation.
CHANGELOG.md	Adds an Unreleased entry summarizing the experimental STT feature and preset.

Suppressed comments (2)

lib/src/core/speech/speech_to_text.dart:560

• Same issue here: the exception details is set to request.audio.format, which is not very actionable in error output. Including the rejected file extension would make diagnostics clearer.

            !const <String>{'wav', 'mp3', 'flac'}.contains(extension)) {
          throw LlamaAudioFormatException(
            'Encoded audio files must use WAV, MP3, or FLAC.',
            request.audio.format,
          );

lib/src/core/speech/speech_to_text.dart:575

• LlamaAudioFormatException uses request.audio.format as details, which will usually render as Instance of ... in logs. Passing the rejected encoding string provides more actionable diagnostics for byte-backed inputs.

          throw LlamaAudioFormatException(
            'Encoded audio bytes must use WAV, MP3, or FLAC.',
            request.audio.format,
          );

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated no new comments.