fix(queue): harden drain threads against BaseException - #129
Merged
Conversation
SpeechQueue._drain() and ChannelQueue._drain() both catch only 'except Exception', leaving them vulnerable to BaseException subclasses (SystemExit, KeyboardInterrupt, MemoryError, GeneratorExit). When a BaseException escapes the inner handler, the drain thread exits while leaving _draining / _running = True permanently. Subsequent enqueue / submit calls skip starting a new drain thread, so jobs pile up in the queue indefinitely with active_jobs=0 and queue_depth growing without bound (the reported queue_depth=16 stale counter bug). Fix: wrap each drain loop in try/finally. The finally block unconditionally resets _draining / _running = False and clears the active-job pointer under the lock, so any exit path — normal (queue empty) or abnormal (BaseException) — leaves the queue in a consistent state. KeyboardInterrupt and SystemExit are re-raised after cleanup so the process can still respond to signals normally. Tests: add TestDrainBaseExceptionResilience to test_speech_queue.py and a new test_output_queue.py that covers the ChannelQueue fix. Both suites cover SystemExit, MemoryError, and the end-to-end resume case (new jobs accepted after a BaseException kill).
I001 import sorting + F401 unused QueuedJob import + format drift in test_output_queue.py / test_speech_queue.py.
7 tasks
chazmaniandinkle
added a commit
that referenced
this pull request
Jul 6, 2026
* feat(vad): add POST /v1/vad/confidence — per-packet ONNX confidence endpoint
Adds a lightweight per-packet VAD endpoint for barge-in loops (Discord VC,
etc.) that do not need full utterance-level speech detection.
Why:
- /v1/vad accepts a WAV multipart upload and runs the full Silero torch path.
That's the right API for utterance VAD but overkill for 512-sample frames
arriving at 20ms intervals from a Discord SocketReader thread.
- The vendored pipecat SileroVADAnalyzer (vendor/pipecat_vad/) uses ONNX,
has no torch dependency, returns per-frame confidence in <5ms, and was
already present in main but had no HTTP surface.
Contract:
POST /v1/vad/confidence[?sample_rate=16000]
Body: raw int16 little-endian PCM (exactly 512 samples @ 16kHz or 256 @ 8kHz)
Response: {confidence: float, available: bool, latency_ms: float}
available=false (and confidence=0.0) when onnxruntime is not installed.
Empty body returns {confidence: 0.0, available: <bool>, latency_ms: 0.0}.
Also:
- Imports is_pipecat_vad_available + voice_confidence from vad.py (already in main)
- Listed in GET /capabilities endpoints manifest as 'vad_confidence'
- 12 tests covering happy path, unavailable path, empty body, sample_rate
param forwarding, capabilities manifest
Updates: fix mod3/http_api.py file header comment and endpoints dict.
* fix(vad): fix Silero ONNX output shape — squeeze (1,1) to scalar
The model __call__ returns shape (batch, frames). With batch_size=1
and a single frame, out[0] is shape (1,) — a 1D array. float() on a
1D array raises 'only 0-dimensional arrays can be converted to Python
scalars', so voice_confidence() was silently returning 0.0 on every
frame. VAD was effectively disabled despite reporting available=true.
Fix: float(np.squeeze(new_confidence).item()) — works for any batch
size and output shape.
Root cause found via direct venv test after the HTTP endpoint showed
0.0 confidence on sine waves and noise. The error was swallowed by the
bare except in voice_confidence().
* fix(tests): add missing is_pipecat_vad_available + voice_confidence to vad stub
_stub_heavy_deps() in test_version.py built a vad stub module but omitted the
two attributes that http_api imports at module level (the F2 pipecat wrapper),
causing ImportError on setup for test_health_version_matches_package_version
and test_capabilities_version_matches_package_version on every run.
* fix(tests): add conftest.py restore_sys_modules fixture; use it in test_version.py
test_version.py's _stub_heavy_deps() injected fake engine/vad/numpy/mlx
modules into sys.modules without restoring them. Any test file collected
after test_version.py got the stub engine instead of the real module, causing
~28F/42E in combined runs. Added conftest.py with a restore_sys_modules
fixture (save-and-restore pattern) and changed http_app in test_version.py
to request it, scoping it down from module to function so each test is
fully isolated.
* fix(config): add [tool.pytest.ini_options] testpaths = ["tests"]
Without testpaths, pytest defaulted to collecting from the worktree root,
picking up vendor/, integrations/, and production modules as potential test
sources, causing spurious collection errors. Pin discovery to tests/.
* fix(tests): CI-mode stubs + HAS_* guards for Apple-Silicon and native-lib test files
conftest.py now:
- Checks real lib availability (HAS_MLX, HAS_MCP, HAS_TORCH) before any
stubs are installed, so importorskip checks reflect genuine presence.
- Installs lightweight MagicMock stubs for torch, sounddevice, mcp, mlx, etc.
when absent so module-level imports in production code don't abort
collection on linux/CI.
test_voice_profiles.py, test_voice_profiles_e2e.py: skip via HAS_MLX check
before the hard mlx_audio module-level imports rather than importorskip (which
would succeed against the stub).
test_speaking_lock.py, test_mcp_route.py: skip via HAS_MCP before the hard
server import that requires mcp at module level.
* fix(ci): remove || echo 'No tests yet' masking; install CI-safe deps
The test job was running pytest with || echo 'No tests yet', which swallowed
collection errors and failures silently — the badge stayed green even when 0
tests ran or the entire collection errored.
Changes:
- Remove the || echo fallback; pytest exit code now propagates as-is.
- Replace pip install -r requirements.txt (which includes Apple-Silicon-only
libs that fail to install on ubuntu-latest) with an explicit CI-safe list:
fastapi, uvicorn, httpx, pysbd, numpy, scipy, pydantic, pytest,
pytest-asyncio, python-multipart, mcp, mistral-common.
- Tests needing mlx/torch/sounddevice skip via HAS_MLX / HAS_MCP guards
(conftest.py) rather than erroring — collection always succeeds and the
job is honestly green.
* fix(engine): log WARNING when speed= is silently ignored by chatterbox, chatterbox-turbo, voxtral
These engines accept speed as a parameter signature but discard it at synthesis
time (only Kokoro and Spark honour supports_speed). Callers passing speed != 1.0
now receive a single WARNING per generate_audio() call so the silent drop is
visible in logs and operator tooling.
Warning fires once per call (outside the per-sentence loop) to avoid log spam.
Engines with supports_speed=True and spark (which maps speed to a discrete set)
are unaffected.
* docs(changelog): note /health queue.depth -> queue.jobs_total as breaking change
The rename happened silently in commit 65cf1b5 (PR #129) with no changelog
entry. Callers reading queue.depth to check backlog size need to update to
queue.jobs_total.
* fix(tests): add needs_mcp / HAS_MLX guards to remaining tests affected by CI stubs
Five test files had tests calling @mcp.tool-decorated server functions or
mlx_audio internals at call time, which become MagicMock under CI stubs:
- test_speech_queue.py: @needs_mcp on TestSpeakReturnFormat,
TestStopReturnFormat, TestSpeechStatusReturnFormat (call server.speak/stop/
speech_status which are @mcp.tool decorated).
- test_dashboard_chat.py: @needs_mcp on TestMod3DashboardPostTool.
- test_bargein_provider_registry.py: @needs_mcp on the three
test_await_voice_input_* functions that import server.
- test_voice_profiles_api.py: HAS_MLX module-level skip (calls
_make_synthetic_conditionals which imports mlx_audio.tts.models.chatterbox).
- test_voice_profile_curation.py: same HAS_MLX module-level skip.
* fix(tests): extend mcp stub submodules; add HAS_MLX guard to test_compositions_api
conftest.py: add mcp.shared.message to the submodule stub list so that
channel_client.py can be exec_module'd in tests (it imports from mcp.server.stdio
and mcp.shared.message at module level).
test_compositions_api.py: _make_synthetic_conditionals() imports mlx_audio at
call time; skip the whole module via HAS_MLX guard like the other mlx-dependent
test files.
* fix(tests): tomllib py3.10 compat + stop stubbing local modules with MagicMock
test_version_module_matches_pyproject: try tomllib (3.11+ stdlib), fall
back to regex parse of pyproject.toml on 3.10 — no tomli dependency.
_stub_heavy_deps: stub only third-party/native packages. Stubbing local
pure-Python modules (bus, modality, session_registry, audio_subscribers)
with bare MagicMock poisoned server.diagnostics() with non-JSON-serializable
values when tests shared a session. Documented the rule in the docstring.
Full suite on this machine: 940 passed, 10 failed — all 10 verified
PRE-EXISTING on origin/main (kernel_session_callback, rtvi lifecycle,
stt-executor timing; machine-environment dependent). Hang reports during
verification were three concurrent pytest sessions deadlocking, not a test.
* fix(ci): restore soundfile dep, fix import sort, add missing HAS_MLX guard
- ruff I001: sort imports in http_api.py (vad import), tests/conftest.py,
and tests/test_bargein_provider_registry.py
- ci.yml: restore soundfile to both pip-install steps (test-import and
test-api jobs); encode_ogg/compose endpoints need it and tests were
going red once the "|| echo No tests yet" mask was removed
- tests/test_voice_profile_compose_api.py: add the HAS_MLX skip-guard
used by its sibling voice-profile test files so it skips cleanly
instead of collection-erroring under the CI dependency set
* chore: trigger CI
* fix(ci): guard torch import in vad.py, clean up lint/format violations
The HTTP API Smoke Test job installs a CI-safe dependency subset that
deliberately excludes torch (matching the Import & Unit Tests job's
comment on Apple-Silicon-only/native libs). Its bare `python -c` script
never loads tests/conftest.py, so the module-level `import torch` in
vad.py had no stub to fall back on and crashed the import chain
(http_api -> vad -> torch) with ModuleNotFoundError.
Make the torch import lazy/optional in vad.py, following the same
HAS_MLX/HAS_TORCH graceful-degradation convention already used
elsewhere in the test suite: _get_model() now raises a clear
RuntimeError only if Silero model loading is actually attempted
without torch installed, instead of failing at import time.
Also fixes Lint & Type Check:
- ruff check: drop two unused imports (MagicMock, patch) and reorder
import blocks in tests/test_vad_confidence_endpoint.py
- ruff format --check: reformat 4 more files (conftest.py,
test_bargein_provider_registry.py, test_dashboard_chat.py,
test_version.py) that had drifted from ruff's formatter in earlier
commits on this branch; origin/main formats clean, confirming this
drift was introduced by this branch and not pre-existing
- pyright: relax reportOptionalMemberAccess for vad.py's existing
pyright suppression comment, since the try/except torch import
makes `torch` Optional from pyright's view
Verified locally against the exact CI toolchain (ruff 0.15.20, pyright
1.1.411, Python 3.12) and by reproducing the smoke test's dependency
subset (no torch) to confirm http_api now imports and serves cleanly.
* fix(ws): send close frame on disconnect-bot before and after RTVI handshake
The ws_audio handler's disconnect-bot path broke out of its loop straight
into finally: subs.unregister(...) without ever calling
await websocket.close(), so no close frame was sent. Worse, a
disconnect-bot frame arriving before the RTVI handshake completed was
silently ignored and fell through into the drain loop's receive(), which
then blocked forever since the client sent nothing further.
Guard close() with an application_state CONNECTED check (matching the
pattern in the existing RTVI version-mismatch close a few lines up) so
it's a no-op if the socket is already closing, and return/unregister
still happens via finally.
Fixes the hang in
tests/test_audio_subscribers.py::TestDisconnectBot::test_disconnect_bot_closes_and_unregisters
that was blocking the Import & Unit Tests CI job.
* fix(tests): override Host header for CSRF-guarded TestClient calls
tests/test_kernel_session_callback.py's mutating client.post/delete calls
never overrode the default "Host: testserver" TestClient sends, so the
app's _localhost_csrf_guard middleware correctly 403'd every one of them
before reaching the handler. Adopt the same _GOOD_HOST header pattern
already established in test_localhost_csrf_guard.py.
* test: skip mlx_lm tokenizer regression guard when mlx is unavailable
mlx_lm is Apple-Silicon/Metal-only and is intentionally excluded from
the Import & Unit Tests CI job's Linux install list, so a bare
`import mlx_lm` raised ModuleNotFoundError instead of skipping cleanly.
Gate TestMlxLmTokenizerRegistration with the existing skip_without_mlx
marker, matching the convention already used for the rest of the mlx
stack in conftest.py.
* fix(tests): use a local executor for STT pool-isolation assertion
test_slow_stt_does_not_block_default_pool imported the shared, process-
global channels._STT_EXECUTOR by reference. Any earlier test file in the
same pytest process that runs `with TestClient(app) as c:` triggers the
app's full lifespan teardown, which shuts down that same executor for
real via shutdown_stt_executor(). Depending on collection order, the
executor was already dead by the time this test ran, raising
"cannot schedule new futures after shutdown".
Give the test its own throwaway ThreadPoolExecutor instance instead,
mirroring the pattern test_shutdown_stt_executor_is_safe already uses,
so the isolation assertion no longer depends on the shared singleton's
lifecycle.
* ci: skip TTS-model-dependent test in Import & Unit Tests job
test_synthesize_with_subscriber_emits_rtvi_over_ws exercises the real
POST /v1/synthesize endpoint with no engine mocking, requiring an actual
TTS backend and model weights that the Linux test-import job never
installs. The test already carries a SKIP_TTS_TESTS-gated skipif guard,
but the workflow never set that env var, so the guard was dead code.
Wire SKIP_TTS_TESTS=1 into the job's test step.
* fix: revert out-of-scope /health queue field rename
The /health endpoint's queue.jobs_total field is reverted back to
queue.depth, restoring consistency with origin/main and with
server.py and output_queue.py, which both still emit "depth" for
the same concept. The rename was unrelated to this PR's actual
scope (test-suite credibility and CI honesty) and had no test
coverage depending on it.
CHANGELOG.md's entry documenting the breaking rename is replaced
with an accurate entry describing this PR's real change: unmasking
the test suite and fixing/skipping the pre-existing failures it had
been hiding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SpeechQueue._drain()(server.py) andChannelQueue._drain()(output_queue.py) both have anexcept Exceptionguard around the job runner, but Python'sBaseExceptionhierarchy includes subclasses that don't inherit fromException:SystemExit(fromsys.exit()or process lifecycle)KeyboardInterrupt(from signal handling)MemoryError(OOM during TTS synthesis)GeneratorExit(from a TTS generator being closed)When any of these escape the inner handler, the drain thread exits with
_draining/_runningstillTrue. Every subsequentenqueue()/submit()call sees the flag and skips starting a new drain thread, so jobs pile up in the queue indefinitely:SpeechQueue.depth(reported indiagnostics()["queue_depth"]) grows without boundactive_jobsstays at 0This is the root cause of the reported queue_depth=16, active_jobs=0 stale counter bug.
The code already documents the hazard (server.py lines 643–647):
Commit d3009a7 added the
except Exceptionguard, which handles the common case (KeyError, ValueError, etc.), but left theBaseExceptiongap.Fix
Wrap each drain loop in
try/finally. Thefinallyblock unconditionally resets the flag (_drainingor_running) and clears the active-job pointer under the lock, so any exit path — normal or abnormal — leaves the queue in a consistent, restartable state.KeyboardInterruptandSystemExitare re-raised after cleanup so the process can still respond to signals normally.Tests
tests/test_speech_queue.py— new classTestDrainBaseExceptionResilience:test_drain_resets_draining_after_system_exittest_drain_resets_draining_after_memory_errortest_drain_accepts_new_jobs_after_base_exception_kill(end-to-end: queue resumes after thread death)tests/test_output_queue.py(new file) — coversChannelQueueandOutputQueueManager:TestChannelQueueBaseExceptionResilience: same three regression cases for ChannelQueueAll 35 new + existing tests pass.