macOS CI flake: float32 tone/stability rows are nondeterministic under the Accelerate FFT backend

[tap]

Symptom

The macOS AppleClang CI job intermittently fails <float> compliance rows while the identical tree passes in a parallel or re-run job. Observed twice on PR #28 (an infra-only PR touching no C++):

• Round 1: itu_echo.EchoStability<float> + Float32Parity.ToneRowWithNarrowbandGuard failed; a duplicate run on the identical tree passed all 181.
• Round 2 (run 29955037671, attempt 1): itu_echo.EchoStability<float> + g168_adapted.ToneStability<float> failed while the simultaneous push-event run (29955033392) on the same head SHA passed. Attempt 2 passed.

The ToneStability failure is not a marginal miss: measured −7.4 dBm0 against the −49.3 gate at fs 16000 (the passing measurement is −104). The <double> variants of the same rows pass everywhere, always.

What was ruled out (measured, 2026-07-23)

• The DUT and harness are deterministic by construction: no threads, no clocks, no <random>; the comfort-noise source is xorshift32 with a hard-coded seed (postfilter.h).
• Uninitialized memory: valgrind --track-origins=yes over a full g168_adapted/0.ToneStability run — 0 errors.
• Linux nondeterminism: 30/30 consecutive passes looping g168_adapted/0.ToneStability:itu_echo/0.EchoStability on Linux (Ooura backend).

Root cause

dsptap's float32 FFT backend on macOS is Apple vDSP/Accelerate (TAP_DSP_FFT_ACCELERATE, on by default; double stays Ooura — which is why <double> never flakes). The wrapper reproduces Ooura's contract "to float epsilon", i.e. deliberately not bit-exact. Accelerate dispatches internally per CPU generation (including AMX paths), and the macos-latest runner pool is a heterogeneous Apple Silicon fleet — so the same binary produces epsilon-different spectra depending on which chip the job lands on.

The marginal float rows amplify epsilon into a binary outcome. ToneStability<float> is a race: from reset, the canceller must converge on a pure 1 kHz tone before the narrowband guard freezes adaptation (~187 blocks of sustained narrowband input, ≈3 s at 16 kHz). Win the race → −104 dBm0; lose it by epsilon → the guard strands a half-converged filter and the residual parks at −7 dBm0 for the remaining 20 s. EchoStability<float>'s 16 kHz gate has only ~0.5 dB margin at attenuation depths where the meter is reading the suppressor floor — the same sensitivity with a smaller expression.

This explains every observation: macOS-only (only platform with the Accelerate backend), float-only (backend is float-only), same-commit pass/fail across simultaneous runs (different runner chips), and the failing set wandering among exactly the documented-marginal tone/stability rows.

Remediation options (not yet done — this issue documents the finding)

1. Deterministic CI: build the macOS job with -DTAP_DSP_FFT_ACCELERATE=OFF so CI runs the Ooura float path like every other platform. Cheap, kills the flake. Cost: CI stops exercising the shipping Accelerate wrapper — consider a small dedicated backend-parity check (spectra match Ooura within a stated epsilon) so the wrapper stays covered.
2. Fix the marginality: the narrowband guard should not be able to strand an unconverged filter — e.g. arm the freeze only once the residual clears a threshold, or freeze to a reduced-µ trickle instead of zero. This addresses the real robustness finding (on some hardware, a from-reset tone leaves the float chain unconverged — a deployment risk, not just a CI nuisance), but touches certified DSP and requires re-measuring the float gate tables.
3. Both: (1) now, (2) as its own measured PR.

Workaround

Re-run the failed macOS job; the flake clears with high probability (it is a per-runner coin flip, not a property of the commit).

🤖 Generated with Claude Code

https://claude.ai/code/session_01S3twSGfwUK8jnUhp5Qx6wx