Repository navigation
Video Spectrum Frequency Resolution
Investigation of issue #89, 2026-10-21. Implementation: 2456441. The reported symptom was identical movement across the middle and lower spectrum bars during video playback.
A generated H.264/AAC video containing 55 Hz on the left and 110 Hz on the right reproduced the problem through the production VideoPlayerService, audio tap, sample ring, and analyzer. The tap supplied stereo noninterleaved Float32 at 48 kHz, normally in blocks of 4096 frames.
VideoSpectrumAccumulator previously predicted the next source timestamp by adding frameCount / sampleRate. Actual MTAudioProcessingTapGetSourceAudio asset ranges had durations such as 0.081375 and 0.093 seconds, while 4096 / 48000 is approximately 0.085333 seconds. Consecutive asset ranges joined within roughly one sample, but comparing their starts against the PCM duration exceeded the existing two-sample tolerance. Every accepted callback reset the partial samples, FFT history, levels, and held peaks. Ring overflow was not involved: the diagnostic runs recorded zero ring drops.
This left normal/fast playback repeatedly analyzing short 1024/512-point windows. Several logarithmic bands were forced to read the same first non-DC FFT bin. At the 10 Hz slow setting, 4096 frames never reached the required 4800-frame chunk before the next reset, so no analysis frames appeared.
Apple defines timeRangeOut as the asset time range corresponding to the provided source audio frames; the API does not promise that its duration equals the returned PCM frame count divided by the processing sample rate. See the Xcode 27.1 SDK's MediaToolbox/MTAudioProcessingTap.h, MTAudioProcessingTapGetSourceAudio, and Apple's API reference.
- Carry the asset range's end timestamp with the PCM, generation, and source epoch through the preallocated sample ring. Test continuity against that end timestamp, independently of the PCM chunk clock. Older synthetic callers without an end timestamp retain the PCM-duration fallback.
- Continue to reset on actual source gaps, stream starts, sample-rate/mode changes, transport generations, and source epochs. Revalidate the generation/source after FFT computation and again before MainActor publication.
- Choose a power-of-two analysis window targeting at least two FFT bins across the narrowest displayed logarithmic band, accounting for the Hann window. The 16-band windows are 8192 at 44.1 kHz, 16384 at 48 kHz, and 65536 at 192 kHz. Memory is bounded at 131072 samples per channel. Audio and video share this history in all response modes; RMS still uses the current chunk.
- Read the latest power-of-two samples from an incomplete window. Assign FFT bins to the half-open frequency range containing them using ceiling boundaries. A narrow band without a bin during startup is zero, rather than borrowing another band's bin.
Display cadence remains 10/30/60 Hz. Low-frequency resolution takes approximately 0.19–0.34 seconds of history to develop at the tested sample rates. Neighboring bands can still react to Hann-window leakage, especially for tones near a band boundary. See Apple's FFT example and windowing discussion.
The native probe's source setup, diagnostic instrumentation, generation commands, driver, and selected logs are retained in reproduction commands and source setup.
| Native AVPlayer result | Before | After |
|---|---|---|
| Normal/fast source continuity | Reset on every accepted callback | Initial reset only during continuous playback |
| Right 110 Hz, normal mode, first four bands | Approximately 0.82, 0.82, 0.82, 0.82 | First three approximately zero; band 3 approximately 0.94 |
| Left 55 Hz | Lower bars rise together | Band 2 approximately 0.96–1.00 |
| Slow-mode startup | No playing analysis frames | Frames every 0.1 seconds |
| Pause | No late playing frames | No late playing frames |
| Replace with silent video | Silent output | All playing frames silent in every mode |
The probe also seeks while playing and records the new source's output. Diagnostic printing in its copied tap callback is for reproducing values and timestamps; it is not production code and is not a real-time performance measurement.
Regression tests cover short 512/1024-frame input, 44.1/48/192 kHz, all response modes, independent left/right low tones, continuous asset ranges whose durations differ from PCM, real discontinuities, ring timestamp preservation, empty bands, and selection of the most recent samples. Existing suites exercise pause/resume, seeking, source invalidation during FFT, hidden visualizers, and stale publication. The new regressions were first run against the old analyzer/FFT/accumulator and failed on the expected conditions.
Final validation: 41 tests in ten related suites pass on macOS; the app builds for the shared iOS Simulator target. scripts/lint.sh --quiet succeeds with no diagnostics in the eight changed/new files and the baseline remains unchanged. Existing unrelated lint and deprecation warnings remain. Full Xcode stdout/stderr is retained locally under .build-logs/issue-89-*.log, with formatted summaries supplied through xcsift -f toon -w.
An optimized local FFT microbenchmark measured approximately 0.03 / 0.05 / 0.31 ms per stereo transform at 44.1 / 48 / 192 kHz, respectively. This measures only warmed FFT and band aggregation on the development Mac, not end-to-end callback latency or iPhone/iPad CPU cost.
The native playback probe ran on macOS. iPhone/iPad validation includes compilation of the shared iOS Simulator target; physical-device playback, Instruments real-time audio tracing, and screen-level LED frame-rate measurements were not performed. The probe proves the source-timeline and spectral-value regressions, not device-specific rendering deadlines.