Repository navigation
Video Realtime Audio Tap
Issue: #29. Fix: PR #43, commit 969f878. Investigation and validation: 2026-10-07, Xcode 27.1 Beta (27A9269), Swift 6, Apple silicon macOS.
VideoAudioTapProcessor.process constructed an AVAudioPCMBuffer wrapper and called SpectrumAnalyzer.analyze from the AVPlayer audio rendering callback. That analysis entry point takes an NSLock, checks system uptime, allocates stereo arrays, and submits a dispatch closure. Waiting for a lock owned by the UI or analysis queue can interrupt real-time audio delivery. The ordinary AVAudioEngine analysis entry point is retained for its existing caller.
Apple's MTAudioProcessingTap.h says that processing is a real-time operation and must avoid allocations and blocking calls. prepare provides the maximum frame count; each prepare/unprepare pair bounds the lifetime of buffers accessible to process. Noninterleaved and interleaved Float32 PCM must both be handled. Other PCM numeric types continue playing unchanged, without this EQ/visualization path.
- Allocate eight stereo slots in
prepare, each sized tomaxFrames; seven can be outstanding. The audio callback applies EQ directly to theAudioBufferList, copies PCM into a free slot, and publishes its index with release ordering. The single consumer acquires that index before copying the slot. The consumer releases its read index only after its copy is complete. A full ring drops visualization input and marks the next accepted buffer as discontinuous. Audio never waits for the consumer. - An 8 ms timer drains the ring on the same serial queue as spectrum analysis. This prevents a separate consumer from creating an unlimited backlog while FFT processing is delayed. Arrays, dispatch closures, FFT work, and MainActor publication are created outside the rendering callback.
- Accumulate short tap buffers into sample windows at the selected 10/30/60 Hz rate. Schedule windows within a large buffer at their playback offsets. Source discontinuities, transport generations, response-mode changes, and input gaps discard partial windows and obsolete scheduled work.
- Capture both the source epoch and analyzer playback generation before
MTAudioProcessingTapGetSourceAudio. Check them when consuming, processing scheduled windows, and publishing on MainActor. Pause/resume, stop, replacement, seek, and unprepare invalidate old samples. Resume during a pending seek keeps analysis disabled until the latest seek completes; superseded seek completions cannot reopen it. - EQ coefficients are immutable, allocated by the settings writer. A sequentially consistent hazard pointer protects the callback's snapshot. The callback publishes the pointer, checks that it still matches the current snapshot, then dereferences it. If publication races the check, that one buffer passes through EQ unchanged; no spin or lock is introduced. Reclaim old snapshots only on the writer, retaining at most the current and active-reader snapshots. Raw coefficient/state pointers avoid per-sample array uniqueness and exclusivity checks.
- Split the unchanged FFT resources, audio frame model, and smoothing constants into small source files so this work requires no SwiftLint baseline additions. No new platform API is required; Synchronization was already used by the video EQ and is available at both deployment targets.
The standalone probes use production source snapshots from main 6ef0b74 and the fix. The before EQ snapshot adds only a read-only test counter; it does not change retention or rendering. Visualization feeds 47 stereo 1024-frame buffers at 48 kHz, paced with 22 ms sleeps (about one second of source audio). The baseline calls the original analyzer entry point; the fix uses the production ring and consumer timer.
| Probe | Before | After |
|---|---|---|
| Retained EQ configurations after 10,000 settings updates | 10,002 | 1 |
| Slow-mode analysis frames from that source audio | 47 | 10 |
Probe reproduction commands, before results, and after results are retained with this Wiki. Thread Sanitizer's separate stress probe performs 20,000 concurrent EQ updates and 20,000 ring writes across wraparound. It verifies complete, monotonically ordered slots and the two-snapshot bound. No sanitizer race was reported. The accepted slot count is scheduling dependent; drops are intentional.
macOS validation: 26 tests in seven related suites pass, including VideoAudioSampleRingTests, VideoSpectrumAnalysisTests, VideoEqualizerRealtimeTests, existing EQ tests, SpectrumAnalyzerIdleTests, VideoPlaybackClockTests, and PlayerPlaybackGenerationTests. Tests cover mono/stereo and both PCM layouts, full/oversized/inactive rings, source ownership, delayed publication, 512/1024/4800-frame inputs in all response modes, and parallel settings updates. A generated tone played through the actual AVPlayer tap validates VU/spectrum output, mono channel duplication, pause/resume, seek during pause/resume and a superseding seek, stop, and close.
The final iOS Simulator build verifies the shared iPhone/iPad target. scripts/lint.sh --quiet passes with existing unrelated warnings and no warnings in changed/new files. The baseline is unchanged. Related tests also build the macOS app.
Two review findings on PR #43 were reproduced and fixed in 886f1eb:
- Analyzer state after invalidation during computation: rejecting only MainActor publication still let an expired loud chunk seed spectrum levels, held peaks, RMS levels/peaks, and their hold ages. FFT histories are now temporary until calculation finishes, with generation/source revalidation before application. Accepting a new source/window or active playback generation clears all history and meter state on the serial queue. This also prevents leakage if invalidation occurs just after the final check. Pause keeps the last accepted state for decay. Normal audio's slow path now uses the same guarded application code while preserving its whole-buffer FFT input.
- Recycled EQ snapshot addresses: the allocator can recycle a previously processed disabled snapshot's address after an unrendered intermediate update. The address comparison then skipped the disabled-to-enabled reset and injected the previous biquad tail. The rendered enabled state is now compared on every callback, independently of addresses. Address atomics remain solely for snapshot lifetime protection; no new locks, allocations, or retry loops were added to rendering.
Three new regression tests fail on 969f878 and pass with the fixes. The source tests use a loud-to-silent epoch transition and a validity predicate that expires after batch/chunk admission, asserting exact silence of both spectrum and all held meter peaks. The EQ test leaves a nonzero filter tail, renders a disabled snapshot, publishes an unrendered intermediate update, and re-enables EQ on zero input for 128 cycles. Before the fix each cycle leaks that tail. An additional audio regression covers silent resume in all three response modes, since both audio input paths had the same stale-state risk.
Final validation: 30 tests in seven related suites pass on macOS; the shared iPhone/iPad iOS Simulator target builds; lint has no violations in modified files and the baseline remains unchanged. The 20,000-update/write Thread Sanitizer stress probe was rerun against the reviewed source with no races reported. Device and Instruments verification limits below still apply.
No physical iPhone/iPad audio-dropout measurement, LED fps screen measurement, Instruments Audio System Trace, or real-time malloc/lock trace was performed. The probes prove the retention and emission-count regressions, and the native macOS test proves the tap/transport behavior; they do not establish device-specific render-thread timing or audible dropout frequency. A device follow-up should compare Time Profiler/Audio System Trace before and after while rapidly changing EQ, seeking, and pausing/resuming a video, then inspect the rendering callback for allocations and waits.
References: Apple tap callback, the Xcode 27.1 SDK's MediaToolbox/MTAudioProcessingTap.h, Swift release/acquire ordering, and Swift SE-0410 atomics.