Stress Testing

Stress Testing

One speech (gettysburg, 21 items), each recording perturbed and re-analyzed. Run make stress; outputs in results/stress.md.

Variant	ok / insufficient / error	FP regions	Near-perfect FP	Recall@0.3 (pace / pause / pitch / energy)	Messages
clean	21 / 0 / 0	1	0	1.00 / 1.00 / 0.75 / 1.00	approximate_alignment
noise 30 dB SNR	21 / 0 / 0	5	0	1.00 / 0.80 / 0.75 / 1.00	approximate_alignment
noise 20 dB SNR	21 / 0 / 0	42	2	0.57 / 0.40 / 0.75 / 0.67	approximate_alignment
noise 10 dB SNR	21 / 0 / 0	27	1	0.71 / 0.60 / 1.00 / 0.67	approximate_alignment, noisy_audio
MP3 48 kbps → 44.1 kHz	21 / 0 / 0	1	0	1.00 / 1.00 / 0.75 / 1.00	approximate_alignment
3 s added lead/trail silence	21 / 0 / 0	1	0	1.00 / 1.00 / 0.75 / 1.00	approximate_alignment

(The silence variant maps predicted times back by 3 s before scoring.)

Findings

• Robust to lossy compression, resampling and added silence.
• Not robust to additive noise. Spurious pauses and energy changes appear at 20 dB SNR.
• The SNR estimate is too optimistic. It only warns at about 10 dB and never withheld a result, so noisy recordings still got scores.
• At 10 dB the results look better than at 20 dB for some dimensions; with so few items, treat that as noise in the measurement, not a trend.

Planned fixes

1. Noise-aware speech/pause detection (adaptive threshold, minimum pause evidence).
2. A better SNR estimate with a hard insufficient_evidence cut-off.
3. Extend stress tests to more speakers, reverb and codec types, and more speeches.