Benchmarking

Benchmark Types

1. Compare Timing

Measures end-to-end per-frame analysis performance to identify bottlenecks and optimize settings.

What It Measures:

• Mean latency per frame (milliseconds)
• Frame processing rate (frames per second)
• Total analysis time vs. audio duration
• Method-specific overhead

Output Files:

• benchmarking/timing_comparison.csv: Raw timing data for each frame
• benchmarking/timing_report.md: Summary statistics and comparison

Duration: ~10 seconds of recording

Use This When:

• Optimizing for real-time performance
• Choosing between analysis methods
• Debugging performance issues
• Testing on different hardware

2. Compare Measurements

Measures accuracy of formant and pitch extraction by comparing methods against Parselmouth as a reference standard.

What It Measures:

• Number of frames analyzed (total count for each analysis)
• Formant correlation (F1, F2, F3)
• Formant errors: RMSE, MAE, bias
• Pitch (f0) correlation
• Pitch errors: RMSE, MAE, bias
• Frame-by-frame agreement
• Per-method timing statistics with 95% confidence intervals (computed from 2.5th and 97.5th percentiles)

Output Files:

• benchmarking/formant_raw_comparison.csv: Per-frame formant comparison
• benchmarking/pitch_raw_comparison.csv: Per-frame pitch comparison
• benchmarking/comprehensive_benchmark.md: Statistical summary with correlations, error metrics, frame counts, and timing confidence intervals

Duration: ~15 seconds of recording

Reference Standard: Praat/Parselmouth is used as the gold standard for comparison

Use This When:

• Validating formant/pitch accuracy
• Comparing custom vs. built-in methods
• Determining if analysis settings are appropriate
• Publishing research requiring method validation