Speech MOS scores drastically higher than before

[nschimme]

Nice project! I switched to your pure Python solution and it works great for audio. There is an issue for speech however, you can see in my benchmark here it compares this library to the original C++ VisQOL and speech often gets significantly higher scores with deltas of 1-2 MOS. My benchmark uses the lattice_tcditugenmeetpackhref_ls2_nl60_lr12_bs2048_learn.005_ep2400_train1_7_raw.tflite model.

https://github.com/nschimme/faac/actions/runs/24147298369?pr=147

The dataset that is misbehaving is: https://github.com/nschimme/TCD-VOIP

[nschimme]

[talker93]

Hi @nschimme — thanks for the detailed report, this turned out to be a really useful catch. Released a fix in v3.5.0 (GitHub release).

Root cause had two layers:

1. Speech mode was only ever running the legacy polynomial exponential mapper (equivalent to C++ --use_lattice_model=false). The C++ default --use_lattice_model=true routes through TFLiteQualityMapper loading lattice_tcditugenmeetpackhref_*.tflite, which I had never ported. v3.5.0 adds TFLiteSpeechQualityMapper that runs inference through the same upstream Google TFLite C++ runtime via ai-edge-litert.

2. Once the lattice mapper was in place I still saw ~0.086 MOS drift vs the C++ binary on CA01, much bigger than the polynomial path's ~0.007. Three upstream Python parity bugs were amplifying through the lattice network:

   • signal_utils.normalize() was doing min-max scaling to [0, 1] instead of C++'s peak-only division to [-1, +1] → corrupted the speech-mode VAD chunk RMS and added spurious patches.
   • nsim.py and the Numba kernel used population stddev (divide by N) for freq_band_stddevs; C++ uses arma::stddev(..., 0) which is the unbiased sample stddev (divide by N-1).
   • _gammatone_spectrogram_numba had fastmath=True; LLVM FP reassociation across 4 cascaded IIR stages × thousands of samples compounded into another ~0.02 MOS drift.

After all four fixes, CA01 conformance against the C++ ground-truth (./bazel-bin/visqol --use_speech_mode) is:

Mode	Python v3.5.0	C++ baseline	Δ
Polynomial	3.3756	3.3745	+0.0011
Lattice (default)	3.3153	3.3130	+0.0023

Install to get the lattice path (matches C++ default):

pip install -U visqol-python[lattice]

The [lattice] extra is opt-in (ai-edge-litert is a binary wheel, only available on Python ≥ 3.10). Without it, speech mode still works but logs a one-time warning and falls back to polynomial.

Would you mind rerunning your TCD-VOIP benchmark on v3.5.0? CA01 is the only speech testdata pair shipped with the upstream google/visqol repo, so I'd love to confirm the parity holds on a real benchmark before closing this. If anything is still > 0.05 MOS off, please reopen — happy to dig in further.

Thanks again for the careful report!