CFSD, SECS metrics for TTS

[imdanboy]

Hi, I tried to implement following TTS objective metrics (#1665 ) by simply utilizing related espnet script and the pretrained models.

1. Conditional Frechet Speech Distance (CFSD)
2. Speaker Embedding Cosine Similarity (SECS)

(Example TTS paper where those metrics are used: ADAPTER-BASED EXTENSION OF MULTI-SPEAKER TEXT-TO-SPEECH MODEL FOR NEW SPEAKERS)

Example results on LJSPEECH eval set:

	CFSD	SECS	MCD	logF0 RMSE
JETS	6.6239 ± 2.4985	0.8465 ± 0.0534	6.6817 ± 0.5373	0.2905 ± 0.0682
VITS	6.8165 ± 1.6455	0.8440 ± 0.0585	6.9935 ± 0.6215	0.2812 ± 0.0589
joint_finetune	6.1623 ± 1.4250	0.8481 ± 0.0564	6.7344 ± 0.6166	0.2839 ± 0.0739

How about this?

[codecov]

Codecov Report

Merging #5235 (52220f6) into master (2839fb7) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5235   +/-   ##
=======================================
  Coverage   74.43%   74.43%           
=======================================
  Files         642      642           
  Lines       57611    57611           
=======================================
  Hits        42885    42885           
  Misses      14726    14726           

Flag	Coverage Δ
test_integration_espnet1	66.28% <ø> (+<0.01%)	⬆️
test_integration_espnet2	47.52% <ø> (ø)
test_python	65.15% <ø> (ø)
test_utils	23.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[kan-bayashi]

Very cool!
Could you update the doc (adding example command)?
https://github.com/espnet/espnet/tree/master/egs2/TEMPLATE/tts1#evaluation

[imdanboy]

@kan-bayashi
Sure, I updated the doc!

[sw005320]

Can you add a test script (in later PR?)?
We can add some consistency checks as well (e.g., same waveforms yields the zero distance, check to see that the case with the numerical issue is correctly handled).
The evaluation script should not include any errors.

[imdanboy]

@sw005320
Ok, I can add a test script in later PR 😄

[sw005320]

Thanks!