Nemo ASR french

[Ning107]

Hello, everyone. I’m using the French checkpoint “stt_fr_quartznet15x5” on the dev set from Fr MCV dataset version 6.1 to check the result. Instead of a WER of about 14% presented in the NGC, I got a WER of about 38%. Below is my code, I’m not sure if I forget something else important to add.
Datasets : All audios files were transformed to the .wav format and 16000 sample rate. And I created with success the manifest file in format {"audio_filepath": "dev/wav_16k/common_voice_fr_19683978.wav", "duration": 9.0, "text": "de même selon la littérature les patients ont souvent moins de symptômes systémiques"}

import nemo
import nemo.collections.asr as nemo_asr
try:
    from ruamel.yaml import YAML
except ModuleNotFoundError:
    from ruamel_yaml import YAML
config_path = './configs/config.yaml'

yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)
print(params)

model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="stt_fr_quartznet15x5")
params['model']['validation_ds']['batch_size'] = 16
params['model']['validation_ds']['manifest_filepath'] = "test_manifest_16k.json"
model.setup_test_data(test_data_config=params['model']['validation_ds'])
model.cuda(1)

# Computing WER
wer_nums = []
wer_denoms = []

# Loop over all test batches
for test_batch in tqdm(model.test_dataloader()):
    test_batch = [x.cuda(1) for x in test_batch]
    targets = test_batch[2]
    targets_lengths = test_batch[3]
    log_probs,encoded_len,greedy_predictions = model(
        input_signal=test_batch[0],input_signal_length=test_batch[1]
    )
    model._wer.update(greedy_predictions,targets,targets_lengths)
    _, wer_num, wer_denom = model._wer.compute()
    wer_nums.append(wer_num.detach().cpu().numpy())
    wer_denoms.append(wer_denom.detach().cpu().numpy())
    
# We need to sum all numerators and denominators first. Then divide.
print(f"WER = {str(sum(wer_nums)/sum(wer_denoms))}")

WER = 0.38733165183457857

[titu1994]

You have to explicitly call model._wer.reset() after the compute() step

[okuchaiev]

also, the NGC numbers are from MCV 6.0 dev, not test set.

[okuchaiev]

another thing to check - do the transcripts in the .json which you prepared have only characters from model.vocabulary?

[Ning107]

Thanks for replying. In fact, the dataset that I used is MCV 6.1 dev ( 6.0 is not available ). And transcripts are pre-processed: removed punctuation and converted to lowercase. So basically, transcripts' characters correspond to the model.vocabulary.

[Ning107]

Hi there, thanks a lot. I found that I need to add the parameter normalize_transcripts as False in the test_data_config for French like blew. And now I can get a WER about 18%. So I have a another question : If I want to fine-tune this model with my own French dataset, do I also need modifer some parameter like this to get a better model ?

model.setup_test_data(
    test_data_config={
        'sample_rate': 16000,
        'manifest_filepath': "manifest/test_mini_non_num_manifest.json",
        'labels': model.decoder.vocabulary,
        'batch_size': 4,
        'normalize_transcripts': False,
    }
)

Thanks in advance for your help !

[titu1994]

This parameter should be False by default in any non-english language. Seems the yaml config you loaded was from the baseline English config which sets it to True. Will add a note in the new tutorial as well, so thank you for finding this !

As to your question - yes, this should be True only for English transcripts. All other languages should have it set to False.