Bad inference result.

[leoniloris]

Hello, I'm trying to reproduce this issue #308 using the same audio but I'm still receiving Gibberish (ish) inferences.

Since I could not find any information on which model and which command they were using in the issue, I'm posting here the info I'm using:

# Download test audio and resample (sampling rate) to 16k

deepspeech.pytorch# wget https://dare.wiscweb.wisc.edu/wp-content/uploads/sites/1051/2008/04/Arthur.mp3
deepspeech.pytorch# sox Arthur.mp3 -c 1 -r 16000 arthur_clip.wav trim 0 15

# Running the inference on the audio clip
deepspeech.pytorch# python transcribe.py --model-path librispeech_pretrained_v2.pth --audio-path arthur_clip.wav --lm-path 3-gram.pruned.3e-7.arpa --alpha 1.65 --beta 0.35

>>>
{
    "output": [
        {
            "transcription": "THE STARY OF OWT OF THE WRAPTH ONCE UPON A TIME THERE WAS A YOUNG RAG AND CUTD IN MYE GUFF ERS MOINE WHENEVER THE HAD THE RIHT SAYES HIM IF HE WOULD LIKE TO COME OUT HUNTING BOT THEM HE WHEN ANSWER IN A HORSE"
        }
    ],
    "_meta": {
        "acoustic_model": {
            "name": "librispeech_pretrained_v2.pth"
        },
        "language_model": {
            "name": "3-gram.pruned.3e-7.arpa"
        },
        "decoder": {
            "lm": true,
            "alpha": 1.65,
            "beta": 0.35,
            "type": "greedy"
        }
    }
}

I'm using the latest release (v2) as well as its respective commit.

As you can see, those are fairly different results from the one @ryanleary got in aforementioned comment

I tested several different configurations with the different models and both 3-gram.pruned.3e-7.arpa and 3-gram.3e-7.arpa as arguments for the transcribe.py script but in every case I got weird results with those uppercase characters and random words.

Am I doing something wrong here?

[leoniloris]

Edit:

Adding the --decoder beam argument makes the LM to be loaded (whereas otherwise it just is not loaded silently)

but still, the predictions are not good, when compared to the ones @ryanleary got in the aforementioned issue.

deepspeech.pytorch# python transcribe.py --model-path librispeech_pretrained_v2.pth --audio-path arthur_clip.wav --lm-path 3-gram.pruned.1e-7.arpa --alpha 1.65 --beta 0.35 --decoder beam

Loading the LM will be faster if you build a binary file.
Reading 3-gram.pruned.1e-7.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
   {
    "output": [
        {
            "transcription": "THE STORY OF OUT OF THE WRATH ONCE UPON A TIME THERE WAS A YOUNG RAG AND CUT IN MYGALE WHENEVER THE HATHERSAGE HIM IF HE WOULD LIKE TO COME OUT HUNTING BOTHAM HE WHEN ANSWER IN A HORSE"
        }
    ],
    "_meta": {
        "acoustic_model": {
            "name": "librispeech_pretrained_v2.pth"
        },
        "language_model": {
            "name": "3-gram.pruned.1e-7.arpa"
        },
        "decoder": {
            "lm": true,
            "alpha": 1.65,
            "beta": 0.35,
            "type": "beam"
        }
    }
}

[SeanNaren]

Could you try specifying: alpha=1.97, beta=4.36, beam-width=1024?

[leoniloris]

The same as the previous ones @SeanNaren .
I put all the commands I ran in the following log,

deepspeech.pytorch# git checkout 9b9c96a16ffb0ebf05a9aa899ed8e0e14fdb0349 # lattest release
deepspeech.pytorch# wget https://github.com/SeanNaren/deepspeech.pytorch/releases/download/v2.0/librispeech_pretrained_v2.pth
deepspeech.pytorch# wget http://www.openslr.org/resources/11/3-gram.pruned.1e-7.arpa.gz # get language model
deepspeech.pytorch# gunzip 3-gram.pruned.1e-7.arpa.gz
deepspeech.pytorch# python transcribe.py --model-path librispeech_pretrained_v2.pth --audio-path ../arthur_clip.wav --lm-path 3-gram.pruned.1e-7.arpa --alpha 1.97 --beta 4.36 --beam-width 1024 --decoder beam

Loading the LM will be faster if you build a binary file.
Reading 3-gram.pruned.1e-7.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************

{
    "output": [
        {
            "transcription": "THE STORY OF OUT OF THE WRATH ONCE UPON A TIME THERE WAS A YOUNG RAG AND PUT IN MY OFFERS MOINE WHENEVER THEY HAD THE RIGHT SAYS HIM IF HE WOULD LIKE TO COME OUT HUNTING TO THEM HE WHEN ANSWER IN A HORSE"
        }
    ],
    "_meta": {
        "acoustic_model": {
            "name": "librispeech_pretrained_v2.pth"
        },
        "language_model": {
            "name": "3-gram.pruned.1e-7.arpa"
        },
        "decoder": {
            "lm": true,
            "alpha": 1.97,
            "beta": 4.36,
            "type": "beam"
        }
    }
}

[leoniloris]

@SeanNaren
Just tried with another language model and the results are the same

deepspeech.pytorch# python transcribe.py --model-path librispeech_pretrained_v2.pth --audio-path ../arthur_clip.wav --lm-path 3-gram.pruned.3e-7.arpa --alpha 1.97 --beta 4.36 --beam-width 1024 --decoder beam
Loading the LM will be faster if you build a binary file.
Reading 3-gram.pruned.3e-7.arpa
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
{
    "output": [
        {
            "transcription": "THE STORY OF OUT OF THE WRATH ONCE UPON A TIME THERE WAS A YOUNG RAG AND PUT IN MY COFFERS MOINE WHENEVER THEY HAD THE RIGHT SAYS HIM IF HE WOULD LIKE TO COME OUT HUNTING TO THEM HE WHEN ANSWER IN A HORSE"
        }
    ],
    "_meta": {
        "acoustic_model": {
            "name": "librispeech_pretrained_v2.pth"
        },
        "language_model": {
            "name": "3-gram.pruned.3e-7.arpa"
        },
        "decoder": {
            "lm": true,
            "alpha": 1.97,
            "beta": 4.36,
            "type": "beam"
        }
    }
}

[leoniloris]

Any news on that? @SeanNaren @ryanleary

[leoniloris]

@SeanNaren @ryanleary
Could you guys reproduce?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[SeanNaren]

My bad, aggressive stale bot... will try to get time to reproduce/investigate

[SeanNaren]

I reproduced results and got output that isn't as appealing as the original comment I must say... However it's hard to judge quality from just one file (should check WER across a set of data to get a more representative number).

Firstly the model has changed since ryan ran those, and I'm not sure if we've updated the alpha/beta weights but it might be wise to follow this again to see if there are any additional gains to be made!