Commonvoice results misleading, complete overlap of train/dev/test sentences

[bmilde]

I was quite surprised to see how low the WERs are for the new Common Voice corpus: https://github.com/kaldi-asr/kaldi/blob/master/egs/commonvoice/s5/RESULTS (4ish% TDNN)

Unfortunately, these results seem to be bogus because there is a near complete overlap of train/dev/test sentences and the LM is only trained on the corpus train sentences (https://github.com/kaldi-asr/kaldi/blob/master/egs/commonvoice/s5/local/prepare_lm.sh). To make matters worse, there aren't really that many unique sentences in the corpus:

unique sentences in train: 6994
unique sentences in dev: 2410
unique sentences in test: 2362
common sentences train/dev (overlap) = 2401
common sentences train/test (overlap) = 2355

This can also be easily verified by e.g. grepping "sadly my dream of becoming a squirrel whisperer may never happen" on the original corpus csvs:

cv-valid-dev.csv:cv-valid-dev/sample-000070.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,seventies,male,us,
cv-valid-dev.csv:cv-valid-dev/sample-000299.mp3,sadly my dream of becoming a squirrel whisperer may never happen,5,2,twenties,female,canada,
cv-valid-dev.csv:cv-valid-dev/sample-002458.mp3,sadly my dream of becoming a squirrel whisperer may never happen,9,1,,,,
cv-valid-dev.csv:cv-valid-dev/sample-003264.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
cv-valid-dev.csv:cv-valid-dev/sample-003656.mp3,sadly my dream of becoming a squirrel whisperer may never happen,2,1,,,,
grep: cv-valid-test: Is a directory
cv-valid-test.csv:cv-valid-test/sample-000221.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,thirties,male,canada,
cv-valid-test.csv:cv-valid-test/sample-001576.mp3,sadly my dream of becoming a squirrel whisperer may never happen,2,1,,,,
cv-valid-test.csv:cv-valid-test/sample-002831.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
cv-valid-test.csv:cv-valid-test/sample-003705.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
cv-valid-test.csv:cv-valid-test/sample-003789.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
grep: cv-valid-train: Is a directory
cv-valid-train.csv:cv-valid-train/sample-000324.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,2,,,,
cv-valid-train.csv:cv-valid-train/sample-000373.mp3,sadly my dream of becoming a squirrel whisperer may never happen,5,1,,,,
cv-valid-train.csv:cv-valid-train/sample-000382.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
cv-valid-train.csv:cv-valid-train/sample-001026.mp3,sadly my dream of becoming a squirrel whisperer may never happen,4,0,,,,
cv-valid-train.csv:cv-valid-train/sample-003106.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,fourties,female,england,
cv-valid-train.csv:cv-valid-train/sample-004591.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
cv-valid-train.csv:cv-valid-train/sample-005048.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
cv-valid-train.csv:cv-valid-train/sample-007144.mp3,sadly my dream of becoming a squirrel whisperer may never happen,3,0,,,,
+ 100s more...

Now this is pretty much a terrible design for a speech corpus, but I suggest to exclude the train sentences from the LM completely, to have somewhat more realistic results. I'm currently rerunning the scripts with a Cantab LM without the train sentences and will report back when I have the results.

[entn-at]

Well, that's the way the corpus was designed by the people at Mozilla (including the overlap in spoken content in the predefined train/dev/test splits). You should address your concerns to the people running this data collection effort (https://voice.mozilla.org/data, https://github.com/mozilla/voice-web). I believe Voxforge is similar in that there is an overlap in prompts.

This recipe is just using the corpus in its intended way (using the train/dev/test splits as provided). The results are not misleading, and to be sure, nobody is claiming that test results on this corpus somehow generalize to any other datasets/conditions.

EDIT: This isn't really a Kaldi problem; I suggest moving the discussion to kaldi-help: https://groups.google.com/forum/#!forum/kaldi-help

[bmilde]

Thanks for your fast reply! Also, I see that you (@entn-at) wrote the scripts for this corpus. Thank you very much for adapting them so fast!

I wrote similar scripts for Eesen (Rnn-ctc) and was puzzled at the WER difference (~16% vs ~4%), but I used Kaldi-lm and it doesn't let you finish training if it discovers the training sentences in dev, thus my difference was that I omitted them in LM training.

Despite this not being a Kaldi problem, I'd still suggest the following enhancements to the Kaldi Common Voice recipe:

• Enhance the recipe by adding a true LM (e.g. trained on Cantab) so that the models can be used on other phrases then these 6000 sentences. This will make the models useable in speech recognition tasks (what Mozilla actually intended) despite the corpus problem. Right now you probably wouldn't be able to decode much else besides these exact sentences and tuning the recipe is pointless since the best results are probably obtained with a large LM/FST prior.
• Add a comment about this peculiarity to the scripts and/or results
• Maybe an option to train without the train sentences in the LM to get better generalization results. I'm currently doing this and the GMM-HMM WERs are basically 2x as large.

Ultimately, I agree that this is a corpus problem. I will address my concerns on the mozilla git and link to this Issue (would be nice to keep the discussion here open as well) - this is surely not what they had in mind.

[jtrmal]

I would suggest to do two different decoding passes probably. If the corpus was designed a specific way then for comparability there should be an easy way to get the reference numbers. I have certain doubts about the utility or usefulness of the Mozilla corpora anyway. Y. On Jan 10, 2018 1:20 PM, "bmilde" <notifications@github.com> wrote: Thanks for your fast reply! Also, I see that you (@entn-at <https://github.com/entn-at>) wrote the scripts for this corpus. Thank you very much for adapting them so fast! I wrote similar scripts for Eesen (Rnn-ctc) and was puzzled at the WER difference (~16% vs ~4%), but I used Kaldi-lm and it doesn't let you finish training if it discovers the training sentences in dev, thus my difference was that I omitted them in LM training. Despite this not being a Kaldi problem, I'd still suggest the following enhancements to the Kaldi Common Voice recipe: - Enhance the recipe by adding a true LM (e.g. trained on Cantab) so that the models can be used on other phrases then these 6000 sentences. This will make the models useable in speech recognition tasks (what Mozilla actually intended) despite the corpus problem. Right now you probably wouldn't be able to decode much else besides these exact sentences and tuning the recipe is pointless since the best results are probably obtained with a large LM/FST prior. - Add a comment about this peculiarity to the scripts and/or results - Maybe an option to train without the train sentences in the LM to get better generalization results. I'm currently doing this and the GMM-HMM WERs are basically 2x as large. Ultimately, I agree that this is a corpus problem. I will address my concerns on the mozilla git and link to this Issue (would be nice to keep the discussion here open as well) - this is surely not what they had in mind. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKisX4ndJmXpCR2anqXsqnH0qH3xxXIXks5tJP6YgaJpZM4RZojk> .

[entn-at]

The recipe in its current form was more of a research exercise and not intended to produce a set of models fit for more general use/distribution (like the models published at http://www.kaldi-asr.org/models.html). I fully agree that your suggestions would make this recipe more valuable to other users and would be more in line with what the people at Mozilla had intended. With @danpovey's and @jtrmal's approval, we could create a second version (s5b?) of the recipe that includes your proposed changes. In that case, I would like to encourage you to make PRs with your changes using the Cantab LM.

Ultimately, though, I somewhat share @jtrmal's doubts regarding the corpus (with its current design). The people working on Mozilla's DeepSpeech implementation seem to be using various corpora for model training (including non-free corpora like Fisher, SWBD, see their import scripts). Perhaps a subset of the CV corpus could be added to egs/multi_en?

[danpovey]

I don't have super strong opinions about the structure... s5b would be a reasonable approach, I guess. kaldi_lm may crash if in the metaparameter optimization it detects that your dev data was likely taken from the training data. The way this works, it likely wouldn't crash if even only small proportion of the dev data was distinct from training. Likely this means that there is near 100% overlap.

…

On Wed, Jan 10, 2018 at 11:17 AM, Ewald Enzinger ***@***.***> wrote: The recipe in its current form was more of a research exercise and not intended to produce a set of models fit for more general use/distribution (like the models published at http://www.kaldi-asr.org/models.html). I fully agree that your suggestions would make this recipe more valuable to other users and would be more in line with what the people at Mozilla had intended. With @danpovey <https://github.com/danpovey>'s and @jtrmal <https://github.com/jtrmal>'s approval, we could create a second version (s5b?) of the recipe that includes your proposed changes. In that case, I would like to encourage you to make PRs with your changes using the Cantab LM. Ultimately, though, I somewhat share @jtrmal <https://github.com/jtrmal>'s doubts regarding the corpus (with its current design). The people working on Mozilla's DeepSpeech implementation seem to be using various corpora for model training (including non-free corpora like Fisher, SWBD, see their import scripts). Perhaps the CV corpus could be added to egs/multi_en? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu0YWZMIHgZh0rsfYJLt137Gy4rq4ks5tJQzTgaJpZM4RZojk> .

[bmilde]

Exactly, the overlap is 99.6%. Kaldi_lm rightfully refuses to work on that, I think it even gracefully showed an error message.

I've also posted the problem here on the mozilla discourse platform: https://discourse.mozilla.org/t/common-voice-v1-corpus-design-problems-overlapping-train-test-dev-sentences/24288

Chance are, they aren't really aware of it.

[jtrmal]

My preference would just adding another decoding script into s5 (and document that in README) to reduce duplication of things. But it's not worth my time to argue about this, so Ewald, your call. y.

…

On Thu, Jan 11, 2018 at 8:48 AM, bmilde ***@***.***> wrote: Exactly, the overlap is 99.6%. Kaldi_lm rightfully refuses to work on that, I think it even gracefully showed an error message. I've also posted the problem here on the mozilla discourse platform: https://discourse.mozilla.org/t/common-voice-v1-corpus- design-problems-overlapping-train-test-dev-sentences/24288 Chance are, they aren't really aware of it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKisX0c3xv4KpSyGa8Afz0Pgw9YfcRZ4ks5tJhEqgaJpZM4RZojk> .

[jtrmal]

I'm not gonna discuss if kaldi_lm is rightful or not, but the usual discounting methods (KN, GT) have issues with "artificially" looking data, often artificially generated or grammar-induced data. Witten-Bell discounting should work in that case (probably not implemented in kaldi_lm). y.

…

On Thu, Jan 11, 2018 at 8:48 AM, bmilde ***@***.***> wrote: Exactly, the overlap is 99.6%. Kaldi_lm rightfully refuses to work on that, I think it even gracefully showed an error message. I've also posted the problem here on the mozilla discourse platform: https://discourse.mozilla.org/t/common-voice-v1-corpus- design-problems-overlapping-train-test-dev-sentences/24288 Chance are, they aren't really aware of it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKisX0c3xv4KpSyGa8Afz0Pgw9YfcRZ4ks5tJhEqgaJpZM4RZojk> .

[mikehenrty]

Hi Kaldi folks, I'm the maintainer of the Common Voice project, and as such am responsible for both our small text corpus and improper split.

First of all, let me say that it was great to see Common Voice data integrated into the Kaldi project so quickly. Also, big thanks to @bmilde for finding and reporting this bug, and for suggesting a way to fix this on our repo. For a discussion of our plan for addressing this, you can check out that bug.

One thing I noticed in this thread:

I have certain doubts about the utility or usefulness of the Mozilla corpora anyway.

Both @jtrmal and @entn-at seemed to have this sentiment (perhaps others?). Are the problems other than our small corpus size and train/dev/test split that you are concerned about? We are in the process of updating Common Voice now, and it would be super helpful to get your feedback so we can make our data more useful to your project (and other STT engines).

On the topic of text corpora, we are working with the University of Illinois to get them to release the Flickr30K text corpus under cc-0, which would allow us to use it's 100K some sentences. For an example of what these look like, you can see here:
https://raw.githubusercontent.com/mozilla/voice-web/master/server/data/not-used/flickr30k.txt

If it's not too much to ask, we would love expert feedback on if the above corpus would be helpful for speech engines. You'll notice the sentences are all description of images, so the utterances have a lot of repeated words. Unfortunately, I am not a speech technologist, so I don't have any intuition as to whether this is the right data or not for utterances. Again, any info you could provide would be very helpful.

Thanks again, and thanks for maintaining Kaldi!

[ognjentodic]

Here are some suggestions:

• use wav files (16kHz, 16bits), not mp3
• if possible, disable OS audio feedback in the app that collects the data; in a number of recordings I've noticed that there were different types of beeps in the beginning; later on I realized this may have been due to OS audio feedback, eg. when tapping a button, on Android
• keep speaker information (some sort of hash), across different recordings
• less focus on reading, more on spontaneous speech (this will end up being more costly, since speech data will need to be transcribed); you can ask people to talk about a number of different topics... have two people talk to each other, or person talk to a bot; you can also take a look at LDC website and get some inspiration from datasets collected in the past (I would not necessarily blindly follow their approaches though)
• additional metadata that could be useful: tag/labels for extraneous speech/noise, etc; device info; user demographics; headset/bluetooth
• capture data in different acoustic environments (and when possible, capture metadata about the environment as well)

[sikoried]

As Ognjen said, unless you're looking for models to be used in a command/control scenario (like Echo or Home), but a more general model, you'd be looking for spontaneous speech, ideally among people (of balanced age and gender). The hard part is transcribing it, since badly (or inconsistently) transcribed data is not of much use (which is also the reason why well transcribed data is so expensive). Here's a comparable real life analogy: say you're learning a new language; you may understand the news on TV (read-out, professional speakers), but have no clue when somebody talks to you on the street (spontaneous, slang, accent, background noise, ...). Korbinian.

…

On Fri, Jan 26, 2018 at 6:59 PM, Ognjen Todic ***@***.***> wrote: Here are some suggestions: - use wav files (16kHz, 16bits), not mp3 - if possible, disable OS audio feedback in the app that collects the data; in a number of recordings I've noticed that there were different types of beeps in the beginning; later on I realized this may have been due to OS audio feedback, eg. when tapping a button, on Android - keep speaker information (some sort of hash), across different recordings - less focus on reading, more on spontaneous speech (this will end up being more costly, since speech data will need to be transcribed); you can ask people to talk about a number of different topics... have two people talk to each other, or person talk to a bot; you can also take a look at LDC website and get some inspiration from datasets collected in the past (I would not necessarily blindly follow their approaches though) - additional metadata that could be useful: tag/labels for extraneous speech/noise, etc; device info; user demographics; headset/bluetooth - capture data in different acoustic environments (and when possible, capture metadata about the environment as well) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2141 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADhueJhnAdRvAmvl5j6OMMbOAIxDjcSZks5tOhJ7gaJpZM4RZojk> .

[vdp]

I completely agree with Ognjen's and Korbinian's suggestions, especially the one about conversational speech. This is indeed the major missing piece when it comes to open speech corpora. At this point collecting more scripted English is not going to be very useful IMO. Except maybe for distant speech or speech in noisy environments, but I'm not familiar with these domains- perhaps someone who is researching robust recognition will chime in. I'd say for clean read English speech the data problem is pretty much solved. For example the LibriSpeech's results for non-accented English, i.e. {dev,test}-clean, are already fairly close to the theoretical minimum- keep in mind that some of those errors are actually due to different spellings of some names etc.
As it was already pointed out a conversational corpus is going to be much more costly in terms of corpus design and data collection effort, as well as transcription fees, but in contrast to read speech it's going to be very useful. Aside from the cost conversational speech is hard to come by as it's rarely released in public, because it tends to be private in nature.

If English conversational speech is not an option, then I would suggest to at least concentrate on languages other than English. The coverage there is more than spotty, so even read speech is going to be useful.

[entn-at]

The prompted nature of the collected speech was my main concern as well. Perhaps you can collect transcribed conversational speech via crowd sourcing as follows (I'm posting some ideas here and explicitly want to invite constructive criticism/comments by everybody; I know this has nothing to do with Kaldi per se, but people here have a lot of experience with data collected for ASR and can give valuable advice to @mikehenrty):

• To collect conversational speech, let users call other users via WebRTC. Conversation will be easier if users already know each other. There are ways of recording a WebRTC call, for example using RTCMultiConnection and RecordRTC. AFAIK, WebRTC gateways like the open-source Janus project have plugins for recording calls. Each caller needs to be informed that the call is being recorded (and for what purpose), of course. There are obvious privacy issues for releasing the data and crowd sourcing the transcription effort, and reminders not to disclose private information only go so far.
• For word-level transcription, use existing LVCSR systems (e.g. based on Kaldi, Mozilla DeepSpeech, or cloud speech APIs) to segment calls (i.e., create relatively short chunks) and transcribe them. Human listeners verifying these automatically transcribed segments could listen to segments and provide feedback in multiple ways:
  1. Transcription correct: yes/no. This is the least-effort feedback option.
  2. Add buttons (or click-/tap-able areas) for each word and between words. Users can click/tap on words to indicate insertion/substitution or between words to indicate deletions. More effort required by the listener.
  3. In addition to (ii.), let users correct incorrectly transcribed segments/enter fully manual transcription. Higher effort required.

Keeping the required effort low is important for crowd sourcing, but as has been pointed out, badly or inconsistently transcribed speech isn't that useful. There is no question that a professionally designed and transcribed speech corpus is preferable in every way, but I'd be interested in feedback/suggestions (also and especially in the form of "This won't work because...").

Disclaimer: I'm not involved in this data collection effort, I'm just trying to be helpful.