Bug report: Ngram training in SLU

[rxpwang]

Describe the bug
ngram training stage in espnet/egs2/TEMPLATE/slu1/slu.sh

Current command in the script:
cut -f 2 -d " " ${data_feats}/lm_train.txt | lmplz -S "20%" --discount_fallback -o ${ngram_num} - >${ngram_exp}/${ngram_num}gram.arpa

This only take the first token in the transcripts for ngram training

Seems like should be following to take all the tokens in transcripts
cut -f 2- -d " " ${data_feats}/lm_train.txt | lmplz -S "20%" --discount_fallback -o ${ngram_num} - >${ngram_exp}/${ngram_num}gram.arpa

[sw005320]

Thanks for pointing it out, @rxpwang.
This is critical.
@siddhu001, can you check it?

[siddhu001]

@rxpwang Thank you for bringing this to my attention. I am currently reviewing the issue and will open a PR if necessary to address it as soon as possible.

[siddhu001]

Thanks @rxpwang for reporting this bug. I have created a PR https://github.com/espnet/espnet/pull/5364/files to fix this.