KWS with CTCloss training and CTC prefix beam search detection.

[duj12]

This PR is for KWS training with CTC loss and detection with CTC prefix beam search.
Aim to improve the robustness of KWS model, and support customized keywords with limited data.
For now only hi_xiaowen data has runing scripts.

I redo the experiment of ds_tcn with max-pooling loss.
Then I add the ds_tcn model with CTC loss, and add a FSMN backbone.
Finally I add a streaming scoring script, to simulate the real detection case of a CTC model.
All results can be found in README.md.

Note, CTC model can be export to onnx(the output is softmax of logits), but runtime is not support now,
I decide to develop a python script in runtime streaming fashion(do online feature extraction and so on) first, and then the onnx c++ and pybind...
When runtime is ready, I will create a new PR.

[mlxu995]

good job! Thank you so much for this valuable update, and we will appreciate if you could help fix the lint errors such as the trailing whitespace.

[duj12]

Here is a Demo, all models are trained with ctc loss, and in this demo, the detection is performed in streaming fashion.
https://www.modelscope.cn/studios/thuduj12/KWS_Nihao_Xiaojing/summary

[jingyonghou]

Thanks for your constructive contribution to this project. Could you please make some revisions to the code, following the flake8 format check? [image: image.png] Jean Du ***@***.***> 于2023年6月28日周三 18:55写道：

…

Here is a Demo, all models are trained with ctc loss, and in this demo, the detection is performed in streaming fashion. https://www.modelscope.cn/studios/thuduj12/KWS_Nihao_Xiaojing/summary — Reply to this email directly, view it on GitHub <#135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEA4OP3XREIL4WQVGRQVGDDXNQEQRANCNFSM6AAAAAAYZD3MQ4> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Best regards, Jingyong Northwestern Polytechnical University Phone: +1 425 394 3659, +86 18392375265 Wechat: zaixialalala

[jingyonghou]

it better to replace Chinese char with Latin char in the code

[duj12]

Has been done in the latest commit.

[jingyonghou]

it better to replace Chinese char with Latin char in the code

[duj12]

done.

[jingyonghou]

is it possible to release the pre-trained model, so people can reproduce the experiment?

[duj12]

Pretrained models are released in the latest commit(README).

[duj12]

Here I do some more experiments on a private datasets

	positive(hello_xiaojing)	negative(noise)
train	18 speakers，2219 segments	55 hours
dev	2 speakers，248 segments	12 hours
test	4 speakers，474 segments	24 hours

The results are as following:

backbone	loss	1-FRR(%)	FAR(/24h)	Threshold
ds_tcn	maxpooling	81.1	2	0.88
ds_tcn	ctc	89.7	1	0.02
fsmn	ctc	93.3	2	0.018

As we can see CTC-KWS model outperform Maxpooling model with DS_TCN backbone.
Since the fsmn and ds_tcn use different data to do pretraining (also, different feature pipeline, and with different epoch), it's hard to say which backbone is better.
But the pretrained fsmn(alibaba released in https://modelscope.cn/models/damo/speech_charctc_kws_phone-xiaoyun/summary ) is better than the ds_tcn model I pretrained (23.pt in https://modelscope.cn/datasets/thuduj12/mobvoi_kws_transcription/files ).
So I recommend you use the pretrained FSMN model to train your models.

[haha010508]

`import librosa
from stream_kws_ctc import KeyWordSpotter

def kws_process(wav_path):
kws = KeyWordSpotter(
ckpt_path='model/avg_30.pt',
config_path='model/config.yaml',
token_path='model/tokens.txt',
lexicon_path='model/lexicon.txt',
threshold=0.02,
min_frames=5,
max_frames=250,
interval_frames=50,
score_beam=3,
path_beam=20,
gpu=-1,
is_jit_model=False,)

kws.set_keywords("你好小镜")

# wav_path = '/path/to/your/wave'

y, _ = librosa.load(wav_path, sr=16000, mono=True)
 # NOTE: model supports 16k sample_rate
wav = (y * (1 << 15)).astype("int16").tobytes()

# We inference every 0.3 seconds, in streaming fashion.
interval = int(0.3 * 16000) * 2
for i in range(0, len(wav), interval):
    chunk_wav = wav[i: min(i + interval, len(wav))]
    result = kws.forward(chunk_wav)
    print(result)

if name == 'main':
file = r'./chentianbo_ceqianfang_anjing_0001.wav'
kws_process(file)`

这个自带的音频都不能唤醒，是不是有啥问题？

[duj12]

`import librosa from stream_kws_ctc import KeyWordSpotter

def kws_process(wav_path): kws = KeyWordSpotter( ckpt_path='model/avg_30.pt', config_path='model/config.yaml', token_path='model/tokens.txt', lexicon_path='model/lexicon.txt', threshold=0.02, min_frames=5, max_frames=250, interval_frames=50, score_beam=3, path_beam=20, gpu=-1, is_jit_model=False,)

kws.set_keywords("你好小镜")

# wav_path = '/path/to/your/wave'

y, _ = librosa.load(wav_path, sr=16000, mono=True)
 # NOTE: model supports 16k sample_rate
wav = (y * (1 << 15)).astype("int16").tobytes()

# We inference every 0.3 seconds, in streaming fashion.
interval = int(0.3 * 16000) * 2
for i in range(0, len(wav), interval):
    chunk_wav = wav[i: min(i + interval, len(wav))]
    result = kws.forward(chunk_wav)
    print(result)

if name == 'main': file = r'./chentianbo_ceqianfang_anjing_0001.wav' kws_process(file)`

这个自带的音频都不能唤醒，是不是有啥问题？

You can check the model you used, In kws_demo, I gave two models, one is for "Hi_XiaoWen", the other is for "Nihao_Xiaojing", check it first. And this demo has web server, you can verify it in https://www.modelscope.cn/studios/thuduj12/KWS_Nihao_Xiaojing/summary

[haha010508]

`import librosa from stream_kws_ctc import KeyWordSpotter
def kws_process(wav_path): kws = KeyWordSpotter( ckpt_path='model/avg_30.pt', config_path='model/config.yaml', token_path='model/tokens.txt', lexicon_path='model/lexicon.txt', threshold=0.02, min_frames=5, max_frames=250, interval_frames=50, score_beam=3, path_beam=20, gpu=-1, is_jit_model=False,)

kws.set_keywords("你好小镜")

# wav_path = '/path/to/your/wave'

y, _ = librosa.load(wav_path, sr=16000, mono=True)
 # NOTE: model supports 16k sample_rate
wav = (y * (1 << 15)).astype("int16").tobytes()

# We inference every 0.3 seconds, in streaming fashion.
interval = int(0.3 * 16000) * 2
for i in range(0, len(wav), interval):
    chunk_wav = wav[i: min(i + interval, len(wav))]
    result = kws.forward(chunk_wav)
    print(result)

if name == 'main': file = r'./chentianbo_ceqianfang_anjing_0001.wav' kws_process(file)`
这个自带的音频都不能唤醒，是不是有啥问题？

You can check the model you used, In kws_demo, I gave two models, one is for "Hi_XiaoWen", the other is for "Nihao_Xiaojing", check it first. And this demo has web server, you can verify it in https://www.modelscope.cn/studios/thuduj12/KWS_Nihao_Xiaojing/summary

I got 2 model, one is 23.pt (key word is 你好问问, or 嗨小问), another is avg_30.pt (key word is 你好晓静)， am i right？ i want to reproduce the result like your web server on my computer, so please help, thanks a lot!

[duj12]

`import librosa from stream_kws_ctc import KeyWordSpotter
def kws_process(wav_path): kws = KeyWordSpotter( ckpt_path='model/avg_30.pt', config_path='model/config.yaml', token_path='model/tokens.txt', lexicon_path='model/lexicon.txt', threshold=0.02, min_frames=5, max_frames=250, interval_frames=50, score_beam=3, path_beam=20, gpu=-1, is_jit_model=False,)

kws.set_keywords("你好小镜")

# wav_path = '/path/to/your/wave'

y, _ = librosa.load(wav_path, sr=16000, mono=True)
 # NOTE: model supports 16k sample_rate
wav = (y * (1 << 15)).astype("int16").tobytes()

# We inference every 0.3 seconds, in streaming fashion.
interval = int(0.3 * 16000) * 2
for i in range(0, len(wav), interval):
    chunk_wav = wav[i: min(i + interval, len(wav))]
    result = kws.forward(chunk_wav)
    print(result)

if name == 'main': file = r'./chentianbo_ceqianfang_anjing_0001.wav' kws_process(file)`
这个自带的音频都不能唤醒，是不是有啥问题？

You can check the model you used, In kws_demo, I gave two models, one is for "Hi_XiaoWen", the other is for "Nihao_Xiaojing", check it first. And this demo has web server, you can verify it in https://www.modelscope.cn/studios/thuduj12/KWS_Nihao_Xiaojing/summary

I got 2 model, one is 23.pt (key word is 你好问问, or 嗨小问), another is avg_30.pt (key word is 你好晓静)， am i right？ i want to reproduce the result like your web server on my computer, so please help, thanks a lot!

Here are two models you can use. https://github.com/duj12/kws_demo/tree/master/model, the code is also here.
And the model(23.pt) you mentioned is a pretrained model without specific key words. You cannot use this pretrianed model to do kws.

[robin1001]

Great job! let's merge and refine in the future.

[Dapannnnn]

Here I do some more experiments on a private datasets

  positive(hello_xiaojing) negative(noise)
train 18 speakers，2219 segments 55 hours
dev 2 speakers，248 segments 12 hours
test 4 speakers，474 segments 24 hours
The results are as following:

backbone loss 1-FRR(%) FAR(/24h) Threshold
ds_tcn maxpooling 81.1 2 0.88
ds_tcn ctc 89.7 1 0.02
fsmn ctc 93.3 2 0.018
As we can see CTC-KWS model outperform Maxpooling model with DS_TCN backbone. Since the fsmn and ds_tcn use different data to do pretraining (also, different feature pipeline, and with different epoch), it's hard to say which backbone is better. But the pretrained fsmn(alibaba released in https://modelscope.cn/models/damo/speech_charctc_kws_phone-xiaoyun/summary ) is better than the ds_tcn model I pretrained (23.pt in https://modelscope.cn/datasets/thuduj12/mobvoi_kws_transcription/files ). So I recommend you use the pretrained FSMN model to train your models.

Hi~,when I run run_fsmn_ctc.sh I encountered the following problem:

This is an error reported when exporting onnx, I think it should be related to the input set, but I don't know how to set the input correctly for the fsmn model.

[duj12]

right now the import of fsmn onnx may still have issue, and ctc model runtime is not support yet. 杜靖 ***@***.***

…

在 2023年8月24日，13:14，Dapannnnn ***@***.***> 写道：  Here I do some more experiments on a private datasets positive(hello_xiaojing) negative(noise) train 18 speakers，2219 segments 55 hours dev 2 speakers，248 segments 12 hours test 4 speakers，474 segments 24 hours The results are as following: backbone loss 1-FRR(%) FAR(/24h) Threshold ds_tcn maxpooling 81.1 2 0.88 ds_tcn ctc 89.7 1 0.02 fsmn ctc 93.3 2 0.018 As we can see CTC-KWS model outperform Maxpooling model with DS_TCN backbone. Since the fsmn and ds_tcn use different data to do pretraining (also, different feature pipeline, and with different epoch), it's hard to say which backbone is better. But the pretrained fsmn(alibaba released in https://modelscope.cn/models/damo/speech_charctc_kws_phone-xiaoyun/summary ) is better than the ds_tcn model I pretrained (23.pt in https://modelscope.cn/datasets/thuduj12/mobvoi_kws_transcription/files ). So I recommend you use the pretrained FSMN model to train your models. Hi~,when I run run_fsmn_ctc.sh I encountered the following problem: This is an error reported when exporting onnx, I think it should be related to the input set, but I don't know how to set the input correctly for the fsmn model. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[teinhonglo]

Hi! Could you provide a paper that is implemented in this pull request?
Oh! Is that paper here?
Chen, Mengzhe, et al. "Compact Feedforward Sequential Memory Networks for Small-footprint Keyword Spotting." Interspeech. 2018.

I have another question: how can I transfer it to the Google speech command?
Could you give a quick hint?

Thanks in advance.
TH

[duj12]

Hi! Could you provide a paper that is implemented in this pull request? Oh! Is that paper here? Chen, Mengzhe, et al. "Compact Feedforward Sequential Memory Networks for Small-footprint Keyword Spotting." Interspeech. 2018.

I have another question: how can I transfer it to the Google speech command? Could you give a quick hint?

Thanks in advance. TH

Fow now the lexicon.txt and tokens,txt in Hi_XiaoWen‘s fsmn_CTC model only support Chinese KWS. You need to build a new lexicon and token dict, maybe you can use some resoures in English ASR projects. The tokens.txt is the CTC model's output units, and the lexicon.txt is the map between the words and the output units.

[AWangji]

Hi! Could you provide a paper that is implemented in this pull request? Oh! Is that paper here? Chen, Mengzhe, et al. "Compact Feedforward Sequential Memory Networks for Small-footprint Keyword Spotting." Interspeech. 2018.
I have another question: how can I transfer it to the Google speech command? Could you give a quick hint?
Thanks in advance. TH

Fow now the lexicon.txt and tokens,txt in Hi_XiaoWen‘s fsmn_CTC model only support Chinese KWS. You need to build a new lexicon and token dict, maybe you can use some resoures in English ASR projects. The tokens.txt is the CTC model's output units, and the lexicon.txt is the map between the words and the output units.

Hi, could you please tell us the detailed implement steps of create new Customized words?

[lin-xiaosheng]

Hi! Could you provide a paper that is implemented in this pull request? Oh! Is that paper here? Chen, Mengzhe, et al. "Compact Feedforward Sequential Memory Networks for Small-footprint Keyword Spotting." Interspeech. 2018.
I have another question: how can I transfer it to the Google speech command? Could you give a quick hint?
Thanks in advance. TH

Fow now the lexicon.txt and tokens,txt in Hi_XiaoWen‘s fsmn_CTC model only support Chinese KWS. You need to build a new lexicon and token dict, maybe you can use some resoures in English ASR projects. The tokens.txt is the CTC model's output units, and the lexicon.txt is the map between the words and the output units.

Hi, could you please tell us the detailed implement steps of create new Customized words?

Yes, I also want to know how to quickly add custom Chinese keywords based on the kew_demo. It would be great if the author could briefly explain that.