provided DS-CNN accuracy (on board)

[4p0pt0Z]

Hi,

When testing the provided DS-CNN on my board, the accuracy seems to be very low. Especially in the real time example, the model almost always predicts "left" as soon as there is some sound.
Everything seems fine listening to the audio loopback, and the DNN provides much better results.
Is there specific parameters to use with DS-CNN in order to get high accuracy ?

Thanks

[leejaeuk]

In my case, DS-CNN provided very high accuracy on my FRDM-K64 board.(both simple and real-time test are good)
what is your board model name?
I think is relate to microphone problem like wave file frequency or any other format.

[4p0pt0Z]

Hi leejaeuk,

I am using the STM32F746G discovery kit, with the on-board microphones. I had to compile the code using mbed-os5 (see #43), but apart from this change I did not change anything.
Thanks

[leejaeuk]

is it any problem? when you compile using mbed-os? You should check a process of mbed compile.
before execute mbed new kws_realtime_test --create-only command, you must execute

-cd Deployment
-git clone https://github.com/ARM-software/CMSIS_5.git
-pip install mbed-cli

or you should using virtual machine ubuntu 16.04 version
and you can use mbed compile tool.

[4p0pt0Z]

No, compiling with mbed-os does not bring up any problems. I followed the steps you describe (pip install inside a conda environment) before creating the project folder.
Anyway, the code compiles and runs, and provides good results with the DNN. The DS-CNN however almost always predicts left in the realtime example.

[leejaeuk]

maybe you can modify Deployment/Examples/realtime_test/main.cpp file to
Deployment/Examples/simple_test/main.cpp file.
how to change DNN -> DS-CNN in your code?

[4p0pt0Z]

The simple_test example works alright (predicts "right" with 99% confidence).
To change from DNN to DS-CNN? I modified

• the include in "kws_f746ng.h"
• the parent class of KWS_F746NG in "kws_f746ng.h" and in the constructor implementation in "kws_f746ng.cpp"

The dataset is composed of close talk recorded samples, however there are far field on the board. Could it be that the DNN is simply more robust to ambient noise than the DS-CNN ?

[4p0pt0Z]

Hi @sherylll ,

where did you get the value 99%? From what I understand the results kws.output are values of q7_t type, i.e. from -128 to 127?

Indeed, but you can for instance translate this value to percentage with (int)kws.output[max_ind]*100/128 as is done in kws_simple_test/main.cpp

In my case, I get 127 for "right" with both DNN and DS-CNN, on a STM32F746G and a STM32F767ZI board.

[sherylll]

Hi @4p0pt0Z

Thanks a lot! I realized that I missed one line so I deleted my last comment...

I also noticed that the scaled MFCC can exceed -127 in the simple test example, as I set a break point here:

ML-KWS-for-MCU/Deployment/Source/MFCC/mfcc.cpp

Line 210 in e3f1685

mfcc_out[i] = sum;

Is DNN robust against such cases, or is my board misbehaving?

[4p0pt0Z]

Seems to be fixed by b6e13d6

[ccnankai]

@4p0pt0Z
have encountered the same problem. the model almost always predicts "down" as soon as there is some sound. you solved it?

[4p0pt0Z]

Hi @ccnankai,
My problem got solved by b6e13d6. Which version do you use ?