-
Notifications
You must be signed in to change notification settings - Fork 226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Own wake word never detected despite precise-test accuracy of 100% #94
Comments
If you're only training with wake words and without not-wake-words, your accuracy will be terrible. Also, accuracy starts to improve after you get past 50 or so wake word samples (and at least twice that many in not wakeword + noises) |
Hello,
Thanks for reply.
I will try to increase sample quantity as you tell me.
In fact I forgot to say that in my tests, I also applied “Reducing False Activation” Method 2 using pdsounds_march2009.7z.
So during my tests I have
· 92 samples in /home/pi/mycroft-precise/my-wakeword/test/not-wake-word/generated directory
· 4 samples in /home/pi/mycroft-precise/my-wakeword /test/wake-word directory
· 141 samples in /home/pi/mycroft-precise/my-wakeword/not-wake-word/generated directory
· 9 samples in /home/pi/mycroft-precise/my-wakeword/wake-word directory
I understand that with this small samples quantity accuracy is 100% and that the wake word is not detected because prediction is low.
But the prediction obtained is around 5E-6 ! Is it normal ?
|
Thank you el-tocino . So i think this issue may be closed, but i think also it will be good to update "how-to-train-your-own-wake-word" wiki page with this information to help new users. Below an upgraded summary of my new Usage 4 test : Usage 4 (source install) : create and test a custom wake word.ReferenceCreation of *.wav wake word filesFirst adjust microphone level to 90% using alsamixer, and disable automatic gain. Wav files post treatmentTo improve samples "quality", I apply following operations (using Audacity free software) : NormalizationDrag and drop all files in Audacity. Noise reductionMy goal was to remove microphone noise from the samples to improve SNR (Signal to Noise Ratio) because recorded samples are very noisy. Creating directory structure to train the modelThen I dispatch obtained wav files in the following /home/pi/mycroft-precise/my-wakeword directory structure : Train the modelReference https://github.com/MycroftAI/mycroft-precise/wiki/Research Reduce false activation (method 2)Download Public Domain SoundsDownload of pdsounds_march2009.7z from http://downloads.tuxfamily.org/pdsounds/pdsounds_march2009.7z, then uncompress it in /home/pi/mycroft-precise/data/random. Transform file to 16bit signed integer mono wav filesIf have to install ffmeg Then create text file /home/pi/mycroft-precise/public_domain_sound_to_16000_wav.sh containing :
Then make it executable and execute it :
Processing reduce false activation process
Test the model
=== False Negatives === === Counts === === Summary === 0.88% false positives Convert the model from keras (.net) to tensorflow (.db)
Testing the model with precise-listen (fails)
|
Thanks for the detailed notes. I'm pulling all our documentation together under mycroft.ai/documentation at the moment so will incorporate this when I get to the Precise section. |
Trying to create an own wake word following tutorial https://github.com/MycroftAI/mycroft-precise/wiki/Training-your-own-wake-word#how-to-train-your-own-wake-word , precise-test script applied to the generated model result with a summary of 100% success, 0.00% false positives and 0.00% false negatives.
Despite this result, precise-listen never detect the own wake word and precise-engine return a precision which is always 0.0
I think that my installation seems correct because using downloaded hey-mycroft.db model, "Hey Mycroft" is correctly detected, either with source or binary install.
Thanks to help me to find what may clamp precision to 0 !
Below, please find summary of installation and usage i applied on raspberry pi 4.
Platform
Hardware : Raspberry Pi 4 + Dongle 3D Sound (lsusb => ID 1b3f:2008 Generalplus Technology Inc.)
Software : Buster Lite + Pulseaudio 12.2
Installation
Source installation
Clone repository
git clone https://github.com/mycroftai/mycroft-precise
git checkout 2d65736
Apply the 2 following modifications (see attached patch)
a) Replacing 2 occurences of 37ef1ab91eeca81fd889bce2967775b2f6918d97 which does'nt exist in repository with 2d65736
b) Apply issue #88 solution (otherwise, precise-listen console scriptwill end with en error "Could not find resource: localhost/dense_1/bias")
Setup
cd /home/pi/mycroft-precise
sudo ./setup.sh
(setup ended with Successfully installed pocketsphinx-0.1.15)source .venv/bin/activate
(needed to set environment variables)Force keras version to V2.2.4
Here Keras was in version 2.3.1, then, using precise-convert console script, an error "TypeError: Expected Operation, Variable, or Tensor, got 0" occured.
Keras version can be checked using command
pip freeze | grep Keras==
To force keras version to V2.2.4 use following command :
Here Keras was in version 2.3.1, so execute following command :
sudo /home/pi/mycroft-precise/.venv/bin/pip install keras==2.2.4
hey-mycroft reference model installation
Download hey-mycroft.tar.gz from precise-data repository https://github.com/MycroftAI/precise-data/tree/models/hey-mycroft.tar.gz
Then copy hey-mycroft.pb and hey-mycroft.pb.params in directory /home/pi/mycroft-precise/
Binary installation
Download from repository https://github.com/MycroftAI/precise-data/blob/dist/armv7l/latest-dev file https://github.com/MycroftAI/mycroft-precise/releases/download/v0.3.0/precise-engine_0.3.0_armv7l.tar.gz
Uncompress precise-engine_0.3.0_armv7l.tar.gz in directory /home/pi/mycroft-precise/precise-engine_0.3.0_armv7l/precise-engine
Creation of a wake word python test script
This script aims to test a model against binary or source install.
I created it in file /home/pi/home/pi/mycroft-precise/wakeword_test.py (see attached wakeword_test.py attached file (default configuration hey-mycroft model and binary installation))
Usage
Usage 1 (binary install) : test of hey-mycroft.db model
cd /home/pi/mycroft-precise
source .venv/bin/activate
python3 wakeword_test.py
Result : works fine, each hey-mycroft wake-word => Activation!
Usage 2 (source install) : test of hey-mycroft.db model
cd /home/pi/mycroft-precise
source .venv/bin/activate
Modify wakeword_test.py for source install (PreciseEnginePath=SourceInstallPreciseEnginePath)
python3 wakeword_test.py
Result : works fine, each hey-mycroft wake-word => Activation!
Usage 3 (source install) : test of hey-mycroft.db model using precise-listen
(precise-listen console scripts runs precise/scripts/listen.py, usage : precise-listen --help )
cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-listen hey-mycroft.pb
Result : wakeword 'hey mycroft is detected' at level displayed by XXXX to xxx transition below
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx---
precise-listen -b hey-mycroft.pb
Result : wakeword 'hey mycroft is detected' as
..........!!!!!.........
Usage 4 (source install) : create and test a custom wake word.
Reference
Reference : https://github.com/MycroftAI/mycroft-precise/wiki/Training-your-own-wake-word#how-to-train-your-own-wake-word
Creation of *.wav wake word files
cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-collect
File label (Ex. recording-##):
(enter my-wakeword-## for instance)Press space to record (esc to exit)...
Here, press space, wait 1 to 2s then say wakeword and press space again
Repeat it twelve to create 12 samples wav files, then press escape.
Then I dispatch obtained wav files in following /home/pi/mycroft-precise/my-wakeword directory structure
my-wakeword/
├── wake-word/
│ ├── my-wakeword.00.wav
│ ├── my-wakeword.01.wav
│ ├── my-wakeword.02.wav
│ ├── my-wakeword.03.wav
│ ├── my-wakeword.04.wav
│ ├── my-wakeword.05.wav
│ ├── my-wakeword.06.wav
│ ├── my-wakeword.07.wav
│ └── my-wakeword.08.wav
├── not-wake-word/
└── test/
├── wake-word/
│ ├── my-wakeword.09.wav
│ ├── my-wakeword.10.wav
│ ├── my-wakeword.11.wav
│ └── my-wakeword.12.wav
└── not-wake-word/
Train the model
precise-train my-wakeword.net my-wakeword -e 21 -b 64 -s 0.1 -sb -mm val_acc -em
precise-train my-wakeword.net my-wakeword -e 100 -b 2048 -s 0.1 -sb -mm val_acc -em
Test the model
precise-test my-wakeword.net my-wakeword/
Result :
Data:
=== False Positives ===
=== False Negatives ===
=== Counts ===
False Positives: 0
True Negatives: 92
False Negatives: 0
True Positives: 4
=== Summary ===
96 out of 96
100.00%
0.00% false positives
0.00% false negatives
Convert the model from keras (.net) to tensorflow (.db)
precise-convert my-wakeword.net
Testing the model with precise-listen (fails)
precise-listen my-wakeword.pb
Result : always
--------------------------------------------------------------------------------
and sometimes when speaking wake word
X-------------------------------------------------------------------------------
Testing the model with modified wakeword_test.py (fails and precision = 0)
In wakeword_test.py i add :
def on_pred(probe): print('Prediction!', probe)
and change :
runner = PreciseRunner(engine, on_prediction=on_pred, on_activation=on_act)
Then running :
python3 python3 wakeword_test.py
And either speaking the wake word in front of the microphone, either playing one of the wake words file which was used to train the model in front of the microphone, I obtain always precision near 0 as following example :
Prediction! 0.00056418408999548 <<<< not speaking wake word
Prediction! 0.0022653245389246185
Prediction! 0.0014790119279027001
Prediction! 0.000680880909922902
Prediction! 8.827383307993124e-05 <<<< speaking wake word
Prediction! 1.897542464599091e-05
Prediction! 4.1821945551526674e-07
Prediction! 4.1821945551526674e-07
Prediction! 4.1821945551526674e-07
Prediction! 4.1821945551526674e-07
Repeating the same test with hey-mycroft reference model, precision values are between 0 and 1, incresing near 0.8 or above when speaking the wake word.
I don't understand why precision seems clamped to 0 for a custom wake word whose precise-test result is 100% ? What may be wrong ?
2d65736ba589af50297de46905799ac1b1485cf1-patch.txt
wakeword_test-py.txt
The text was updated successfully, but these errors were encountered: