Own wake word never detected despite precise-test accuracy of 100%

[tdy91]

Trying to create an own wake word following tutorial https://github.com/MycroftAI/mycroft-precise/wiki/Training-your-own-wake-word#how-to-train-your-own-wake-word , precise-test script applied to the generated model result with a summary of 100% success, 0.00% false positives and 0.00% false negatives.
Despite this result, precise-listen never detect the own wake word and precise-engine return a precision which is always 0.0
I think that my installation seems correct because using downloaded hey-mycroft.db model, "Hey Mycroft" is correctly detected, either with source or binary install.

Thanks to help me to find what may clamp precision to 0 !

Below, please find summary of installation and usage i applied on raspberry pi 4.

Platform

Hardware : Raspberry Pi 4 + Dongle 3D Sound (lsusb => ID 1b3f:2008 Generalplus Technology Inc.)
Software : Buster Lite + Pulseaudio 12.2

Installation

Source installation

Clone repository

git clone https://github.com/mycroftai/mycroft-precise
git checkout 2d65736

Apply the 2 following modifications (see attached patch)

a) Replacing 2 occurences of 37ef1ab91eeca81fd889bce2967775b2f6918d97 which does'nt exist in repository with 2d65736
b) Apply issue #88 solution (otherwise, precise-listen console scriptwill end with en error "Could not find resource: localhost/dense_1/bias")

Setup

cd /home/pi/mycroft-precise
sudo ./setup.sh (setup ended with Successfully installed pocketsphinx-0.1.15)
source .venv/bin/activate (needed to set environment variables)

Force keras version to V2.2.4

Here Keras was in version 2.3.1, then, using precise-convert console script, an error "TypeError: Expected Operation, Variable, or Tensor, got 0" occured.
Keras version can be checked using command pip freeze | grep Keras==
To force keras version to V2.2.4 use following command :
Here Keras was in version 2.3.1, so execute following command :
sudo /home/pi/mycroft-precise/.venv/bin/pip install keras==2.2.4

hey-mycroft reference model installation

Download hey-mycroft.tar.gz from precise-data repository https://github.com/MycroftAI/precise-data/tree/models/hey-mycroft.tar.gz
Then copy hey-mycroft.pb and hey-mycroft.pb.params in directory /home/pi/mycroft-precise/

Binary installation

Download from repository https://github.com/MycroftAI/precise-data/blob/dist/armv7l/latest-dev file https://github.com/MycroftAI/mycroft-precise/releases/download/v0.3.0/precise-engine_0.3.0_armv7l.tar.gz
Uncompress precise-engine_0.3.0_armv7l.tar.gz in directory /home/pi/mycroft-precise/precise-engine_0.3.0_armv7l/precise-engine

Creation of a wake word python test script

This script aims to test a model against binary or source install.
I created it in file /home/pi/home/pi/mycroft-precise/wakeword_test.py (see attached wakeword_test.py attached file (default configuration hey-mycroft model and binary installation))

Usage

Usage 1 (binary install) : test of hey-mycroft.db model

cd /home/pi/mycroft-precise
source .venv/bin/activate
python3 wakeword_test.py
Result : works fine, each hey-mycroft wake-word => Activation!

Usage 2 (source install) : test of hey-mycroft.db model

cd /home/pi/mycroft-precise
source .venv/bin/activate
Modify wakeword_test.py for source install (PreciseEnginePath=SourceInstallPreciseEnginePath)
python3 wakeword_test.py
Result : works fine, each hey-mycroft wake-word => Activation!

Usage 3 (source install) : test of hey-mycroft.db model using precise-listen

(precise-listen console scripts runs precise/scripts/listen.py, usage : precise-listen --help )
cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-listen hey-mycroft.pb
Result : wakeword 'hey mycroft is detected' at level displayed by XXXX to xxx transition below
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx---
precise-listen -b hey-mycroft.pb
Result : wakeword 'hey mycroft is detected' as ..........!!!!!.........

Usage 4 (source install) : create and test a custom wake word.

Reference

Reference : https://github.com/MycroftAI/mycroft-precise/wiki/Training-your-own-wake-word#how-to-train-your-own-wake-word

Creation of *.wav wake word files

cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-collect
File label (Ex. recording-##): (enter my-wakeword-## for instance)
Press space to record (esc to exit)...
Here, press space, wait 1 to 2s then say wakeword and press space again
Repeat it twelve to create 12 samples wav files, then press escape.

Then I dispatch obtained wav files in following /home/pi/mycroft-precise/my-wakeword directory structure
my-wakeword/
├── wake-word/
│ ├── my-wakeword.00.wav
│ ├── my-wakeword.01.wav
│ ├── my-wakeword.02.wav
│ ├── my-wakeword.03.wav
│ ├── my-wakeword.04.wav
│ ├── my-wakeword.05.wav
│ ├── my-wakeword.06.wav
│ ├── my-wakeword.07.wav
│ └── my-wakeword.08.wav
├── not-wake-word/
└── test/
├── wake-word/
│ ├── my-wakeword.09.wav
│ ├── my-wakeword.10.wav
│ ├── my-wakeword.11.wav
│ └── my-wakeword.12.wav
└── not-wake-word/

Train the model

precise-train my-wakeword.net my-wakeword -e 21 -b 64 -s 0.1 -sb -mm val_acc -em
precise-train my-wakeword.net my-wakeword -e 100 -b 2048 -s 0.1 -sb -mm val_acc -em

Test the model

precise-test my-wakeword.net my-wakeword/
Result :
Data:
=== False Positives ===

=== False Negatives ===

=== Counts ===
False Positives: 0
True Negatives: 92
False Negatives: 0
True Positives: 4

=== Summary ===
96 out of 96
100.00%

0.00% false positives
0.00% false negatives

Convert the model from keras (.net) to tensorflow (.db)

precise-convert my-wakeword.net

Testing the model with precise-listen (fails)

precise-listen my-wakeword.pb
Result : always
--------------------------------------------------------------------------------
and sometimes when speaking wake word
X-------------------------------------------------------------------------------

Testing the model with modified wakeword_test.py (fails and precision = 0)

In wakeword_test.py i add :

def on_pred(probe): print('Prediction!', probe)

and change :

runner = PreciseRunner(engine, on_prediction=on_pred, on_activation=on_act)

Then running :

python3 python3 wakeword_test.py

And either speaking the wake word in front of the microphone, either playing one of the wake words file which was used to train the model in front of the microphone, I obtain always precision near 0 as following example :

Prediction! 0.00056418408999548 <<<< not speaking wake word
Prediction! 0.0022653245389246185
Prediction! 0.0014790119279027001
Prediction! 0.000680880909922902
Prediction! 8.827383307993124e-05 <<<< speaking wake word
Prediction! 1.897542464599091e-05
Prediction! 4.1821945551526674e-07
Prediction! 4.1821945551526674e-07
Prediction! 4.1821945551526674e-07
Prediction! 4.1821945551526674e-07

Repeating the same test with hey-mycroft reference model, precision values are between 0 and 1, incresing near 0.8 or above when speaking the wake word.

I don't understand why precision seems clamped to 0 for a custom wake word whose precise-test result is 100% ? What may be wrong ?

2d65736ba589af50297de46905799ac1b1485cf1-patch.txt

wakeword_test-py.txt

[el-tocino]

If you're only training with wake words and without not-wake-words, your accuracy will be terrible. Also, accuracy starts to improve after you get past 50 or so wake word samples (and at least twice that many in not wakeword + noises)

[tdy91]

Hello, Thanks for reply. I will try to increase sample quantity as you tell me. In fact I forgot to say that in my tests, I also applied “Reducing False Activation” Method 2 using pdsounds_march2009.7z. So during my tests I have · 92 samples in /home/pi/mycroft-precise/my-wakeword/test/not-wake-word/generated directory · 4 samples in /home/pi/mycroft-precise/my-wakeword /test/wake-word directory · 141 samples in /home/pi/mycroft-precise/my-wakeword/not-wake-word/generated directory · 9 samples in /home/pi/mycroft-precise/my-wakeword/wake-word directory I understand that with this small samples quantity accuracy is 100% and that the wake word is not detected because prediction is low. But the prediction obtained is around 5E-6 ! Is it normal ?

[tdy91]

Thank you el-tocino .
You are right, the own wake word begin to work increasing to 50 wake word samples and applying again https://github.com/MycroftAI/mycroft-precise/wiki/Training-your-own-wake-word#how-to-train-your-own-wake-word using reducing false activations Method 2.

So i think this issue may be closed, but i think also it will be good to update "how-to-train-your-own-wake-word" wiki page with this information to help new users.

Below an upgraded summary of my new Usage 4 test :

Usage 4 (source install) : create and test a custom wake word.

Reference

Reference : https://github.com/MycroftAI/mycroft-precise/wiki/Training-your-own-wake-word#how-to-train-your-own-wake-word

Creation of *.wav wake word files

First adjust microphone level to 90% using alsamixer, and disable automatic gain.
cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-collect
File label (Ex. recording-##): (enter my-wakeword-## for instance)
Press space to record (esc to exit)...
Here, press space, wait 1 to 2s then say wakeword and press space again.
Repeat it to create a minimum of 50 samples wav files, then press escape to exit.
Please note : samples duration must be greater then 1.5s, otherwise it will be ignored.

Wav files post treatment

To improve samples "quality", I apply following operations (using Audacity free software) :

Normalization

Drag and drop all files in Audacity.
Menu Effect\Normalize, let default parameters (remove DC offset checked, normalize peak amplitude -1 dB checked), then OK
File, Export, Export multiple and save in another directory.

Noise reduction

My goal was to remove microphone noise from the samples to improve SNR (Signal to Noise Ratio) because recorded samples are very noisy.
Drag and drop all files in Audacity (if they are not already opened)
Select about 1s of noise from one of the samples.
Menu Effect\Noise Reduction and click Get Noise Profile button
Then select all the files using Ctrl-A keyboard shortcut
Menu Effecte\Noise reduction again, I adjusted Noise reduction to 36 dB, leave other parameters to their default valus (sensitivity 6, frequency smoothiong 3, Noise reduce), then click OK
File, Export, Export multiple and save in another directory

Creating directory structure to train the model

Then I dispatch obtained wav files in the following /home/pi/mycroft-precise/my-wakeword directory structure :
my-wakeword/
├── wake-word/
│ ├── my-wakeword.00.wav
│ ├── my-wakeword.01.wav
│ ├── my-wakeword.02.wav
│ ├── my-wakeword.03.wav
│ ├── my-wakeword.04.wav
│ ├── my-wakeword.05.wav
│ ├── my-wakeword.06.wav
│ ├── ...
│ └── my-wakeword.50.wav (minimum)
├── not-wake-word/
└── test/
├── wake-word/
│ ├── my-wakeword.00.wav
│ ├── my-wakeword.01.wav
│ ├── ...
│ └── my-wakeword.10.wav
└── not-wake-word/

Train the model

Reference https://github.com/MycroftAI/mycroft-precise/wiki/Research
precise-train my-wakeword.net my-wakeword -e 21 -b 64 -s 0.1 -sb -mm val_acc -em
precise-train my-wakeword.net my-wakeword -e 100 -b 2048 -s 0.1 -sb -mm val_acc -em

Reduce false activation (method 2)

Download Public Domain Sounds

Download of pdsounds_march2009.7z from http://downloads.tuxfamily.org/pdsounds/pdsounds_march2009.7z, then uncompress it in /home/pi/mycroft-precise/data/random.
I use WinSCP to do it and doesn't try following commands on raspberry pi 4 buster lite :
cd data/random
wget http://downloads.tuxfamily.org/pdsounds/pdsounds_march2009.7z
Maybe install p7zip here
7z x pdsounds_march2009.7z
cd /home/pi/mycroft-precise

Transform file to 16bit signed integer mono wav files

If have to install ffmeg
sudo apt-get install ffmpeg

Then create text file /home/pi/mycroft-precise/public_domain_sound_to_16000_wav.sh containing :

#!/bin/bash
`SOURCE_DIR=data/random/mp3
DEST_DIR=data/random

for i in $SOURCE_DIR/*.mp3; do echo "Converting $i..."; fn=${i##*/}; ffmpeg -i "$i" -acodec pcm_s16le -ar 16000 -ac 1 -f wav "$DEST_DIR/${fn%.*}.wav"; done

Then make it executable and execute it :

chmod +x /home/pi/mycroft-precise/public_domain_sound_to_16000_wav.sh
cd /home/pi/mycroft-precise
./public_domain_sound_to_16000_wav.sh

Processing reduce false activation process

cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-train-incremental my-wakeword.net my-wakeword -e 100 -b 2048 -s 0.1 -sb -mm val_acc -em

Test the model

cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-test my-wakeword.net my-wakeword/
Result :
=== False Positives ===
my-wakeword/test/not-wake-word/generated/plang pleng pling-81.wav

=== False Negatives ===

=== Counts ===
False Positives: 1
True Negatives: 112
False Negatives: 0
True Positives: 11

=== Summary ===
123 out of 124
99.19%

0.88% false positives
0.00% false negatives

Convert the model from keras (.net) to tensorflow (.db)

cd /home/pi/mycroft-precise
source .venv/bin/activate
precise-convert my-wakeword.net

Testing the model with precise-listen (fails)

precise-listen my-wakeword.pb
Result : wakeword is detected at level displayed by XXXX to xxx transition below

--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
X-------------------------------------------------------------------------------
XXXXXXXXXX----------------------------------------------------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxx-------------------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxx-------------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxx--------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxx-------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxx-----------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxx----------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxx---------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxx--------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxx-------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxx-------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxx-------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxx-------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxxx-------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxxxx--------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxxx----------------
^CXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxxxxxx-----------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxxxxxxxxxxxxxxxx---------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXxxxxx-----------------------------------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX--------------------------------------------
XXXXXXXXXXXXXXXXXXX-------------------------------------------------------------
X-------------------------------------------------------------------------------
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------

[krisgesling]

Thanks for the detailed notes.

I'm pulling all our documentation together under mycroft.ai/documentation at the moment so will incorporate this when I get to the Precise section.