Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EN Extremely bad wake word recognition #9

Closed
StuartIanNaylor opened this issue Dec 21, 2021 · 9 comments
Closed

EN Extremely bad wake word recognition #9

StuartIanNaylor opened this issue Dec 21, 2021 · 9 comments

Comments

@StuartIanNaylor
Copy link

StuartIanNaylor commented Dec 21, 2021

Hi downloaded https://github.com/espressif/esp-box/releases/download/v0.2.1/ESP-Box_Demo_EN_V0.2.1.bin and flashed.

I have spent over ten minutes several reboots to get only 3 recognitions with worry that my neighbours may be thinking I have gone mad.

If you do manage the wake word then the command turn on/off light works great.
Actually getting the wake word to activate is another matter in a silent room.

I have a feeling it could be your dataset and now it actually has a english male voice it doesn't fit well or this firmware / model is not good.
I have no problems recording some samples if you tell me format and sampling rate...
Maybe community members could contribute if its merely a esp32-s3-box dataset accent problem.

Actually with more playing seems you need to be 2-3 meters away (further the better) for it to work relatively well if you pull the unit out on place close on your desk it doesn't work well maybe AGC?
Maybe its as simple as the MAP of mic spacing is 50mm whilst the box is 34mm, dunno but currently does not seem to be well.

The hardware design and software of the esp-box is brilliant and is really polished but the results currently of ESP_SR wow I really don't want to comment and due to the module being a .a with no source code there is little I can do, but wait fingers crossed.
ESP_DL lacks LSTM/GRU support which for audio is likely to be restrictive but always struggle what the xtensa hifi / esp32-s3 are capable of.
Maybe TFLite-Micro might come to the esp32-s3's rescue with vector optimisation as what is available and that its closed source is a huge stumbling block for anyone wanting to do ML on the S3

@s60sc
Copy link

s60sc commented Dec 22, 2021

I received my ESP Box today.
I've not been able to get the wake word to activate at all. I've tried both the 0.1.1 and 0.2.1 firmware, and also tried waking it with the voice commands in this demo video without success.

@s60sc
Copy link

s60sc commented Dec 24, 2021

Tried a Skainet example which uses the wake word Alexa and this works fine.

Compiled the ESP_Box demo to use wakeword model alexa8 instead of hiesp8 and now wakes easily.

@StuartIanNaylor
Copy link
Author

Tried a Skainet example which uses the wake word Alexa and this works fine.

Compiled the ESP_Box demo to use wakeword model alexa8 instead of hiesp8 and now wakes easily.

Maybe the Hi_ESP_Q8 model as I did get it to work but learning how to position myself and control my speech for wake word recognition took considerable time.
But maybe your recompile as multinet in the supplied firmware seemed equally buggy as after saying 'turn white' anything was recognised as 'turn white'.

I need to get back up to speed with esp and install the IDF and get to grips with the framework so will also compile and test but currently the supplied firmware gives an extremely bad impression of the box capabilities which in use is near useless.

@s60sc how did you find the multinet commands?

https://github.com/espressif/esp-sr/blob/509cf82658cf2b645aa9ae89b6764beea48e7eae/include/esp_map.h#L23
Also did you change to 34mm as with the box mic arrangement?

@s60sc
Copy link

s60sc commented Dec 26, 2021

The multiword model worked fine for me. I did not change the mic arrangement. I will do more work on the ESP-BOX in the new year. I have uploaded a version of the firmware with the Alexa wakeword in this repository

@feizi
Copy link

feizi commented Jan 4, 2022

Hi, we have updated a new "Hi,ESP" model by adding more samples.
You can try it by pulling the latest esp-sr release/v1.0 branch.
The model of "Hi,ESP" is same as "Alexa". The main difference is the sample. "Alexa" samples come from different countries and accents so "Alexa" model feels more accurate. We will continue to improve the performance of "Hi,ESP" model.

@s60sc
Copy link

s60sc commented Jan 4, 2022

Tried the skainet en_speech_commands_recognition example after updating the esp-sr component, installed on the ESP-Box.
The new hiesp8 model crashes the ESP-Box without even logging an exception. The alexa8 model works fine.

hiesp8 version:

model_name: hiesp8 model_data: /srmodel/hiesp8/wn8_data
HUFZIP MC Quantized wakeNet8: WakeNet8_v3h8_hiesp_5_0.59_0.60, mode:2, p:1, (Jan  4 2022 15:08:13)

alexa8 version:

model_name: alexa8 model_data: /srmodel/alexa8/wn8_data
HUFZIP MC Quantized wakeNet8: wakeNet8_v5h8_alexa_5_0.57_0.59, mode:2, p:1, (Jan  4 2022 15:08:13)

Core dump if that helps:

Quantized8 Multinet5: MN5Q8_v1_english_8_0.9_0.90, (Dec 30 2021 14:40:34)
------------detect start------------
Initiating core dump!
I (4849) esp_core_dump_uart: Press Enter to print core dump to UART...
I (4856) esp_core_dump_uart: Print core dump to uart...
Core dump started (further output muted)
Received  16 kB...
Core dump finished!
espcoredump.py v0.4-dev
===============================================================
==================== ESP32 CORE DUMP START ====================

Crashed task handle: 0x3fcb0500, name: 'detect', GDB name: 'process 1070269696'

================== CURRENT THREAD REGISTERS ===================
exccause       0x1c (LoadProhibitedCause)
excvaddr       0x10
epc1           0x4202b8ef
epc2           0x0
epc3           0x0
epc4           0x0
epc5           0x0
epc6           0x0
eps2           0x0
eps3           0x0
eps4           0x0
eps5           0x0
eps6           0x0
pc             0x4201be38          0x4201be38 <dl_layer_glu_i16+120>
lbeg           0x4037cd84          1077398916
lend           0x4037cda7          1077398951
lcount         0x0                 0
sar            0x1a                26
ps             0x60b20             396064
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x820083a7          -2113895513
a1             0x3fcb67b0          1070294960
a2             0x3fce5634          1070487092
a3             0x3fca8b98          1070238616
a4             0xfffffff1          -15
a5             0xfffffff6          -10
a6             0x0                 0
a7             0x3d830390          1031996304
a8             0xfffffff1          -15
a9             0x3fce5968          1070487912
a10            0x3fce5634          1070487092
a11            0xfffffff0          -16
a12            0x3d830400          1031996416
a13            0x3da1af90          1034006416
a14            0x0                 0
a15            0x7bb55c8           129717704

==================== CURRENT THREAD STACK =====================
#0  0x4201be38 in dl_layer_glu_i16 (args=0x3fce5968, in=<optimized out>, out=0x3fce6844) at ../components/dl_lib/layer/dl_layer_glu.c:107
#1  0x420083a7 in wakeNet7_model_run_step1 (model=0x3fce42b0) at ../components/wakenet/wakenet8_quantized.c:736
#2  0x42008fa8 in model_detect (model=0x3fce42b0, samples=<optimized out>) at ../components/wakenet/wakenet8_quantized.c:1002
#3  0x4200a362 in afe_fetch (afe=0x3fce195c, out=0x3fcebd58) at ../components/esp_audio_front_end/esp_afe_sr_2mic.c:464
#4  0x42007d74 in detect_Task (arg=0x3fce195c) at ../main/main.c:110
#5  0x4038b9a8 in vPortTaskWrapper (pxCode=0x42007cec <detect_Task>, pvParameters=0x3fce195c) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130

======================== THREADS INFO =========================
  Id   Target Id          Frame
* 1    process 1070269696 0x4201be38 in dl_layer_glu_i16 (args=0x3fce5968, in=<optimized out>, out=0x3fce6844) at ../components/dl_lib/layer/dl_layer_glu.c:107
  2    process 1070240844 0x400559e0 in ?? ()
  3    process 1070238956 0x4206813e in esp_pm_impl_waiti () at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:182
  4    process 1070234096 0x400559e0 in ?? ()
  5    process 1070229428 xQueueSemaphoreTake (xQueue=0x3fca6358, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/queue.c:1563
  6    process 1070227788 0x40389758 in xQueueSemaphoreTake (xQueue=0x3fca5cf0, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
  7    process 1070269348 0x403895b5 in xQueueReceive (xQueue=0x3fce1708, pvBuffer=0x3fce16f4, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38

==================== THREAD 1 (TCB: 0x3fcb0500, name: 'detect') =====================
#0  0x4201be38 in dl_layer_glu_i16 (args=0x3fce5968, in=<optimized out>, out=0x3fce6844) at ../components/dl_lib/layer/dl_layer_glu.c:107
#1  0x420083a7 in wakeNet7_model_run_step1 (model=0x3fce42b0) at ../components/wakenet/wakenet8_quantized.c:736
#2  0x42008fa8 in model_detect (model=0x3fce42b0, samples=<optimized out>) at ../components/wakenet/wakenet8_quantized.c:1002
#3  0x4200a362 in afe_fetch (afe=0x3fce195c, out=0x3fcebd58) at ../components/esp_audio_front_end/esp_afe_sr_2mic.c:464
#4  0x42007d74 in detect_Task (arg=0x3fce195c) at ../main/main.c:110
#5  0x4038b9a8 in vPortTaskWrapper (pxCode=0x42007cec <detect_Task>, pvParameters=0x3fce195c) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130

==================== THREAD 2 (TCB: 0x3fca944c, name: 'IDLE') =====================
#0  0x400559e0 in ?? ()
#1  0x4038bc59 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/include/freertos/portmacro.h:571
#2  vPortExitCritical (mux=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:318
#3  0x4202c09e in esp_task_wdt_reset () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/task_wdt.c:330
#4  0x4202c0aa in idle_hook_cb () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/task_wdt.c:80
#5  0x42003974 in esp_vApplicationIdleHook () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/freertos_hooks.c:51
#6  0x40389e30 in prvIdleTask (pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/tasks.c:3974
#7  0x4038b9a8 in vPortTaskWrapper (pxCode=0x40389e24 <prvIdleTask>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130

==================== THREAD 3 (TCB: 0x3fca8cec, name: 'IDLE') =====================
#0  0x4206813e in esp_pm_impl_waiti () at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:182
#1  0x42003982 in esp_vApplicationIdleHook () at C:/Utilities/ESP32_IDF_V44R/components/esp_system/freertos_hooks.c:63
#2  0x40389e30 in prvIdleTask (pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/tasks.c:3974
#3  0x4038b9a8 in vPortTaskWrapper (pxCode=0x40389e24 <prvIdleTask>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130

==================== THREAD 4 (TCB: 0x3fca79f0, name: 'esp_timer') =====================
#0  0x400559e0 in ?? ()
#1  0x4038bc59 in vPortClearInterruptMaskFromISR (prev_level=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/include/freertos/portmacro.h:571
#2  vPortExitCritical (mux=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:318
#3  0x4038b1fd in ulTaskGenericNotifyTake (uxIndexToWait=0, xClearCountOnExit=1, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/tasks.c:5388
#4  0x420073bb in timer_task (arg=0x0) at C:/Utilities/ESP32_IDF_V44R/components/esp_timer/src/esp_timer.c:384
#5  0x4038b9a8 in vPortTaskWrapper (pxCode=0x420073ac <timer_task>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130

==================== THREAD 5 (TCB: 0x3fca67b4, name: 'ipc1') =====================
#0  xQueueSemaphoreTake (xQueue=0x3fca6358, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/freertos/queue.c:1563
#1  0x4038212b in ipc_task (arg=0x1) at C:/Utilities/ESP32_IDF_V44R/components/esp_ipc/src/esp_ipc.c:59
#2  0x4038b9a8 in vPortTaskWrapper (pxCode=0x403820fc <ipc_task>, pvParameters=0x1) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130

==================== THREAD 6 (TCB: 0x3fca614c, name: 'ipc0') =====================
#0  0x40389758 in xQueueSemaphoreTake (xQueue=0x3fca5cf0, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
#1  0x4038212b in ipc_task (arg=0x0) at C:/Utilities/ESP32_IDF_V44R/components/esp_ipc/src/esp_ipc.c:59
#2  0x4038b9a8 in vPortTaskWrapper (pxCode=0x403820fc <ipc_task>, pvParameters=0x0) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130Exception in thread Thread-1:
Traceback (most recent call last):
  File "threading.py", line 932, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "subprocess.py", line 1366, in _readerthread
OSError: [Errno 22] Invalid argument
WARNING: Attempt to terminate the GDB process failed, because it is already terminated. Skip


==================== THREAD 7 (TCB: 0x3fcb03a4, name: 'feed') =====================
#0  0x403895b5 in xQueueReceive (xQueue=0x3fce1708, pvBuffer=0x3fce16f4, xTicksToWait=<optimized out>) at C:/Utilities/ESP32_IDF_V44R/components/hal/esp32s3/include/hal/cpu_ll.h:38
#1  0x42027698 in i2s_read (i2s_num=I2S_NUM_1, dest=<optimized out>, size=6400, bytes_read=0x3fcb021c, ticks_to_wait=4294967295) at C:/Utilities/ESP32_IDF_V44R/components/driver/i2s.c:2219
#2  0x42007ca5 in feed_Task (arg=0x3fce195c) at ../main/main.c:67
#3  0x4038b9a8 in vPortTaskWrapper (pxCode=0x42007c58 <feed_Task>, pvParameters=0x3fce195c) at C:/Utilities/ESP32_IDF_V44R/components/freertos/port/xtensa/port.c:130


======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x600fe000 0x0 RW
.rtc.dummy 0x600fe000 0x0 RW
.rtc.force_fast 0x600fe000 0x0 RW
.rtc.data 0x50000000 0x10 RW A
.rtc_noinit 0x50000010 0x0 RW
.rtc.force_slow 0x50000010 0x0 RW
.iram0.vectors 0x40378000 0x403 R XA
.iram0.text 0x40378404 0x16407 R XA
.dram0.data 0x3fc9e810 0x37f4 RW A
.noinit 0x3fca2004 0x0 RW
.flash.text 0x42000020 0x6ae27 R XA
.flash.appdesc 0x3c070020 0x100 R  A
.flash.rodata 0x3c070120 0x212e4 RW A
.flash.rodata_noload 0x3c091404 0x0 RW
.iram0.text_end 0x4038e80b 0x0 RW
.iram0.bss 0x4038e80c 0x0 RW
.dram0.heap_start 0x3fca4358 0x0 RW
.coredump.tasks.data 0x3fcb0500 0x158 RW
.coredump.tasks.data 0x3fcb66f0 0x304 RW
.coredump.tasks.data 0x3fca944c 0x158 RW
.coredump.tasks.data 0x3fca98c0 0x2e4 RW
.coredump.tasks.data 0x3fca8cec 0x158 RW
.coredump.tasks.data 0x3fca91c0 0x284 RW
.coredump.tasks.data 0x3fca79f0 0x158 RW
.coredump.tasks.data 0x3fca7750 0x298 RW
.coredump.tasks.data 0x3fca67b4 0x158 RW
.coredump.tasks.data 0x3fca6530 0x27c RW
.coredump.tasks.data 0x3fca614c 0x158 RW
.coredump.tasks.data 0x3fca5ec0 0x284 RW
.coredump.tasks.data 0x3fcb03a4 0x158 RW
.coredump.tasks.data 0x3fcb00e0 0x2bc RW

===================== ESP32 CORE DUMP END =====================
===============================================================
Done!
Coredump checksum='315a1d99'
I (6357) esp_core_dump_uart: Core dump has been written to uart.

@feizi
Copy link

feizi commented Jan 5, 2022

have fixed the bug of wakenet8_hiesp, pls try again.

@s60sc
Copy link

s60sc commented Jan 5, 2022

Now works, and hiesp8 model is significantly improved.

Updated factory demo firmwares in this repository

@StuartIanNaylor
Copy link
Author

Seems to be now fixed so will close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants