Precise Wake Word Listener #1199

MatthewScholefield · 2017-11-03T20:19:27Z

This adds precise as an optional wake word listener instead of Pocketsphinx. For testing, it has been made default by modifying the config with {"hotwords": {"hey mycroft": {"module": "precise"}}}. Once this has been tested, I can remove the Make Precise default commit before a merge.

Notes on testing: It downloads a 70 MB file the first time it runs. On the unit, this will be a 30MB file and it will notify the user with text on the screen. During this initial download period, Mycroft will not respond to the wake word.

forslund · 2017-11-06T15:13:04Z

I'm having a bit of trouble getting precise running on my machine (Debian jessie:

From speech client (after download is complete)

[25960] Error loading Python lib '/tmp/_MEI0vQ7v1/libpython3.5m.so.1.0': dlopen: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/_MEI0vQ7v1/libpython3.5m.so.1.0)
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/ake/projects/python/mycroft-core/mycroft/client/speech/hotword_factory.py", line 190, in check_stdout
    if float(line) > 0.5:
ValueError: could not convert string to float:

I get the same issue if I run the precise-stream from the commandline as well.

I'm on 64-Bit debian Stretch right now, will try on my Ubuntu desktop later tonight.

Another thing, could this use the downloader in utils to consolidate the code a bit? Or is it too clunky?

MatthewScholefield · 2017-11-06T15:21:51Z

@forslund About the error, it's because I built the executable on too new of an OS. I'm working on running automation scripts in an Ubuntu 14.04 docker container which should fix this issue.

Also, I didn't even realize we had a download util. However, things would get a little more ugly if I split the code up since it is non blocking. But, I'm fine modifying Downloader to add a blocking parameter even though that sort of defeats the purpose of the class since it inherits from the Thread class.

forslund · 2017-11-06T15:52:12Z

Point well made.

Will patiently wait for the updated executable :)

MatthewScholefield · 2017-11-06T22:24:58Z

@forslund OK, it should be fixed. I rebased so you will have to git fetch && git checkout feature/precise && git reset --hard origin/precise

forslund

Works really nicely! I'm a little bit in love actually :)

The only big issue I've seen is that it downloads the executable each startup.

forslund · 2017-11-07T22:34:40Z

mycroft/client/speech/hotword_factory.py

+            Popen('echo "' + cmd + '" > /dev/ttyAMA0', shell=True)
+
+        arch = platform.machine()
+        exe_file = expanduser('~/.mycroft/precise/' + self.exe_name)


This path is not one of the paths checked in resolve_resource_file() so the exe is downloaded each startup.

Should be resolved now. Good catch!

forslund · 2017-11-07T22:34:40Z

mycroft/client/speech/hotword_factory.py

+        exe_file = expanduser('~/.mycroft/precise/' + self.exe_name)
+        url = self.url_base + 'dist/' + arch + '/' + self.exe_name
+
+        snd_msg('mouth.text=Updating Listener...')


Could this be done using the enclosure api? Right now I get an error message saying it can't open the /dev/ttyAMA0 device. (also I think the /dev/ttyAMA0 should not be hardcoded, instead the enclosure->port value from the config might be better)

I guess, but that gets messy because then the websocket has to get passed from RecognizerLoop to the HotwordFactory to the HotwordEngine base class, with a handler being written to communicate enclosure text commands from the internal recognizerloop EventEmitter to the actual websocket. Do you think it'd better to just remove it from communicating to the arduino?

We can leave it as is for the time being(maybe use the config value instead of hardcoded path), this was not a big issue, just something I thought of when looking at the output. Feedback to the user is good providing an explanation when the device won't respond.

I'm a bit surprised that it works though, I'd expect the enclosure process to block the serial device and hinder other processes to write to it.

MatthewScholefield · 2017-11-07T22:48:10Z

mycroft/client/speech/hotword_factory.py

+                self.cooldown -= 1
+                self.has_found = False
+                continue
+            if float(line) > 0.5:


Edit: Fixed! I just interpreted Codacy's comment wrong. It actually was complaining because of:

if condition: val = True else: val = False

vs

val = condition

MatthewScholefield · 2017-11-07T22:48:44Z

mycroft/client/speech/hotword_factory.py

+        import shutil
+        from urllib2 import urlopen
+        LOG.info('Downloading: ' + url)
+        req = urlopen(url)


Since the prefix is fixed to https://, I don't think Codacy's comment on this below matters.

forslund · 2017-11-07T23:11:26Z

Works! I'll test more tomorrow morning but if Michael tests it and wants to merge I trust his judgment. I've not yet tested broken downloads for example.

JarbasAI · 2017-11-07T23:24:36Z

is precise only trained for "hey mycroft" or can we change it?

https://github.com/MycroftAI/mycroft-precise-python-experiments is this the code of precise? (so i can train new words)

MatthewScholefield · 2017-11-07T23:31:28Z

@JarbasAI Right now, hey mycroft is the only trained wake word, but it loads from a model file according to the wake word configuration value, meaning we can easily add more in the future. The models are held here. I'm working on building a static executable to easily train new wake words.

JarbasAI · 2017-11-07T23:51:26Z

great :) was wondering if you would go the snowboy route and have them trainable in your website or something

is it easy to train new words?

how much wake word samples do you need to achieve good results?

can you give some estimate on training time consideration aswel?

is the precise repo up to date-ish? could i use that repo's code to train a new model and play around?

MatthewScholefield · 2017-11-08T00:04:17Z

@JarbasAI The repo isn't up to date yet. There's still some work left to ease the process of training. At the moment, it requires a fair amount of data to prevent false positives which is what I'm working on automating. But, for the time being, testing it as an experimental feature will serve to collect the data to prevent false positives.

Also, for new wake words, since we still support pocketsphinx, those that choose to contribute their data are already building up a data for new wake words.

forslund · 2017-11-08T11:35:08Z

I've tested some more and it seems to work really good. I'm still a bit worried about the direct write to the display but I've not tried it on an actual Mark-1 so there might be some magic I'm not aware of

LearnedVector · 2017-11-08T18:09:45Z

So holding I think we should hold off on pulling this in until we figure out the issues with the models

MatthewScholefield · 2017-11-08T18:15:19Z

Even if we don't make it default? Since it downloads new models every day, we should be able to improve it after the merge where we can test on the device, and it won't affect regular users. However, at the very least, I have to make one more commit to this branch changing the source to s3.

…

On Wed, Nov 8, 2017, 12:09 PM Michael Nguyen ***@***.***> wrote: So holding I think we should hold off on pulling this in until we figure out the issues with the models — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1199 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFmlSwjf6CXellmETSa2IamkJtIj4UDVks5s0e5rgaJpZM4QRoNg> .

forslund · 2017-11-08T18:17:21Z

(and make it not-default)

forslund · 2017-11-08T18:25:28Z

Is it the fact that precise is default that makes Travis fail or is that something else?

LearnedVector · 2017-11-08T18:27:43Z

I didn't catch the non-default part! i agree with merging it on non-default.

MatthewScholefield · 2017-11-08T18:35:32Z

@forslund Well, it's failing because it's default and the tests don't use the new .update mechanism.

forslund · 2017-11-08T19:22:31Z

Sigh, sorry github (or the universe) was acting up

forslund · 2017-11-08T19:27:07Z

@MatthewScholefield Ok I'll make a not to revisit those tests

MatthewScholefield · 2017-11-09T18:18:44Z

@forslund @Mn0491 I've made it not default and it now reads from config for the URL so it can be overridden later. For a final test:

git checkout feature/precise
git fetch
git reset --hard origin/feature/precise # or upstream/feature/precise
./start-mycroft.sh debug
# Shouldn't see anything about precise in logs and `hey mycroft` should work with pocketsphinx
! cat ~/.mycroft/mycroft.conf &&  # Make sure nothing in config
echo '{"hotwords": {"hey mycroft": {"module": "precise"}}}' > ~/.mycroft/mycroft.conf
./start-mycroft.sh debug
# Should see found precise executable in logs and wake word should still work (although perhaps with a lot of false positives)

This allows changing the server in the future without an upgrade

ppisljar · 2017-11-16T19:08:46Z

Is precise open sourced as well ? (cant find it)

MatthewScholefield requested review from forslund and LearnedVector November 3, 2017 20:19

MatthewScholefield added the CLA: Yes Contributor License Agreement exists (see https://github.com/MycroftAI/contributors) label Nov 3, 2017

MatthewScholefield force-pushed the feature/precise branch 2 times, most recently from 612bd80 to 3fd5d37 Compare November 3, 2017 20:53

MycroftAI deleted a comment Nov 3, 2017

MatthewScholefield force-pushed the feature/precise branch from 3fd5d37 to c0e438d Compare November 3, 2017 21:30

MycroftAI deleted a comment Nov 6, 2017

MatthewScholefield force-pushed the feature/precise branch from 767b38a to 6e23396 Compare November 6, 2017 21:55

MycroftAI deleted a comment Nov 6, 2017

forslund requested changes Nov 7, 2017

View changes

MatthewScholefield commented Nov 7, 2017

View changes

MycroftAI deleted a comment Nov 7, 2017

forslund approved these changes Nov 7, 2017

View changes

MatthewScholefield force-pushed the feature/precise branch from bd1f241 to 5ffb425 Compare November 9, 2017 17:56

MycroftAI deleted a comment Nov 9, 2017

MatthewScholefield added 4 commits November 9, 2017 13:36

Add Precise hotword engine

b3f11d8

Fix downloading precise every startup

788dc5d

Extract precise server settings into config file

5694b7f

This allows changing the server in the future without an upgrade

Make bool assignment more concise

ed1deeb

MatthewScholefield force-pushed the feature/precise branch from 26f0187 to 6107840 Compare November 9, 2017 19:40

Add wake word module hash to filename

ec223dc

MatthewScholefield force-pushed the feature/precise branch from 6107840 to ec223dc Compare November 9, 2017 19:58

forslund merged commit 01d47fb into dev Nov 9, 2017

MatthewScholefield deleted the feature/precise branch November 16, 2017 22:01

Precise Wake Word Listener #1199

Precise Wake Word Listener #1199

MatthewScholefield commented Nov 3, 2017

forslund commented Nov 6, 2017

MatthewScholefield commented Nov 6, 2017

forslund commented Nov 6, 2017

MatthewScholefield commented Nov 6, 2017

forslund left a comment

forslund Nov 7, 2017

MatthewScholefield Nov 7, 2017 •

edited

forslund Nov 7, 2017

MatthewScholefield Nov 7, 2017

forslund Nov 7, 2017

MatthewScholefield Nov 7, 2017 •

edited

MatthewScholefield Nov 7, 2017 •

edited

forslund commented Nov 7, 2017

JarbasAI commented Nov 7, 2017 •

edited

MatthewScholefield commented Nov 7, 2017 •

edited

JarbasAI commented Nov 7, 2017 •

edited

MatthewScholefield commented Nov 8, 2017 •

edited

forslund commented Nov 8, 2017 •

edited

LearnedVector commented Nov 8, 2017

MatthewScholefield commented Nov 8, 2017 via email

forslund commented Nov 8, 2017

forslund commented Nov 8, 2017

LearnedVector commented Nov 8, 2017

MatthewScholefield commented Nov 8, 2017

forslund commented Nov 8, 2017

forslund commented Nov 8, 2017

MatthewScholefield commented Nov 9, 2017 •

edited

ppisljar commented Nov 16, 2017

Precise Wake Word Listener #1199

Precise Wake Word Listener #1199

Conversation

MatthewScholefield commented Nov 3, 2017

forslund commented Nov 6, 2017

MatthewScholefield commented Nov 6, 2017

forslund commented Nov 6, 2017

MatthewScholefield commented Nov 6, 2017

forslund left a comment

Choose a reason for hiding this comment

forslund Nov 7, 2017

Choose a reason for hiding this comment

MatthewScholefield Nov 7, 2017 • edited

Choose a reason for hiding this comment

forslund Nov 7, 2017

Choose a reason for hiding this comment

MatthewScholefield Nov 7, 2017

Choose a reason for hiding this comment

forslund Nov 7, 2017

Choose a reason for hiding this comment

MatthewScholefield Nov 7, 2017 • edited

Choose a reason for hiding this comment

MatthewScholefield Nov 7, 2017 • edited

Choose a reason for hiding this comment

forslund commented Nov 7, 2017

JarbasAI commented Nov 7, 2017 • edited

MatthewScholefield commented Nov 7, 2017 • edited

JarbasAI commented Nov 7, 2017 • edited

MatthewScholefield commented Nov 8, 2017 • edited

forslund commented Nov 8, 2017 • edited

LearnedVector commented Nov 8, 2017

MatthewScholefield commented Nov 8, 2017 via email

forslund commented Nov 8, 2017

forslund commented Nov 8, 2017

LearnedVector commented Nov 8, 2017

MatthewScholefield commented Nov 8, 2017

forslund commented Nov 8, 2017

forslund commented Nov 8, 2017

MatthewScholefield commented Nov 9, 2017 • edited

ppisljar commented Nov 16, 2017

MatthewScholefield Nov 7, 2017 •

edited

MatthewScholefield Nov 7, 2017 •

edited

MatthewScholefield Nov 7, 2017 •

edited

JarbasAI commented Nov 7, 2017 •

edited

MatthewScholefield commented Nov 7, 2017 •

edited

JarbasAI commented Nov 7, 2017 •

edited

MatthewScholefield commented Nov 8, 2017 •

edited

forslund commented Nov 8, 2017 •

edited

MatthewScholefield commented Nov 9, 2017 •

edited