Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the Assistant Library for OK Google hotwording #64

Merged
merged 4 commits into from May 31, 2017

Conversation

drigz
Copy link
Member

@drigz drigz commented May 25, 2017

This requires installation of the wheel for the Google Assistant
Library, which includes a closed-source hotword implementation.

drigz added 2 commits May 25, 2017
This allows the user to configure responses on IFTTT, so that local
actions have an IFTTT response with the Assistant's voice.
This requires installation of the wheel for the Google Assistant
Library, which includes a closed-source hotword implementation.
@drigz drigz requested review from proppy and ensonic May 25, 2017
src/main.py Outdated

# The ok-google trigger is handled with the Assistant Library, so we need
# to catch this case early.
if args.trigger == 'ok-google':
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be nice to also make the button work with the library (it supports start_conversation).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@proppy not sure if it helps; but, I pulled down this branch and tested it. is this the start-conversation you are talking about?

"Hey Google, how far away is Japan?"
< "Japan is 6xxx miles away as way of the crow flies."
"Hey Google, and from California?"
< "Japan is 5xxx miles away from California."

console output below. notice the "how far away is Japan" and the "and from California" follow up:

$ src/main.py --trigger="ok-google"
[2017-05-26 22:25:17,165] INFO:root:ON_MUTED_CHANGED:
  {'is_muted': False}
[2017-05-26 22:25:17,166] INFO:root:ON_START_FINISHED
[2017-05-26 22:25:17,167] INFO:main:ready...
Say "OK, Google" then speak, or press Ctrl+C to quit...
[2017-05-26 22:25:24,515] INFO:root:ON_CONVERSATION_TURN_STARTED
[2017-05-26 22:25:24,516] INFO:main:listening...
[2017-05-26 22:25:28,365] INFO:root:ON_END_OF_UTTERANCE
[2017-05-26 22:25:28,366] INFO:main:thinking...
[2017-05-26 22:25:28,385] INFO:root:ON_RECOGNIZING_SPEECH_FINISHED:
  {'text': 'how far away is Japan'}
[2017-05-26 22:25:28,907] INFO:root:ON_RESPONDING_STARTED:
  {'is_error_response': False}
[2017-05-26 22:25:33,156] INFO:root:ON_RESPONDING_FINISHED
[2017-05-26 22:25:33,157] INFO:root:ON_CONVERSATION_TURN_FINISHED:
  {'with_follow_on_turn': False}
[2017-05-26 22:25:33,158] INFO:main:ready...
[2017-05-26 22:25:34,972] INFO:root:ON_CONVERSATION_TURN_STARTED
[2017-05-26 22:25:34,974] INFO:main:listening...
[2017-05-26 22:25:37,748] INFO:root:ON_END_OF_UTTERANCE
[2017-05-26 22:25:37,749] INFO:main:thinking...
[2017-05-26 22:25:37,784] INFO:root:ON_RECOGNIZING_SPEECH_FINISHED:
  {'text': 'and from California'}
[2017-05-26 22:25:38,856] INFO:root:ON_RESPONDING_STARTED:
  {'is_error_response': False}
[2017-05-26 22:25:42,704] INFO:root:ON_RESPONDING_FINISHED
[2017-05-26 22:25:42,705] INFO:root:ON_CONVERSATION_TURN_FINISHED:
  {'with_follow_on_turn': False}
[2017-05-26 22:25:42,707] INFO:main:ready...

Also, I played the Trivia game and answered five questions in turn.

"Hey Google, let's play a game."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided against this as it seemed to hard to document having three different backends and three different triggers, but hotword trigger only works with one backend.

Copy link
Contributor

@ensonic ensonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for giving it a try!

@@ -33,6 +33,7 @@ sudo pip3 install --upgrade pip virtualenv
cd "${scripts_dir}/.."
virtualenv --system-site-packages -p python3 env
env/bin/pip install -r requirements.txt
env/bin/pip install --upgrade https://github.com/googlesamples/assistant-sdk-python/releases/download/0.3.0/google_assistant_library-0.0.2-py2.py3-none-linux_armv7l.whl
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this go to requirements.txt?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wheels is available on pypi now:
https://pypi.python.org/pypi/google-assistant-library


# Uncomment to play Assistant responses for local actions. You should make
# sure that you have IFTTT applets for your actions to get the correct
# response, and also that your actions do not call say().
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove one of the leading spaces in the two lines above?

@@ -33,6 +33,16 @@ def get_phrases(self):
"""Get a list of all phrases that are expected by the handlers."""
return [phrase for h in self.handlers for phrase in h.get_phrases()]

def would_handle(self, command):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can_handle()? Same below.

src/main.py Outdated
@@ -121,7 +121,7 @@ def main():
parser.add_argument('-O', '--output-device', default='default',
help='Name of the audio output device')
parser.add_argument('-T', '--trigger', default='gpio',
help='Trigger to use {\'clap\', \'gpio\'}')
help='Trigger to use {"clap", "gpio", "ok-google"}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can do: choices=['clap', 'gpio', 'ok-google'],
https://docs.python.org/2.7/library/argparse.html#choices

src/main.py Outdated
print(' https://developers.google.com/assistant/sdk/prototype/'
'getting-started-pi-python/run-sample'
'#get_the_library_and_sample_code')
raise
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we rather point to our requirements.txt?

@@ -204,6 +273,45 @@ def do_recognition(args, recorder, recognizer, player):
time.sleep(1)


class StatusUi(object):

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove blank line before doc string

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blank-line-before-docstring style is used consistently at the project. However, it seems it's based on an old version of PEP8: hhatto/autopep8#194

I'd prefer to fix in a another PR.

src/main.py Outdated
"""Gives the user status feedback.

The LED and optionally a trigger sound tell the user when the box is
ready, listening and thinking.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"... when the box is ready, listening or thinking."?

@codecov-io
Copy link

codecov-io commented May 30, 2017

Codecov Report

Merging #64 into master will increase coverage by 0.19%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #64      +/-   ##
=========================================
+ Coverage    9.76%   9.95%   +0.19%     
=========================================
  Files           3       3              
  Lines        1720    1727       +7     
  Branches      295     297       +2     
=========================================
+ Hits          168     172       +4     
- Misses       1542    1543       +1     
- Partials       10      12       +2
Impacted Files Coverage Δ
src/actionbase.py 90.32% <66.66%> (-9.68%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 250b8cb...7fd0903. Read the comment docs.

@drigz
Copy link
Member Author

drigz commented May 30, 2017

PTAL

@sheridat
Copy link

sheridat commented May 30, 2017

I am a newcomer - so this question might not make a lot of sense but the sentence above "This requires installation of the wheel for the Google Assistant Library, which includes a closed-source hotword implementation" has me worried.

I associate wheel with updating python libraries via pip.

People are aware that there is a new version of the google assistant SDK is available BUT that it is for armv7 capable processors. People are also aware that it supports "hotword"

Many of us are running the AIY software on Pi Zero's and Pi W's which are armv6 based and therefore we cannot use that new version of the the assistant SDK.

My worry is that the change you are making here means that once it is merged - if I was to git pull this repository and run the "install deps script" the google assistant software will be updated and my installation will stop working because the google assistant will want an armv7 processor.

But as we say in england - I may be (entirely) barking up the wrong tree.

@eduncan911
Copy link

You have chased the cat up the correct tree.

The SDK update in this PR will restrict installation in ARMv7 going forward.

The binary is specific to detecting the hot words, OK Google and Hey Google. They have rigorously tested it for false positives. But why they kept it closed sourced we don't know. For ARMv7 though, they mention they are leveraging some of the newer CPU instructions to speed things up.

To address your concern though, just don't upgrade. Or, fork it and use your on branch. You can merge changes in, but kept the wheel out of your branch.

@drigz
Copy link
Member Author

drigz commented May 30, 2017

We could remove google-assistant-library from requirements.txt so that it's still usable on ARMv6, (maybe ok_google_requirements.txt?) and amend the error message to say "ok-google triggering is not available on your platform" for ARMv6 etc, and "Please pip install -r ok_google_requirements.txt" for ARMv7. What do you think?

@sheridat
Copy link

ummmmmhhhhhhhhhhh. Is it realistic for the newcomers to start using Git to maintain their own fork/branch and maintain it. We are struggling with how to update it. A compromise such as that outlined by Drigz seems a better solution and only a little inconvenient for people using the armv7 processor. I would hate to think that some kids using this on a Zero or W suddenly need to become Git gurus.

@eduncan911
Copy link

eduncan911 commented May 30, 2017

I know the struggles quite well, as I am not a Python guru myself and stumble through most of this.

I think this discussion really belongs over at the Assistant SDK repo though, not in this PR:

https://github.com/googlesamples/assistant-sdk-python

You can read a marketing overview of what the Assistant does (or will do soon):

https://developers.google.com/assistant/sdk/

Basically, the Assistant SDK does a lot more than just hot wording. It is what does the interrupting of the vocal commands, for example. Removing it would pretty much render this "Voice Recognizer" project dead and useless. The "hot wording" is just one feature in a set of a much larger feature set that this repo is leveraging.

This repo is basically just a wrapper framework with some triggers and actions to leverage the Assistant SDK core library.

@sheridat
Copy link

I see - and I can see the clamour for new features.

@drigz
Copy link
Member Author

drigz commented May 31, 2017

@ensonic, the library is now optional. Users who git pull and try to use -T ok-google will get a message saying the dep is only for Pi 2/3, and how to install it.

WDYT?

@drigz drigz merged commit 5b2cd9c into google:master May 31, 2017
@sheridat
Copy link

sheridat commented Jun 1, 2017

Hi, I've closed my issue above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants