Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the Assistant Library for OK Google hotwording #64

Merged
merged 4 commits into from May 31, 2017

Conversation

@drigz
Copy link
Member

drigz commented May 25, 2017

This requires installation of the wheel for the Google Assistant
Library, which includes a closed-source hotword implementation.

drigz added 2 commits May 25, 2017
This allows the user to configure responses on IFTTT, so that local
actions have an IFTTT response with the Assistant's voice.
This requires installation of the wheel for the Google Assistant
Library, which includes a closed-source hotword implementation.
@drigz drigz requested review from proppy and ensonic May 25, 2017
@googlebot googlebot added the cla: yes label May 25, 2017
src/main.py Outdated

# The ok-google trigger is handled with the Assistant Library, so we need
# to catch this case early.
if args.trigger == 'ok-google':

This comment has been minimized.

@proppy

proppy May 25, 2017 Member

It'd be nice to also make the button work with the library (it supports start_conversation).

This comment has been minimized.

@eduncan911

eduncan911 May 27, 2017

@proppy not sure if it helps; but, I pulled down this branch and tested it. is this the start-conversation you are talking about?

"Hey Google, how far away is Japan?"
< "Japan is 6xxx miles away as way of the crow flies."
"Hey Google, and from California?"
< "Japan is 5xxx miles away from California."

console output below. notice the "how far away is Japan" and the "and from California" follow up:

$ src/main.py --trigger="ok-google"
[2017-05-26 22:25:17,165] INFO:root:ON_MUTED_CHANGED:
  {'is_muted': False}
[2017-05-26 22:25:17,166] INFO:root:ON_START_FINISHED
[2017-05-26 22:25:17,167] INFO:main:ready...
Say "OK, Google" then speak, or press Ctrl+C to quit...
[2017-05-26 22:25:24,515] INFO:root:ON_CONVERSATION_TURN_STARTED
[2017-05-26 22:25:24,516] INFO:main:listening...
[2017-05-26 22:25:28,365] INFO:root:ON_END_OF_UTTERANCE
[2017-05-26 22:25:28,366] INFO:main:thinking...
[2017-05-26 22:25:28,385] INFO:root:ON_RECOGNIZING_SPEECH_FINISHED:
  {'text': 'how far away is Japan'}
[2017-05-26 22:25:28,907] INFO:root:ON_RESPONDING_STARTED:
  {'is_error_response': False}
[2017-05-26 22:25:33,156] INFO:root:ON_RESPONDING_FINISHED
[2017-05-26 22:25:33,157] INFO:root:ON_CONVERSATION_TURN_FINISHED:
  {'with_follow_on_turn': False}
[2017-05-26 22:25:33,158] INFO:main:ready...
[2017-05-26 22:25:34,972] INFO:root:ON_CONVERSATION_TURN_STARTED
[2017-05-26 22:25:34,974] INFO:main:listening...
[2017-05-26 22:25:37,748] INFO:root:ON_END_OF_UTTERANCE
[2017-05-26 22:25:37,749] INFO:main:thinking...
[2017-05-26 22:25:37,784] INFO:root:ON_RECOGNIZING_SPEECH_FINISHED:
  {'text': 'and from California'}
[2017-05-26 22:25:38,856] INFO:root:ON_RESPONDING_STARTED:
  {'is_error_response': False}
[2017-05-26 22:25:42,704] INFO:root:ON_RESPONDING_FINISHED
[2017-05-26 22:25:42,705] INFO:root:ON_CONVERSATION_TURN_FINISHED:
  {'with_follow_on_turn': False}
[2017-05-26 22:25:42,707] INFO:main:ready...

Also, I played the Trivia game and answered five questions in turn.

"Hey Google, let's play a game."

This comment has been minimized.

@drigz

drigz May 30, 2017 Author Member

I decided against this as it seemed to hard to document having three different backends and three different triggers, but hotword trigger only works with one backend.

Copy link
Collaborator

ensonic left a comment

Thanks for giving it a try!

@@ -33,6 +33,7 @@ sudo pip3 install --upgrade pip virtualenv
cd "${scripts_dir}/.."
virtualenv --system-site-packages -p python3 env
env/bin/pip install -r requirements.txt
env/bin/pip install --upgrade https://github.com/googlesamples/assistant-sdk-python/releases/download/0.3.0/google_assistant_library-0.0.2-py2.py3-none-linux_armv7l.whl

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

Shouldn't this go to requirements.txt?

This comment has been minimized.


# Uncomment to play Assistant responses for local actions. You should make
# sure that you have IFTTT applets for your actions to get the correct
# response, and also that your actions do not call say().

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

remove one of the leading spaces in the two lines above?

@@ -33,6 +33,16 @@ def get_phrases(self):
"""Get a list of all phrases that are expected by the handlers."""
return [phrase for h in self.handlers for phrase in h.get_phrases()]

def would_handle(self, command):

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

can_handle()? Same below.

src/main.py Outdated
@@ -121,7 +121,7 @@ def main():
parser.add_argument('-O', '--output-device', default='default',
help='Name of the audio output device')
parser.add_argument('-T', '--trigger', default='gpio',
help='Trigger to use {\'clap\', \'gpio\'}')
help='Trigger to use {"clap", "gpio", "ok-google"}')

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

We can do: choices=['clap', 'gpio', 'ok-google'],
https://docs.python.org/2.7/library/argparse.html#choices

src/main.py Outdated
print(' https://developers.google.com/assistant/sdk/prototype/'
'getting-started-pi-python/run-sample'
'#get_the_library_and_sample_code')
raise

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

Shouldn't we rather point to our requirements.txt?

@@ -204,6 +273,45 @@ def do_recognition(args, recorder, recognizer, player):
time.sleep(1)


class StatusUi(object):

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

remove blank line before doc string

This comment has been minimized.

@drigz

drigz May 30, 2017 Author Member

The blank-line-before-docstring style is used consistently at the project. However, it seems it's based on an old version of PEP8: hhatto/autopep8#194

I'd prefer to fix in a another PR.

src/main.py Outdated
"""Gives the user status feedback.
The LED and optionally a trigger sound tell the user when the box is
ready, listening and thinking.

This comment has been minimized.

@ensonic

ensonic May 26, 2017 Collaborator

"... when the box is ready, listening or thinking."?

@codecov-io
Copy link

codecov-io commented May 30, 2017

Codecov Report

Merging #64 into master will increase coverage by 0.19%.
The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #64      +/-   ##
=========================================
+ Coverage    9.76%   9.95%   +0.19%     
=========================================
  Files           3       3              
  Lines        1720    1727       +7     
  Branches      295     297       +2     
=========================================
+ Hits          168     172       +4     
- Misses       1542    1543       +1     
- Partials       10      12       +2
Impacted Files Coverage Δ
src/actionbase.py 90.32% <66.66%> (-9.68%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 250b8cb...7fd0903. Read the comment docs.

@drigz
Copy link
Member Author

drigz commented May 30, 2017

PTAL

@sheridat
Copy link

sheridat commented May 30, 2017

I am a newcomer - so this question might not make a lot of sense but the sentence above "This requires installation of the wheel for the Google Assistant Library, which includes a closed-source hotword implementation" has me worried.

I associate wheel with updating python libraries via pip.

People are aware that there is a new version of the google assistant SDK is available BUT that it is for armv7 capable processors. People are also aware that it supports "hotword"

Many of us are running the AIY software on Pi Zero's and Pi W's which are armv6 based and therefore we cannot use that new version of the the assistant SDK.

My worry is that the change you are making here means that once it is merged - if I was to git pull this repository and run the "install deps script" the google assistant software will be updated and my installation will stop working because the google assistant will want an armv7 processor.

But as we say in england - I may be (entirely) barking up the wrong tree.

@eduncan911
Copy link

eduncan911 commented May 30, 2017

You have chased the cat up the correct tree.

The SDK update in this PR will restrict installation in ARMv7 going forward.

The binary is specific to detecting the hot words, OK Google and Hey Google. They have rigorously tested it for false positives. But why they kept it closed sourced we don't know. For ARMv7 though, they mention they are leveraging some of the newer CPU instructions to speed things up.

To address your concern though, just don't upgrade. Or, fork it and use your on branch. You can merge changes in, but kept the wheel out of your branch.

@drigz
Copy link
Member Author

drigz commented May 30, 2017

We could remove google-assistant-library from requirements.txt so that it's still usable on ARMv6, (maybe ok_google_requirements.txt?) and amend the error message to say "ok-google triggering is not available on your platform" for ARMv6 etc, and "Please pip install -r ok_google_requirements.txt" for ARMv7. What do you think?

@sheridat
Copy link

sheridat commented May 30, 2017

ummmmmhhhhhhhhhhh. Is it realistic for the newcomers to start using Git to maintain their own fork/branch and maintain it. We are struggling with how to update it. A compromise such as that outlined by Drigz seems a better solution and only a little inconvenient for people using the armv7 processor. I would hate to think that some kids using this on a Zero or W suddenly need to become Git gurus.

@eduncan911
Copy link

eduncan911 commented May 30, 2017

I know the struggles quite well, as I am not a Python guru myself and stumble through most of this.

I think this discussion really belongs over at the Assistant SDK repo though, not in this PR:

https://github.com/googlesamples/assistant-sdk-python

You can read a marketing overview of what the Assistant does (or will do soon):

https://developers.google.com/assistant/sdk/

Basically, the Assistant SDK does a lot more than just hot wording. It is what does the interrupting of the vocal commands, for example. Removing it would pretty much render this "Voice Recognizer" project dead and useless. The "hot wording" is just one feature in a set of a much larger feature set that this repo is leveraging.

This repo is basically just a wrapper framework with some triggers and actions to leverage the Assistant SDK core library.

@sheridat
Copy link

sheridat commented May 30, 2017

I see - and I can see the clamour for new features.

@drigz
Copy link
Member Author

drigz commented May 31, 2017

@ensonic, the library is now optional. Users who git pull and try to use -T ok-google will get a message saying the dep is only for Pi 2/3, and how to install it.

WDYT?

@drigz drigz merged commit 5b2cd9c into google:master May 31, 2017
2 checks passed
2 checks passed
cla/google All necessary CLAs are signed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@sheridat
Copy link

sheridat commented Jun 1, 2017

Hi, I've closed my issue above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
You can’t perform that action at this time.