Snips ASR and NLU component #8156

michaelfester · 2017-06-22T13:27:53Z

Description:

Snips on-device ASR and NLU custom component.

For installation and setup of the Snips SDK, check out the Snips Installation Guide.

The Snips SDK must be installed and running locally, and Home Assistant should use the Snips SDK as an MQTT broker for simplicity. For instance, if the Snips platform is running locally on port 9898, the following should be added to configuration.yaml:

mqtt:
  broker: 127.0.0.1
  port: 9898

Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.io#2892

Example entry for `configuration.yaml`:

Just like Alexa and API.ai, Snips transforms voice and text into intents. Intents are created via the Snips Console, and can then be handled in Home Assistant by adding the intents, and accompanying actions, to the snips entry in configuration.yaml:

snips:
  intents:
    Lights:
      action:
        - service_template: >
            {%- if light_state == "on" -%}
              light.turn_on
            {%- else -%}
              light.turn_off
            {%- endif -%}
          data_template:
            entity_id: light.living_room

homeassistant · 2017-06-22T13:27:54Z

Hi @michaelfester,

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

houndci-bot · 2017-06-22T13:28:04Z

homeassistant/components/snips.py

+        except: pass
+        try: return slot["value"]
+        except: pass
+        return None


no newline at end of file

houndci-bot · 2017-06-22T13:28:04Z

homeassistant/components/snips.py

+        try: return slot["value"]["value"]
+        except: pass
+        try: return slot["value"]
+        except: pass


multiple statements on one line (colon)

houndci-bot · 2017-06-22T13:28:04Z

homeassistant/components/snips.py

+        except: pass
+        try: return slot["value"]["value"]
+        except: pass
+        try: return slot["value"]


multiple statements on one line (colon)

houndci-bot · 2017-06-22T13:28:04Z

homeassistant/components/snips.py

+        try: return slot["value"]["value"]["value"]
+        except: pass
+        try: return slot["value"]["value"]
+        except: pass


multiple statements on one line (colon)

houndci-bot · 2017-06-22T13:28:04Z

homeassistant/components/snips.py

+    def get_value(self, slot):
+        try: return slot["value"]["value"]["value"]
+        except: pass
+        try: return slot["value"]["value"]


multiple statements on one line (colon)

houndci-bot · 2017-06-22T13:28:04Z

homeassistant/components/snips.py

+        if not payload: return
+        response = json.loads(payload)
+        if not response: return
+


blank line contains whitespace

houndci-bot · 2017-06-22T13:28:05Z

homeassistant/components/snips.py

+    def handle_intent(self, payload):
+        if not payload: return
+        response = json.loads(payload)
+        if not response: return


multiple statements on one line (colon)

houndci-bot · 2017-06-22T13:28:05Z

homeassistant/components/snips.py

+        self.intents = intents
+
+    def handle_intent(self, payload):
+        if not payload: return


multiple statements on one line (colon)

houndci-bot · 2017-06-22T13:28:05Z

homeassistant/components/snips.py

+        for name, intent in intents.items():
+            if CONF_ACTION in intent:
+                a = intent[CONF_ACTION]
+                intent[CONF_ACTION] = script.Script(hass, intent[CONF_ACTION], "Snips intent {}".format(name))


line too long (110 > 79 characters)

houndci-bot · 2017-06-22T13:28:05Z

homeassistant/components/snips.py

+
+        for name, intent in intents.items():
+            if CONF_ACTION in intent:
+                a = intent[CONF_ACTION]


local variable 'a' is assigned to but never used

houndci-bot · 2017-06-22T13:39:11Z

homeassistant/components/snips.py

+        for name, intent in intents.items():
+            if CONF_ACTION in intent:
+                intent[CONF_ACTION] = script.Script(hass, intent[CONF_ACTION],
+                    "Snips intent {}".format(name))


continuation line under-indented for visual indent

houndci-bot · 2017-06-22T13:39:11Z

homeassistant/components/snips.py

+
+    return True
+
+class IntentHandler(object):


expected 2 blank lines, found 1

houndci-bot · 2017-06-22T13:39:11Z

homeassistant/components/snips.py

+    }
+}, extra=vol.ALLOW_EXTRA)
+
+def setup(hass, config):


expected 2 blank lines, found 1

houndci-bot · 2017-06-22T13:39:11Z

homeassistant/components/snips.py

+from homeassistant.helpers import template, script, config_validation as cv
+import homeassistant.loader as loader
+import voluptuous as vol
+import asyncio


'asyncio' imported but unused

balloob · 2017-06-24T06:43:46Z

Very cool! Looking forward to Snips support.

This will also require a test to be written. Could be something like this (untested):

import asyncio

from homeassistant.bootstrap import async_setup_component
from tests.common import async_fire_mqtt_message, mock_service

EXAMPLE_MSG = """
{
  // snips message from hermes/nlu/intentParsed for intent Lights
}
"""

@asyncio.coroutine
def test_snips_call_action(hass, mqtt_mock):
    """Test calling action via Snips."""
    calls = mock_service(hass, 'test', 'service')

    result = yield from async_setup_component(hass, 'snips', {
        'snips': {
            'intents': {
                'Lights': {
                    'action': {
                        'service': 'test.service'
                    }
                }
            }
        }
    })
    assert result

    async_fire_mqtt_message(hass, 'hermes/nlu/intentParsed',
                            EXAMPLE_MSG)
    yield from hass.async_block_till_done()

    assert len(calls) == 1

balloob

Off to a good start, will need some more work. There is also a bunch of lint issues that need to be addressed https://travis-ci.org/home-assistant/home-assistant/jobs/245793758#L282

balloob · 2017-06-24T06:39:41Z

homeassistant/components/snips.py

+DOMAIN = 'snips'
+DEPENDENCIES = ['mqtt']
+CONF_TOPIC = 'topic'
+DEFAULT_TOPIC = '#'


Since message_received only cares about messages from hermes/nlu/intentParsed, better to just listen to that topic instead and not allow it to be configured?

balloob · 2017-06-24T06:40:24Z

homeassistant/components/snips.py

+
+    def message_received(topic, payload, qos):
+        if topic == 'hermes/nlu/intentParsed':
+            LOGGER.info("New intent: {}".format(payload))


Please change this to debug.

balloob · 2017-06-24T06:40:30Z

homeassistant/components/snips.py

+
+
+def setup(hass, config):
+    LOGGER.info("The 'snips' component is ready!")


Please remove this.

balloob · 2017-06-24T06:40:36Z

homeassistant/components/snips.py

+            LOGGER.info("New intent: {}".format(payload))
+            handler.handle_intent(payload)
+
+    LOGGER.info("Subscribing to topic " + str(topic))


Please remove this.

balloob · 2017-06-24T06:41:42Z

homeassistant/components/snips.py

+    def handle_intent(self, payload):
+        if not payload:
+            return
+        response = json.loads(payload)


This can raise a TypeError if invalid JSON.

balloob · 2017-06-24T06:45:34Z

homeassistant/components/snips.py

+
+    def get_name(self, response):
+        try:
+            return response['intent']['intentName'].split('__')[-1]


It's probably the easiest to write a voluptuous schema to validate the schema that Snips is sending. That way you can get rid of all other checks if some piece of data is available.

balloob · 2017-06-24T06:46:02Z

homeassistant/components/snips.py

+    def get_name(self, response):
+        try:
+            return response['intent']['intentName'].split('__')[-1]
+        except:


Catch KeyError. Bare exceptions are not allowed.

balloob · 2017-06-24T06:46:42Z

homeassistant/components/snips.py

+
+    def parse_slots(self, response):
+        slots = response["slots"]
+        if not slots:


This is not possible. Python will raise if a key does not exists. Unless the JSON value of slots can be set to null ? Again, probably easier to write a voluptuous schema to validate this info.

balloob · 2017-06-24T06:47:10Z

homeassistant/components/snips.py

+
+    def get_value(self, slot):
+        try:
+            return slot["value"]["value"]["value"]


Errrrrr, what is happening here? 🤔

houndci-bot · 2017-06-25T13:16:13Z

tests/components/test_snips.py

+
+@asyncio.coroutine
+def test_snips_call_action(hass, mqtt_mock):
+    """Test calling action via Snips."""


indentation contains mixed spaces and tabs

houndci-bot · 2017-06-25T13:16:13Z

tests/components/test_snips.py

+}
+"""
+
+@asyncio.coroutine


expected 2 blank lines, found 1

houndci-bot · 2017-06-25T13:16:13Z

tests/components/test_snips.py

+				"value": "green"
+			},
+		}
+	]


indentation contains tabs

houndci-bot · 2017-06-25T13:16:13Z

tests/components/test_snips.py

+				"kind": "Custom",
+				"value": "green"
+			},
+		}


indentation contains tabs

houndci-bot · 2017-06-25T13:16:14Z

tests/components/test_snips.py

+			"value": {
+				"kind": "Custom",
+				"value": "green"
+			},


indentation contains tabs

houndci-bot · 2017-06-25T13:16:14Z

tests/components/test_snips.py

+		"intent_name": "Lights",
+		"probability": 1
+	},
+	"slots": [


indentation contains tabs

houndci-bot · 2017-06-25T13:16:14Z

tests/components/test_snips.py

+	"intent": {
+		"intent_name": "Lights",
+		"probability": 1
+	},


indentation contains tabs

houndci-bot · 2017-06-25T13:16:14Z

tests/components/test_snips.py

+	"text": "turn the lights green",
+	"intent": {
+		"intent_name": "Lights",
+		"probability": 1


indentation contains tabs

houndci-bot · 2017-06-25T13:16:14Z

tests/components/test_snips.py

+{
+	"text": "turn the lights green",
+	"intent": {
+		"intent_name": "Lights",


indentation contains tabs

houndci-bot · 2017-06-25T13:16:14Z

tests/components/test_snips.py

+EXAMPLE_MSG = """
+{
+	"text": "turn the lights green",
+	"intent": {


indentation contains tabs

bastshoes · 2017-06-30T19:13:40Z

Hello.
In most cases HASS users already have MQTT installed in their setup. Currently HASS do not support multiple MQTT brokers.
As I understand from documentation Snips Dockers image includes MQTT broker and it available on port 9898.
Is it possible to use existing broker with Snips component?

michaelfester · 2017-07-01T11:00:30Z

Hi @bastshoes! Currently this is not possible but we are fixing this :)

balloob · 2017-07-01T20:15:13Z

Component never worked because the name extraction code was wrong. It referenced intentName instead of intent_name (as per Snips docs). Slot parsing had the same problem. You referenced slotName instead of slot_name (as per Snips docs).

Another issue was that mock_service was no longer allowed to be called in an async context. Rebased the branch and imported async_mock_service instead.

Converted it to async because there was no reason to use threads.

I've added some schema validation to validate incoming data instead of having to check each value individually.

Now all tests should pass and this is good to go.

balloob · 2017-07-01T20:58:42Z

Cherry-picked for 0.48.

* Snips ASR and NLU component * Fix warning * Fix warnings * Fix lint issues * Add tests * Fix tabs * Fix newline * Fix quotes * Fix docstrings * Update tests * Remove logs * Fix lint warning * Update API * Fix Snips

michaelfester · 2017-07-02T09:26:42Z

Excellent stuff. Thanks @balloob!

Mika56 · 2017-07-02T11:32:29Z

Are you certain about the intentName and slotName keys?
I had multiple errors (Intent has invalid schema: required key not provided @ data['slots'][0]['slot_name'], Intent has invalid schema: required key not provided @ data['text'], ...)

I had to build a MQTT bridge that adapts the output from Snips:

var mqtt = require('mqtt');

var snipsClient = mqtt.connect('mqtt://localhost:9898');
var hassClient = mqtt.connect('mqtt://192.168.1.247:1884');

snipsClient.on('connect', function() { 
	console.log('Connected to Snips broker'); 
});

hassClient.on('connect', function() {
	console.log('Connected to HASS broker');
});

snipsClient.on('message', function(topic, message) {
	console.log(' < '+topic);
	message = message.toString();
	if(message) {
		message = JSON.parse(message);
		//console.log(message);
		if(message.hasOwnProperty('intent')) {
			message.intent.intent_name = message.intent.intentName;
		}
		if(message.hasOwnProperty('slots')) {
			for(var i=0;i<message.slots.length;i++) {
				message.slots[i].slot_name = message.slots[i].slotName;
			}
		}
		if(message.hasOwnProperty('input')) {
			message.text = message.input;
		}
		//console.log(message);
		hassClient.publish(topic, JSON.stringify(message));
	}
	else {
		hassClient.publish(topic, message);
	}
});

snipsClient.subscribe('#');

balloob · 2017-07-02T18:37:24Z

@Mika56 that's weird. I see intent_name and slot_name used all over the docs https://github.com/snipsco/snips-platform-documentation/wiki/5.-Learn-more:-Key-Concepts#3-natural-language-understanding and https://medium.com/snips-ai/integrating-snips-with-home-assistant-314723645c77#6bac

I do admit that I didn't test it with an actual Snips instance.

@michaelfester can you chime in what the right variable names are ?

text or input
slot_name or slotName
intent_name or intentName

michaelfester · 2017-07-07T15:17:40Z

Yes, change of API. Update throughout :)

* Snips ASR and NLU component * Fix warning * Fix warnings * Fix lint issues * Add tests * Fix tabs * Fix newline * Fix quotes * Fix docstrings * Update tests * Remove logs * Fix lint warning * Update API * Fix Snips

homeassistant added new-platform integration: snips cla-needed labels Jun 22, 2017

balloobbot added the component label Jun 22, 2017

houndci-bot reviewed Jun 22, 2017

View reviewed changes

homeassistant added cla-signed and removed cla-needed labels Jun 22, 2017

houndci-bot reviewed Jun 22, 2017

View reviewed changes

balloob requested changes Jun 24, 2017

View reviewed changes

houndci-bot reviewed Jun 25, 2017

View reviewed changes

home-assistant deleted a comment from houndci-bot Jun 25, 2017

michaelfester mentioned this pull request Jun 27, 2017

Documentation for Snips components home-assistant/home-assistant.io#2892

Merged

homeassistant added the cla-signed label Jun 27, 2017

michaelfester added 13 commits July 1, 2017 12:38

Snips ASR and NLU component

14465e7

Fix warning

b56d51d

Fix warnings

b1e7b3e

Fix lint issues

33b686d

Add tests

af57e0a

Fix tabs

985e809

Fix newline

0006512

Fix quotes

ef07800

Fix docstrings

54800e9

Update tests

de2a3d4

Remove logs

dc08506

Fix lint warning

dc8c886

Update API

eedc1b0

balloob force-pushed the snips-component-integration branch from c3bc81c to 4241d4e Compare July 1, 2017 20:13

homeassistant added the cla-signed label Jul 1, 2017

balloob added this to the 0.48 milestone Jul 1, 2017

balloob approved these changes Jul 1, 2017

View reviewed changes

balloob force-pushed the snips-component-integration branch from 4241d4e to 858c354 Compare July 1, 2017 20:16

homeassistant added the cla-signed label Jul 1, 2017

Fix Snips

ffab784

balloob force-pushed the snips-component-integration branch from 858c354 to ffab784 Compare July 1, 2017 20:27

homeassistant added the cla-signed label Jul 1, 2017

balloob merged commit b82003a into home-assistant:dev Jul 1, 2017

balloob mentioned this pull request Jul 1, 2017

0.48 #8270

Merged

adrienball mentioned this pull request Jul 3, 2017

Fix Snips json schema #8317

Merged

balloob mentioned this pull request Jul 13, 2017

0.49 #8468

Merged

balloob added the cherry-picked label Jul 13, 2017

home-assistant locked and limited conversation to collaborators Oct 20, 2017



		def setup(hass, config):
		LOGGER.info("The 'snips' component is ready!")

Snips ASR and NLU component #8156

Snips ASR and NLU component #8156

Conversation

michaelfester commented Jun 22, 2017 • edited by fabaff

Description:

Example entry for configuration.yaml:

homeassistant commented Jun 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balloob commented Jun 24, 2017

balloob left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bastshoes commented Jun 30, 2017

michaelfester commented Jul 1, 2017 • edited

balloob commented Jul 1, 2017 • edited

balloob commented Jul 1, 2017

michaelfester commented Jul 2, 2017

Mika56 commented Jul 2, 2017

balloob commented Jul 2, 2017 • edited

michaelfester commented Jul 7, 2017

michaelfester commented Jun 22, 2017 •

edited by fabaff

Example entry for `configuration.yaml`:

michaelfester commented Jul 1, 2017 •

edited

balloob commented Jul 1, 2017 •

edited

balloob commented Jul 2, 2017 •

edited