Unpacking RESTful sensor JSON results into attributes.

[nickovs]

Description:

Currently the RESTful sensor can only unpack a single JSON value using a template. There are many cases where the REST result contains multiple values. There are also many cases where making the request is costly, for example when accessing commercial or rate-limited APIs, fetching results from tiny, low power IoT devices or making the request across a slow network. As a result it is useful to be able to fetch all of the resulting values in a single request.

This patch adds a json_attributes configuration option to the RESTful sensor which indicates that the result of the REST request should be treated as a JSON dictionary and the keys in that dictionary should be mapped to attributes in the sensor. This allows all of the resulting values to be accessed in other parts of the system. Furthermore it allows new sensors to be created that monitor the value of individual attributes using the template sensor platform.

Related issue (if applicable): Addresses the feature request in the community thread "RESTful Sensor - read multiple values in one request".

Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.io#4046

Example entry for configuration.yaml:

By way of example, the configuration below creates a set of sensors for the temperature, pressure, humidity and weather conditions retrieved from OpenWeatherMap.org using a single REST call.

- platform: rest
  name: OWM_report
  json_attributes: True
  value_template: '{{value_json["weather"][0]["description"].title()}}'
  resource: http://api.openweathermap.org/data/2.5/weather?zip=80302,us&APPID=VERYSECRETAPIKEY

- platform: template
  sensors:
    owm_weather:
      value_template: '{{states.sensor.owm_report.status}}'
      icon_template: '{{"http://openweathermap.org/img/w/"+states.sensor.owm_report.attributes.weather[0]["icon"]+".png"}}'
      entity_id: sensor.owm_report
    owm_temp:
      friendly_name: 'Outside temp'
      value_template: '{{states.sensor.owm_report.attributes.main["temp"]-273.15}}'
      unit_of_measurement: "°C"
      entity_id: sensor.owm_report
    owm_pressure:
      friendly_name: 'Outside pressure'
      value_template: '{{states.sensor.owm_report.attributes.main["pressure"]}}'
      unit_of_measurement: "hP"
      entity_id: sensor.owm_test
    owm_humidity:
      friendly_name: 'Outside humidity'
      value_template: '{{states.sensor.owm_report.attributes.main["humidity"]}}'
      unit_of_measurement: "%"
      entity_id: sensor.owm_test

Checklist:

If user exposed functionality or configuration variables are added/changed:

• Documentation added/updated in home-assistant.github.io

If the code does not interact with devices:

• Local tests with tox run successfully. Your PR cannot be merged unless tests pass
• Tests have been added to verify that the new code works.

[homeassistant]

Hi @nickovs,

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[nickovs]

@tinloaf Thanks for the comments. Regarding your concerns:

1. The vast majority of REST API endpoints return JSON dictionaries and this is almost invariably the case for the sort of endpoints that for which people will use this option. If the returned result is not a dictionary then the code will log a warning and ignore the result.
2. It is possible that the REST endpoint will return a large result but the user doesn't have to use this if that's going to be a problem. It would be possible to have some arbitrary a cap on the complexity of the results that we might store but I submit that whatever value you choose for a cap like that there will be some people who think it is wrong. This is an optional setting so the user can always choose to do it the old way.
3. The existing RESTful sensor already allows the owner of the REST endpoint to insert content into frontend when people use the returned values. Attributes get escaped during rendering and if you use a template sensor to extract values from the attributes then you're in the same place that the current code puts you.

[nickovs]

@tinloaf: If the size of the JSON result is a serious concern then it would be possible to add another configuration option to specify either some dictionary sub-tree of the JSON object from which to extract values or a set of pairs of attribute names and templates to extract. That said, this seems overly complex to me, both in terms of code complexity and difficulty for the user to configure. The solution here has the advantage of being much simpler on both counts.

[tinloaf]

I'm not completely sure, someone with more knowledge of the state machine might have a better guess at that than I. The thing is: The state is currently automatically limited to 255 characters. Reading JSON into attributes might result in large amounts of data going through the state machine.

[nickovs]

@tinloaf: If the concern is about the size of data put into the state value then I could change the code to suppress the JSON result string from going into the state value directly (but still allow it through a template). The attributes are Python objects, not strings, so there is no size restriction. Looking at the events and states recorder database tables the database does not seem to record the full set of attributes, only the state values, so the volume of data that ends up in the database is exactly the same as if you use multiple RESTful sensors hitting the same REST API multiple times.

[pvizeli]

Make json_attr as list to select which attribute do you want

[pvizeli]

Remove this

[nickovs]

When you say "this", presumably you mean the self.update() rather than the whole section.

[pvizeli]

Revert the True at the end

[nickovs]

Sorry about that. The flag was added after I forked the code and I missed that in the merge.

[nickovs]

@pvizeli I've changed the code so that the json_attributes configuration parameter is now a list of keys to fetch from the a returned JSON dictionary that will be mapped as attributes of the sensor, instead of just being a boolean. I used the ensure_list_csv() function from the config validation helper code, so users can provide the list either as a YAML list or as a comma separated string.

The test cases have been updated accordingly and I've also pushed updated documentation.

The new version also fixes the two issues that you picked up that resulted from my manual merge to bring the base up to date with the 0.58 codebase.

Let me know if you have any other suggestions or requests.

[fabaff]

According to your docs the default is an empty list (-> default=[]). To be in sync with other platform we should only accept a list and not a comma-separated string ->vol.Optional(CONF_JSON_ATTRS): vol.All(cv.ensure_list, [cv.string]),

[nickovs]

I've added the default (although the existing code treats the a value of None the same as an empty list).

Regarding the use of cv.ensure_list_csv, is this function deprecated? The function seems to be there explicitly to allow users to enter lists either as a YAML list or as a CVS list, which is very helpful to users and allows them to choose the representation that they find most readable and compact. The function gets used by calendar/todoist.py, insteon_plm, sensor/mvglive.py and switch/rpi_rf.py and serves a very useful purpose. If the intention is to do way with this useful capability then I can remove it from here but if the function is going to stay around is there really a compelling reason not to use it?

[fabaff]

No, it's not deprecated and was introduced for the customizing part. I was wrong...it works for all possible ways:

    json_attributes:
      - "time"
      - "date"
      - "milliseconds_since_epoch"
    json_attributes: ["time", "date", "milliseconds_since_epoch"]
    json_attributes: time, date, milliseconds_since_epoch
    json_attributes: time,date,milliseconds_since_epoch
    json_attributes: "time", "date", "milliseconds_since_epoch"

But I still would prefer if we don't promote a, b, c because it will lead to confusion for users which are not that familiar with YAML and are looking at other components/platforms.

[nickovs]

Great. The documentation pull request that I submitted uses option d for the example but I'm happy to document it with whichever form you think should be promoted.

Comments addressed

[MartinHjelmare]

Looks like this branch history was messed up. We should probably revert it.