Multiple queries / sensors

[michaeldavie]

There's been some discussion around supporting multiple sensors that cover different periods, e.g. month-to-date, last 24 hours, etc. I had built these into my original custom component, and I'd like to discuss how to extend your library / component to do the same.

The approach I took was to code a few queries into the library, and then create a sensor for each query returned on the Home Assistant side. The set of queries I used was largely based on what is available in the Flume iOS app. I think we could adapt your library to do the same, while retaining the existing scan_interval minutes-to-now for compatibility.

Let me know what you think, and where I might be able to help.

[bachya]

I am interested in this, as well, and can assist in sniffing traffic, architecture, etc. if needed!

[ChrisMandich]

@michaeldavie I'd recommend using the statistics or utility meter sensor to aggregate the queries. @bdraco curious at what your thoughts are to this as you have been contributing heavily to this project.

[bdraco]

The flume api is pretty sluggish sometimes so if you can minimize the load of multiple queries on their systems I think it would be better for everyone.

https://flumetech.readme.io/reference#query-a-user-device

The api does support sending multiple queries at once so ideally any changes will coalesce the queries into a single call.

[ChrisMandich]

Disregard this message, this doesn't work. Will be updating the HASS integration to support these requests.

Using the utility meter the following settings worked for me. Although, it's only from the date configured forward. I've only had it running for a couple of hours but the hourly sensor appears to match the Flume website.

utility_meter:
  flume_sensor_hourly:
    source: sensor.flume_sensor_home
    cycle: hourly
  flume_sensor_daily:
    source: sensor.flume_sensor_home
    cycle: daily
  flume_sensor_weekly:
    source: sensor.flume_sensor_home
    cycle: weekly
  flume_sensor_monthly:
    source: sensor.flume_sensor_home
    cycle: monthly

[ChrisMandich]

I take that back, Utility meter and the flume sensor are fairly inaccurate.

[bachya]

I did some research today and want to post my findings (as well as my thoughts for what improvements could be made to help this create a top-tier Home Assistant integration, assuming that's the intended use case). I'll probably duplicate knowledge that @michaeldavie has already thought about via his custom component or what everyone here might already know; apologies. 😄

In my testing, I did not experience the API sluggishness that @bdraco mentions; each request/response took a max of 2-ish seconds. I think this is reasonable. If more speed were desired, you could look at using aiohttp instead of requests (although that would be a paradigm shift to asyncio – not a trivial decision).

API Research

I see 7 primary API calls (many of which are already somewhat present here):

1. POST /oauth/token: gets access and refresh tokens
2. GET /users/<USER_ID>: gets information related to the user's account
3. GET /users/<USER_ID>/locations: gets a list of all locations belonging to that user account
4. GET /users/<USER_ID>/notifications: gets a list of notifications for a user account
5. GET /users/<USER_ID>/devices: gets a list of devices belonging to a user account
6. GET /users/<USER_ID>/devices/<DEVICE_ID>/budgets: gets "budget" info (thresholds, actual vs. anticipated values, etc.) for a device.
7. POST /users/<USER_ID>/devices/<DEVICE_ID>/query: a wildly generic API route that allows data payloads to be built via a series of one or more queries.

Implementation Thoughts

I see a variety of things that this library could do in support of a Home Assistant integration (beyond that already-great work being done):

Translate Flume notifications into Home Assistant events.

In this scenario, the GET /users/<USER_ID>/notifications could be monitored for events that haven't already been "processed" by the Home Assistant integration; upon finding one, Home Assistant could be instructed to fire an event on its event bus. This would allow users to write automations based on these notifications (e.g., Send me a Slack message when any new Flume notification comes through).

Add device_registry support in Home Assistant.

It appears that the response for GET /users/<USER_ID>/devices includes the Flume base station in addition to the water meter sensor. This could be used to add device info for each Flume sensor (which could support future goals of Home Assistant, such as dynamically mapping entities against their parent device). More info: https://developers.home-assistant.io/docs/device_registry_index/

Add device budget info as sensors.

The GET /users/<USER_ID>/devices/<DEVICE_ID>/budgets endpoint creates a lot of great data – at first glance:

• Budget name
• Budget value and thresholds
• Current value

Exposing these could lead to great automation possibilities, such as sending alerts when actual usage exceeds a budgeted value.

Go crazy with queries.

As I mentioned (and you all likely know), the POST /users/<USER_ID>/devices/<DEVICE_ID>/query endpoint is a very generic endpoint. That said, the possibilities are limitless. For instance, if I pass a query (which is just a JSON payload) that looks like this:

{
  "queries": [
    {
      "bucket": "DAY",
      "units": "GALLONS",
      "since_datetime": "2020-04-14 00:00:00",
      "request_id": "totalForDay",
      "operation": "SUM"
    },
    {
      "request_id": "averageForTrailing10Days",
      "units": "GALLONS",
      "bucket": "DAY",
      "operation": "AVG",
      "since_datetime": "2020-04-04 13:21:15"
    },
    {
      "operation": "SUM",
      "until_datetime": "2020-04-14 13:20:15",
      "bucket": "MIN",
      "since_datetime": "2020-04-14 13:20:15",
      "request_id": "dataPointsForWaterRunningStatus",
      "units": "GALLONS"
    },
    {
      "since_datetime": "2020-04-13 14:00:00",
      "bucket": "MIN",
      "until_datetime": "2020-04-14 13:21:15",
      "units": "GALLONS",
      "group_multiplier": "60",
      "request_id": "dataPointsForRecentUsage"
    }
  ]
}

... I get data points for each "block" of the query (total usage for a specific day, average for the trailing 100 days, etc.). A good amount of research and planning would be required, but I could see the integration either (a) grabbing all (or a subset) of this data automatically via a hardcoded query or (b) allowing the user to "build" queries via Home Assistant's configuration.yaml and dynamically creating sensors based on the results. This would probably be best pursued in the longer term.

~

Thanks for letting me post my random thoughts!

[michaeldavie]

I didn't know about the utility meter sensor, but that seems like a really good option. If we were to go that route, maybe it would make sense to have the Flume sensor simply return the meter's all-time consumption, like a traditional meter, rather than the consumption for the last X minutes. Then the user could use the utility meter to build whatever sensors they wanted.

Do you think that would work?

[robertsorensen]

I used Flume for a while and I used the utility_meter integration since Flume became officially supported in HA. I've never managed to to get it to report consumptions that matched the Flume App but I actually think this is because of utility_meter rather than the Flume integration.
Here is what my utility_meter gives me.

Here is a bar chart for the Flume sensor (1 hour; 60 entry pr hour)

The values on this bar chart matches what the Flume app reports (though every entry is 1 minutes off compared to the App). It's easy to see that adding the values gives an hourly value higher than ~5.35 galloons as reported by utility_meter. (Sorry about the units but I haven't bothered fixing that yet)

[michaeldavie]

In order to get this library to work with the utility meter sensor, I think we would need to add a Flume sensor that functions as an accumulator, i.e. it only goes up and never resets. The current sensor seems to function as a sliding window with a width of the scan interval.

If that makes sense I could take a crack at it.

[michaeldavie]

It turns out that an "all-time" query that will only go up isn't possible due to API limitations, so I reverted to my original plan of supporting a wider set of queries. On the Home Assistant side, these can be turned into separate sensors.

[robertsorensen]

Hi Michael

I can't say that I'm an expert in either Flume, Home Assistant or Utility Meter.
From what I've understood on the Flume integration it returns the water usages since "last time" meaning since the last API call. I think that should be straightforward and when fed into Utility Meter should give exactly what everyone wants - that is accumulated water usage either on a hourly, daily weekly etc basis.
I can't yet pin-point where the Flume Integration reports the wrong value - to me and with the data I've seen it matches the Flume App and this leads me to believe that the issue could be the Utility Meter. Now - this integration has existed for a long time and I use it with an Eagle-200 energy meter. From what I can tell - here it reports the right values. Point here is - someone should properly compare the Flume API with what the Flume App reports and compare and then look into what the Utility meter calculates.
All this said - from my understanding and reading of the API I think it should be "simple" to add API calls that report accumulated hourly, daily, weekly usage simply by replacing the "since" value in the API call with either a time starting "at the top of the hour, at midnight today, Sunday at midnight etc.

[michaeldavie]

From what I've seen, I think what you're describing would require at least three sensors chained together:

• The Flume sensor's current sliding window
• A statistics sensor to accumulate a running total (provided the windows do not overlap)
• A utility meter sensor to calculate each desired period

This seems like a lot to rely on a user to configure for fairly basic functionality. Take a look at my pull request linked above and see what you think of the queries I've added, which can be created directly as sensors.

[ChrisMandich]

Module updates completed and update in PyPi.

• Add Notifications
• Supported Device registry
• Add Device Budget
• Add Crazy Queries

Can someone peer-review when we get the chance?
Once confirmed I'll start working on getting these three integrated with HASS for now over the next week. Thanks @michaeldavie for the help!

[skynet01]

Any updates on this? It looked pretty close to being finished in May :)

[ChrisMandich]

@skynet01. A pull request has been open with HA. Here's a screenshot of the updated sensor.