Failed data gets - error detection required

[PeterH24x7]

Very rarely, there is spurious data recorded in the HA Energy Manager. It shows up as an extreme 1-hour spike (e.g. 50 to 100kWh which is just not possible) in the usage and generation data and may be a result of a failed get, or errored data from Solar Analytics. Some error detection (and correction) is required to fix these data records for the sake or impacting long-term data collection.

Unfortunately, when checking the history, the source data has been overwritten within 5 minutes. Some clever investigation and debugging is required.

[PeterH24x7]

Here are some screen images of the errored results in the Energy Manager and the details shown in the logbook. Unfortunately the "unavailable" results of sa_data_by_5min are lost when the next successful get occurs. The sa_live_site_data minutely get still seems to be working okay.

[PeterH24x7]

(UPDATED)
Here's some beta solution code that I'm still testing - it should deal with any "unavailable" rest platform get sensor by simply preserving the previous state and attributes of the template sensor data.

#
# >>> THIS IS BETA CODE SUBJECT TO FURTHER TESTING - USE AT OWN RISK <<<
#
# -----------------------------------------------------------------------------
#
# SOLAR ANALYTICS integration for Home Assistant Energy Monitoring
# 
# Developed by Peter Hormann and with thanks to Glen S.
#
# Created: 7-Jan-2022
#
# Updates: 
#     16-Jan-2022 - added power reporting (last 5 minute average).
#     16-Jul-2022 - enabled more accurate and more frequent 1 minute power reporting
#                 - added 5 minute energy updates for all channels
#                 - moved sa_site_id to secrets.yaml.
#     11-Aug-2022 - added some error checking to deal with daily energy total errors when the data get is 'unavailable' or 'unknown' - CURRENTLY IN TEST
#                 - removed "force_update: true" - where the data get may be 'unavailable'
#                 - removed "state_class: measurement" and "device_class: energy" for rest platform sensors (not applicable)
#
# This code is provided as-is and without any warrantee or guarantees. 
# Further public contributions and enhancements are welcomed.
# 
# Installation: 
#   1. Copy this yaml file to the HA config directory.
#   2. Include this file name as a package in the HA configuration.yaml. 
#      For example -
#        packages:
#          pack_1: !include solar_analytics.yaml
#   3. Define sa_username, sa_password and sa_site_id in the secrets.yaml file.
#   4. After first run, review the attributes sa_data_by_5min to identify the 
#      device specific channel names and update the sa_todays_... xyz sensor to suit.
#   6. Add the new energy sensors from 4. above to the HA Energy Manager as "Monitor 
#      Individual Devices". Go to HA settings and then search for “Energy Configuration” 
#      to make the changes.
#   7. Also in HA Energy Configuration, enter the details for
#      > Grid consumption - as sa_todays_energy_imported
#      > Return to grid - as sa_todays_energy_exported
#      > Solar production - as sa_todays_energy_generated_total
#   8. If all working is correctly, the Energy dashboard should start populating data within 
#      a couple of hours.
#
# Reference for more information and updates:
#   https://github.com/PeterH24x7/-Solar-Analytics-integration-for-Home-Assistant-Energy-monitoring
#
# -----------------------------------------------------------------------------


#
# Define the Solar Analytics site_id value is required in URLs used in gets below - e.g. 36304. 
#   See the Solar Analytics user portal to find your site id or see the collected sa_site_list attributes as defined below.
#
input_text:
    sa_site_id:
      initial: !secret sa_site_id


sensor:

#  
# Solar Analytics - get the site details associated with the given user account login.
# Updated every 24 hours.
#
  - platform: rest
    name: sa_site_list
    resource: https://portal.solaranalytics.com.au/api/v3/site_list?hardware=true&capacity=true&subscription=true
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: "{{ now() }}"
# note: assumes only a single device per account, there could be a second device (e.g. replaced device due to fault/upgrade) at $.data.[1]
    json_attributes_path: "$.data.[0]"
    json_attributes:
      - "e_status"
      - "fault_class"
      - "fault_id*"
      - "has_pv"
      - "mer_status"
      - "overall_status"
      - "retailer_user"
      - "s_cli_site_name"
      - "site_id"
      - "site_inactive"
      - "capacity"
      - "devices"
      - "sub_type"
    scan_interval: 62400

#
# Solar Analytics - get the site status for the specified site_id.
# Updated every hour.
#
  - platform: rest
    name: sa_status
    resource_template: https://portal.solaranalytics.com.au/api/v3/site_status/{{ states('input_text.sa_site_id') }}
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: "{{ value_json['data']['mer_status'] }}"
    json_attributes_path: "$.data"
    json_attributes:
      - "dashboard_status"
      - "event_id"
      - "event_list"
      - "fault_status"
      - "mer_percentage"
      - "mer_status"
      - "mer_text"
    scan_interval: 3600
#    force_update: true

  - platform: template
    sensors:
      sa_dashboard_status:
        friendly_name: "Dashboard Status"
        value_template: "{{ state_attr('sensor.sa_status', 'dashboard_status') }}"
      sa_mer_status:
        friendly_name: "System Status"
        value_template: "{{ state_attr('sensor.sa_status', 'mer_status') }}"
      sa_mer_percentage: 
        friendly_name: "PV Performance"
        value_template: "{{ state_attr('sensor.sa_status', 'mer_percentage') }}"
        unit_of_measurement: "%"

#
# Solar Analytics - get 5 minute energy data.
# Used to calculate today's cumulative total energy for consumed, generated, imported and exported energy as used by the HA Energy module.
# Updated every 5 minutes.
# Note: the previous version calculated sa_consumption_power, sa_generation_power and sa_import_export_power (as an average over 
#   5 minutes) are now directly sourced using get sa_live_site_data sensor further down below.
# 15-Jul-22 - added trunc=false to include decimal components of 5 minute date (default is true). 
#    Also added air conditioner, EV, hot water and stove-oven energy loads as shown in the attributes of sa_data_by_5min. 
#    These channels should be added/edited/deleted to match the specific device set-up.
#
  - platform: rest
    name: sa_data_by_5min
    resource_template: https://portal.solaranalytics.com.au/api/v2/site_data/{{ states('input_text.sa_site_id') }}?all=true&gran=minute&trunc=false&power=false&tstart={{ now().strftime("%Y%m%d") }}&tend={{ now().strftime("%Y%m%d") }}   
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: >-
      {% set most_recent_sensor_data = value_json['data'] | rejectattr('energy_consumed', 'equalto', None) | list | last %}
        {{ most_recent_sensor_data.t_stamp }}    
    json_attributes:
      - "data"
#    state_class: measurement
#    device_class: energy
    scan_interval: 300
# Force update disabled for case when sensor becomes 'unavailable'.
#    force_update: true

  - platform: template
    sensors:
      sa_todays_energy_consumed_total:
        friendly_name: Total Energy Consumed
        unit_of_measurement: "Wh"
#        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'energy_consumed', 'equalto', None ) | sum( attribute='energy_consumed' ) | int }}"
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'energy_consumed', 'equalto', None ) | sum( attribute='energy_consumed' ) | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_energy_consumed_total' ) }}
          {% endif %}
        icon_template: mdi:home-lightning-bolt-outline
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_energy_consumed_total.time_stamp" ) }}
            {% endif %}

      sa_todays_energy_generated_total:
        friendly_name: Total Energy Generated
        unit_of_measurement: "Wh"
#        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'energy_generated', 'equalto', None ) | sum( attribute='energy_generated' ) | int }}"
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'energy_generated', 'equalto', None ) | sum( attribute='energy_generated' ) | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_energy_generated_total' ) }}
          {% endif %}
        icon_template: mdi:solar-power
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_energy_generated_total.time_stamp" ) }}
            {% endif %}

      sa_todays_air_conditioner_total:
        friendly_name: Heating Cooling Energy
        unit_of_measurement: "Wh"
#        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_air_conditioner', 'equalto', None ) | sum( attribute='load_air_conditioner' ) | int }}"
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_air_conditioner', 'equalto', None ) | sum( attribute='load_air_conditioner' ) | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_air_conditioner_total' ) }}
          {% endif %}
        icon_template: mdi:hvac
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_air_conditioner_total.time_stamp" ) }}
            {% endif %}

      sa_todays_electric_vehicle_total:
        friendly_name: Electric Vehicle Energy
        unit_of_measurement: "Wh"
#        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_ev_charger', 'equalto', None ) | sum( attribute='load_ev_charger' ) | int }}"
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_ev_charger', 'equalto', None ) | sum( attribute='load_ev_charger' ) | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_electric_vehicle_total' ) }}
          {% endif %}
        icon_template: mdi:car-electric
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_electric_vehicle_total.time_stamp" ) }}
            {% endif %}

      sa_todays_hot_water_total:
        friendly_name: Hot Water Energy
        unit_of_measurement: "Wh"
#        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_hot_water', 'equalto', None ) | sum( attribute='load_hot_water' ) | int }}"
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_hot_water', 'equalto', None ) | sum( attribute='load_hot_water' ) | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_hot_water_total' ) }}
          {% endif %}
        icon_template: mdi:shower-head
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_hot_water_total.time_stamp" ) }}
            {% endif %}

      sa_todays_stove_oven_total:
        friendly_name: Stove Oven Energy
        unit_of_measurement: "Wh"
#        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_stove', 'equalto', None ) | sum( attribute='load_stove' ) | int }}"
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {{ states.sensor.sa_data_by_5min.attributes.data | rejectattr( 'load_stove', 'equalto', None ) | sum( attribute='load_stove' ) | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_stove_oven_total' ) }}
          {% endif %}
        icon_template: mdi:stove
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_stove_oven_total.time_stamp" ) }}
            {% endif %}

      sa_todays_energy_imported:
        friendly_name: Total Energy Imported
        unit_of_measurement: "Wh"
#        value_template: >
#          {% set energy = namespace(imported=0) %}
#          {% for sensor_data in states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_consumed', 'equalto', None) if sensor_data.energy_consumed > sensor_data.energy_generated %}
#            {% set energy.imported = energy.imported + sensor_data.energy_consumed - sensor_data.energy_generated %}
#          {% endfor %}
#          {{ energy.imported | int }}
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {% set energy = namespace( imported = 0 ) %}
            {% for sensor_data in states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_consumed', 'equalto', None) if sensor_data.energy_consumed > sensor_data.energy_generated %}
              {% set energy.imported = energy.imported + sensor_data.energy_consumed - sensor_data.energy_generated %}
            {% endfor %}
            {{ energy.imported | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_energy_imported' ) }}
          {% endif %}
        icon_template: mdi:transmission-tower-export
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_energy_imported.time_stamp" ) }}
            {% endif %}

      sa_todays_energy_exported:
        friendly_name: Total Energy Exported
        unit_of_measurement: "Wh"
#        value_template: >
#          {% set energy = namespace(exported=0) %}
#          {% for sensor_data in states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_generated', 'equalto', None) if sensor_data.energy_generated > sensor_data.energy_consumed %}
#            {% set energy.exported = energy.exported + sensor_data.energy_generated - sensor_data.energy_consumed %}
#          {% endfor %}
#          {{ energy.exported | int }}
        value_template: >
          {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
            {% set energy = namespace( exported = 0 ) %}
            {% for sensor_data in states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_generated', 'equalto', None) if sensor_data.energy_generated > sensor_data.energy_consumed %}
              {% set energy.exported = energy.exported + sensor_data.energy_generated - sensor_data.energy_consumed %}
            {% endfor %}
            {{ energy.exported | int }}
          {% else %}
            {{ states( 'sensor.sa_todays_energy_exported' ) }}
          {% endif %}
        icon_template: mdi:transmission-tower-import
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
#          time_stamp: "{{ states.sensor.sa_data_by_5min.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_data_by_5min' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_data_by_5min.state }}
            {% else %}
              {{ state_attr( "sensor.sa_todays_energy_exported.time_stamp" ) }}
            {% endif %}

#
# Solar Analytics - get live per minute power data - added 9-Jul-22.
# Used to generate most recent power information.
# Updated every 60 seconds.
# Note: this is not a published API 'get', but is used by the SA portal to display minute to minute power for a site.
#
  - platform: rest
    name: sa_live_site_data
    resource_template: https://portal.solaranalytics.com.au/api/v3/live_site_data?site_id={{ states('input_text.sa_site_id') }}&last_six=true
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: >-
      {% set most_recent_sensor_data = value_json['data'] | list | last %}
        {{ as_timestamp(most_recent_sensor_data.t_stamp) | timestamp_custom ('%Y-%m-%d %H:%M:%S')}}    
    json_attributes:
      - "data"
#    state_class: measurement
#    device_class: power
    scan_interval: 60
#    force_update: true

  - platform: template
    sensors:          
      sa_consumption_power:
        friendly_name: Power Consumption
        unit_of_measurement: "W"
#        value_template: >
#          {% set most_recent_sensor_data = states.sensor.sa_live_site_data.attributes.data | list | last %}
#            {{ most_recent_sensor_data.consumed | int }}
        value_template: >
          {% if states( 'sensor.sa_live_site_data' ) not in ("unavailable", "unknown") %}
            {% set most_recent_sensor_data = states.sensor.sa_live_site_data.attributes.data | list | last %}
              {{ most_recent_sensor_data.consumed | int }}
          {% else %}
            {{ states( 'sensor.sa_consumption_power' ) }}
          {% endif %}
        icon_template: mdi:home-lightning-bolt-outline
        device_class: "power"
        attribute_templates:
          state_class: measurement
#          time_stamp: "{{ states.sensor.sa_live_site_data.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_live_site_data' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_live_site_data.state }}
            {% else %}
              {{ state_attr( "sensor.sa_consumption_power.time_stamp" ) }}
            {% endif %}

      sa_generation_power:
        friendly_name: Power Generation
        unit_of_measurement: "W"
#        value_template: >      
#          {% set most_recent_sensor_data = states.sensor.sa_live_site_data.attributes.data | list | last %}
#            {{ most_recent_sensor_data.generated | int }}
        value_template: >
          {% if states( 'sensor.sa_live_site_data' ) not in ("unavailable", "unknown") %}
            {% set most_recent_sensor_data = states.sensor.sa_live_site_data.attributes.data | list | last %}
              {{ most_recent_sensor_data.generated | int }}
          {% else %}
            {{ states( 'sensor.sa_generation_power' ) }}
          {% endif %}
        icon_template: mdi:transmission-tower-import
        device_class: "power"
        attribute_templates:
          state_class: measurement
#          time_stamp: "{{ states.sensor.sa_live_site_data.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_live_site_data' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_live_site_data.state }}
            {% else %}
              {{ state_attr( "sensor.sa_generation_power.time_stamp" ) }}
            {% endif %}

      sa_import_export_power:
        friendly_name: Power Import Export
        unit_of_measurement: "W"
#        value_template: >      
#          {{ (states.sensor.sa_consumption_power.state | int) - (states.sensor.sa_generation_power.state | int) }}
        value_template: >
          {% if states( 'sensor.sa_live_site_data' ) not in ("unavailable", "unknown") %}
            {{ (states.sensor.sa_consumption_power.state | int) - (states.sensor.sa_generation_power.state | int) }}
          {% else %}
            {{ states( 'sensor.sa_import_export_power' ) }}
          {% endif %}
        icon_template: mdi:transmission-tower
        device_class: "power"
        attribute_templates:
          state_class: measurement
#          time_stamp: "{{ states.sensor.sa_live_site_data.state }}"
          time_stamp: >
            {% if states( 'sensor.sa_live_site_data' ) not in ("unavailable", "unknown") %}
              {{ states.sensor.sa_live_site_data.state }}
            {% else %}
              {{ state_attr( "sensor.sa_import_export_power.time_stamp" ) }}
            {% endif %}

# -----------------------------------------------------------------------------

[PeterH24x7]

After ~5 days of operation the updated code appears to be working as expected. There has been no spurious data gets and the intraday energy total and channel energy continue to match those provided by the Solar Analytics user portal. Will keep testing for another 5 days before declaring the issue fixed.

[PeterH24x7]

Our HFC based home internet service was offline for 32 hours Wed/Thurs until yesterday evening. While my GW-router wireless back-up did operate the service was very poor (low bandwidth, packet-loss) due to so many others similarly being impacted. I had various cloud based sensors "unavailable" at times and remote access via SSL was next to impossible. Nonetheless the SA code above worked as expected and there were no spurious hourly reports and the end-of-day totals and channel data matched that reported by the Solar Analytics portal. I think that I'm ready to declare a test success and publish the updated code.

[PeterH24x7]

Appears to be all fixed. Cleaned-up and tested code now published as Release 3.