HA statistics - share ideas

[S-Przybylski]

Dear all
i am interested in statistics about my heating. I just started to learn a little bit about the historical statisics in HA (and yes i am aware that functions like max, min, mean, integrate and differentiate could easily be done with Grafana - i also use it). But i am also interested in simple functions within HA.

So my first question was, how long does the heating heats the warm water and heating circuit itself?
Please note: The yesterday value will be calculated

sensors.yaml:
##########################
#Heating Test historical statistics
##########################
- platform: history_stats
  name: heating_warmwater_hst_runtime_per_day
  entity_id: binary_sensor.heating_warmwater
  state: 'on'
  type: time
  end: '{{ now().replace(hour=0).replace(minute=0).replace(second=0) }}'
  duration:
    hours: 24

- platform: history_stats
  name: heating_warmwater_hst_ratio_per_day
  entity_id: binary_sensor.heating_warmwater
  state: 'on'
  type: ratio
  end: '{{ now().replace(hour=0).replace(minute=0).replace(second=0) }}'
  duration:
    hours: 24

- platform: history_stats
  name: heating_warmwater_hst_ratio_between_5_and_23
  entity_id: binary_sensor.heating_warmwater
  state: 'on'
  type: ratio
# between 5 -23h
  end: '{{ ((as_timestamp(now().replace(hour=0).replace(minute=0).replace(second=0)) | int) - 3600)  }}'
  duration:
    hours: 18

- platform: history_stats
  name: heating_heating_hst_runtime_per_day
  entity_id: binary_sensor.heating_heating
  state: 'on'
  type: time
  end: '{{ now().replace(hour=0).replace(minute=0).replace(second=0) }}'
  duration:
    hours: 24

- platform: history_stats
  name: heating_heating_hst_ratio_per_day
  entity_id: binary_sensor.heating_heating
  state: 'on'
  type: ratio
  end: '{{ now().replace(hour=0).replace(minute=0).replace(second=0) }}'
  duration:
    hours: 24

and the template sensors to identify if the heating works in heating mode or warm water mode:

binary_sensors.yaml:
##########################
# for Heating Test historical statistics
##########################
- platform: template
  sensors:
    heating_warmwater:
      friendly_name: "Heating Warmwater"
      value_template: >-
       {% if is_state("binary_sensor.heating", "on") and is_state("sensor.Warm_Water_3_way_valve", "on") and is_state("sensor.boiler_pump", "on") %} {{ true }}
       {% else %} {{ false }}
       {% endif %}
# used to get frequent updates!!!
      entity_id:
        - binary_sensor.heating
        - sensor.Warm_Water_3_way_valve
        - sensor.current_flow_temperature

- platform: template
  sensors:
    heating_heating:
      friendly_name: "Heating circuit"
      value_template: >-
       {% if is_state("binary_sensor.heating", "on") and is_state("sensor.Warm_Water_3_way_valve", "off") and is_state("sensor.boiler_pump", "on") %} {{ true }}
       {% else %} {{ false }}
       {% endif %}
# used to get frequent updates!!!
      entity_id:
        - binary_sensor.heating
        - sensor.Warm_Water_3_way_valve
        - sensor.current_flow_temperature

What do you think about that?

Do you have other interessing ideas?

[proddy]

I've added this to the wiki

[majdzik84]

Activating one-time hot water charging
DHW once (OneTimeWater)

switch.yaml

      one_time_water:
        friendly_name: OneTimeWater
        value_template: "{{ is_state('sensor.one_time_water', 'on') }}"
        turn_on:
          service: script.turn_on
          entity_id: script.one_time_water_on
        turn_off:
          service: script.turn_on
          entity_id: script.one_time_water_off

scripts.yaml

  one_time_water_on:
    sequence:
      - service: mqtt.publish
        data_template:
          topic: 'home/ems-esp/boiler_cmd_wwonetime'
          payload: '1'
          
  one_time_water_off:
    sequence:
      - service: mqtt.publish
        data_template:
          topic: 'home/ems-esp/boiler_cmd_wwonetime'
          payload: '0'

sensors.yaml

  - platform: mqtt
    state_topic: 'home/ems-esp/boiler_data'
    name: 'one_time_water'
    value_template: '{{ value_json.wWOnetime }}'

[proddy]

thanks @majdzik84 . added to https://github.com/proddy/EMS-ESP/wiki/Home-Assistant

[S-Przybylski]

I tried to calculate the energy consumption of the heating as a first try.
(Updated: Translation error of sensor and message adaption)

Any suggestions ?

To use the reset function, please activate python scripting and create a file located at ./python_scripts/set_state.py (content see below)

sensors.yaml:

#####################################################################################
# Version 0.1
# GB172-20 (2014)
# Calculate current boiler power and try to estimate gas consumption
# GB172-20 Vendor says: 25% = 5,2kW, 100% = 20,6kW and 123% 23,8kW
# Factor to adapt values: my boiler seems to have 1.635% more power then calculated
#####################################################################################
- platform: template
  sensors:
    heating_power_boiler:
      friendly_name: "Boiler current heating power"
      unit_of_measurement: 'W'
      value_template: >-
        {%- set current_power = states('sensor.burner_current_power')| int -%}
        {%- if is_state("binary_sensor.heating", "on") -%}
           {%- if current_power == 0 -%}
             0
           {%- elif current_power <= 25 -%}
             {{ ((((current_power * 208) | float) * 1.01635) | round(0)) | int }}
           {%- elif current_power <= 100 -%}
             {{ ((((((current_power - 25) | float) * 205) + 5200) * 1.01635) | round(0)) | int }}
           {%- elif current_power > 100 -%}
             {{ ((((((current_power - 100)| float) * 139.13) + 20600) * 1.01635) | round(0)) | int }}
           {%- endif -%}
        {%- else -%}
          0
        {%- endif -%}
      entity_id:
        - binary_sensor.heating
        - sensor.burner_current_power
        - sensor.boiler_pump
        - sensor.Warm_Water_3_way_valve
        - sensor.current_flow_temperature

#####################################################################################
# optional:

# Heating power warmwater
- platform: template
  sensors:
    heating_power_ww:
      friendly_name: "current heating power warmwater"
      unit_of_measurement: 'W'
      value_template: >-
        {% if is_state("sensor.Warm_Water_3_way_valve", "on") and is_state("binary_sensor.heating", "on") %} {{ states.sensor.heating_power_boiler.state }}
        {%- else -%}
        0
        {%- endif -%}
      entity_id:
        - sensor.heating_power_boiler
        - binary_sensor.heating
        - sensor.burner_current_power
        - sensor.Warm_Water_3_way_valve
        - sensor.current_flow_temperature

- platform: template
  sensors:
    heating_power_hc:
      friendly_name: "current heating power heat circuit"
      unit_of_measurement: 'W'
      value_template: >-
        {% if is_state("sensor.Warm_Water_3_way_valve", "off") and is_state("binary_sensor.heating", "on") %} {{ states.sensor.heating_power_boiler.state }}
        {%- else -%}
        0
        {%- endif -%}
      entity_id:
        - sensor.heating_power_boiler
        - binary_sensor.heating
        - sensor.burner_current_power
        - sensor.Warm_Water_3_way_valve
        - sensor.current_flow_temperature

#####################################################################################
# integrators to calculate engery consumption
# note: these integrators could be reset by using an external python script

- platform: integration
  source: sensor.heating_power_boiler
  name: energy_consumption_boiler
  unit_time: h
  unit_prefix: k
  method: left
  round: 3

- platform: integration
  source: sensor.heating_power_ww
  name: energy_consumption_ww
  unit_time: h
  unit_prefix: k
  method: left
  round: 3

- platform: integration
  source: sensor.heating_power_hc
  name: energy_consumption_hc
  unit_time: h
  unit_prefix: k
  method: left
  round: 3
#####################################################################################
# gas meter calculation
# mainly used to check that the calculated power consumption is valid!
# calculation:
# my last bill shows at least two factors to calculate the kWh from m³
#    key figure:        9.8719
#    location_update:   0.9617
#    resulting factor:  9.49380 kWh/m³ (key_figure * location_update)
# factor can be updated using the field input_number.gas_meter_factor

- platform: template
  sensors:
    gas_meter_total:
      friendly_name: "GAS meter (calculated)"
      unit_of_measurement: 'm³'
      value_template: >-
         {{ ((((states('sensor.energy_consumption_boiler') | float) * 1000) / (states('input_number.gas_meter_factor') | float )) + (states('input_number.gas_meter_bias') | float)) | round(3) }}
      entity_id:
        - sensor.energy_consumption_boiler
        - input_number.gas_meter_bias
        - input_number.gas_meter_factor
############## sensors.yaml end ######################################################

scripts.yaml:

#####################################################################################
# Reset GAS meter
  reset_gas_meter:
    alias: Reset GAS meter
    sequence:
# idea: copy the current gas meter value into the bias and reset the integrators
# used a external python script - see: https://community.home-assistant.io/t/how-to-manually-set-state-value-of-sensor/43975/3
# see parts of https://github.com/gb53smith/Hass
# note: utility_meter could not be reset by this script
      - service: input_number.set_value
        data_template:
          entity_id: input_number.gas_meter_bias
          value: "{{ states('sensor.gas_meter_total') }}"
      - service: python_script.set_state
        data_template:
          entity_id: sensor.energy_consumption_boiler
          state: 0
      - service: python_script.set_state
        data_template:
          entity_id: sensor.energy_consumption_ww
          state: 0
      - service: python_script.set_state
        data_template:
          entity_id: sensor.energy_consumption_hc
          state: 0
############## scripts.yaml end ######################################################

groups.yaml:

######################################################################################
boiler_energy_stats:
  name: 'Energy statistics'
  view: no
  control: hidden
  entities:
    - sensor.heating_boiler
    - sensor.heating_ww
    - sensor.heating_hc
    - sensor.energy_consumption_boiler
    - sensor.energy_consumption_ww
    - sensor.energy_consumption_hc
    - sensor.gastherme_energy_consumption_boiler_daily
    - sensor.gastherme_energy_consumption_ww_daily
    - sensor.gastherme_energy_consumption_boiler_monthly
    - sensor.gastherme_energy_consumption_hc_daily
    - sensor.gastherme_energy_consumption_ww_monthly
    - sensor.gastherme_energy_consumption_hc_monthly
    - sensor.gas_meter_total
    - script.reset_gas_meter
    - input_number.gas_meter_bias
    - input_number.gas_meter_factor
############## groups.yaml end ######################################################

configuration.yaml:

######################################################################################
# Enable CUSTOMIZED PYTHON SCRIPTS
python_script:

######################################################################################
# input_number
# to enter the m³ to kWh factor see sensor calculation

input_number:
  gas_meter_bias:
    name: "Gas meter bias"
    min: 00001
    max: 90000
    step: 0.001
    unit_of_measurement: "m³"
    icon: mdi:fire
    mode: box
  gas_meter_factor:
########### update: corrected message 100 instead of 1000 - my factor is 9494
    name: "Factor m³ to kWh x 100"
    min: 8500
    max: 20000
    step: 1
    unit_of_measurement: ""
    icon: mdi:fire
    mode: box
#
######################################################################################
#
# optional
# try to calculate daily and monthly values within HA
# note: 
#      reset could only be performed when adding taiffs or by stopping HA and manually update the .storage/core.restore_state
#      historian of HA resricts long term analytics!!!
# could be better if calculated by grafana
# or add additional automation / sensor to gather each day the value of the last day attribute

utility_meter:
  gastherme_energy_consumption_boiler_daily:
    source: sensor.energy_consumption_boiler
    cycle: daily
  gastherme_energy_consumption_boiler_monthly:
    source: sensor.energy_consumption_boiler
    cycle: monthly

  gastherme_energy_consumption_ww_daily:
    source: sensor.energy_consumption_ww
    cycle: daily
  gastherme_energy_consumption_ww_monthly:
    source: sensor.energy_consumption_ww
    cycle: monthly

  gastherme_energy_consumption_hc_daily:
    source: sensor.energy_consumption_hc
    cycle: daily
  gastherme_energy_consumption_hc_monthly:
    source: sensor.energy_consumption_hc
    cycle: monthly
######################################################################################

set_state.py

#==================================================================================================
#  python_scripts/set_state.py 
# thanks rodpayne - see https://community.home-assistant.io/t/how-to-manually-set-state-value-of-sensor/43975/3
#==================================================================================================
#--------------------------------------------------------------------------------------------------
# Set the state or other attributes for the entity specified in the Automation Action
#--------------------------------------------------------------------------------------------------
inputEntity = data.get('entity_id')
if inputEntity is None:
    logger.warning("===== entity_id is required if you want to set something.")
elif hass.states.get(inputEntity) is None:
    logger.warning("===== unknown entity_id: %s", inputEntity)
else:
    inputStateObject = hass.states.get(inputEntity)
    inputState = inputStateObject.state
    inputAttributesObject = inputStateObject.attributes.copy()

    for item in data:
        newAttribute = data.get(item)
        logger.debug("===== item = {0}; value = {1}".format(item,newAttribute))
        if item == 'entity_id':
            continue            # already handled
        elif item == 'state':
            inputState = newAttribute
        else:
            inputAttributesObject[item] = newAttribute
    hass.states.set(inputEntity, inputState, inputAttributesObject)

[S-Przybylski]

I've added this to the wiki

Dear @proddy
i do not find my first enhancement in the wiki ... Is it lost somewhere?

[proddy]

Not here? https://github.com/proddy/EMS-ESP/wiki/home-assistant

[S-Przybylski]

Dear @proddy
i think we talk about different things...
I mean the initial historical statistics configuration listed in this issue at first.
I think your answer belongs to the heartbeat configuration.
Or am i wrong?

[S-Przybylski]

Is it worth to create a separate wiki page for advanced HA stuff like statistics?

[proddy]

I think I forgot to add that. Would like to add to the wiki yourself? I’ll grant you contributer privileges

[proddy]

also these could also be specific directories in the github as the files are quite large for a wiki. Just do a PR

[S-Przybylski]

ok i can provide that later on.

[majdzik84]

Helo @S-Przybylski !
Thanks for your work. May I ask what these values ​​are? I'm trying to adapt your calculations to GB162-15

[S-Przybylski]

Dear @majdzik84
thanks for your kindness,
I tried to estimate the gas consumption resulting on the values i found. In general the values are the gradient of the different segments of the heating curve (hopefully that they are valid).
208: 5200/25 = 208 per power_%
205: (20600-5200)/(100-25)=205,33 per power_%
139,13: (23800-20600)/(123-100)=123,13 per power_%
My personal correction factor (1,01635) results of a anaylsis of only one day as a try. I definitly see that the compensation needs to be optimised.... Perhaps we can optimise that together. The reset functionallity doesn't work as expected. Inside the integrator there are hidden values, that prevent the reset.... Needs to be optimised too.

[majdzik84]

Thank you for the explanation.
I was reminded that once I read a topic about the introduction of m3 and gas cost calculation in EMS.

https://github.com/proddy/EMS-ESP/issues/91

[proddy]

Thank you for the explanation.
I was reminded that once I read a topic about the introduction of m3 and gas cost calculation in EMS.

emsesp/EMS-ESP#91

Indeed, the idea was to let EMS-ESP keep a running count of the data (persisted in non-volatile RAM) and have the home controllers (e.g. HA) do the number crunching and calculations using the formulas.

[S-Przybylski]

Dear @proddy
dear @majdzik84
To implement the main calculation inside the esp, we need from my point of view:

1. heating curves of the different heatings (which could be stored in the esp firmware/...)
   for gb172-20 i made a simple suggestion with three segments (perhaps with the capability of an correction factor)
2. integrator functionallity for power consumption (kwh) - type: left riemann estimation per hour (different values for warmwater and heating circuit i suggest) https://en.wikipedia.org/wiki/Riemann_sum (implementation could be done similar to ha)
3. a functionallity like utility_meter (ha) to sum the hourly, daily, monthly values to be displayed by grafana/ha ...

To calculate the final m³ consumption:
4. the local calculation factor of the gas energy value (different gas types) - this value changes in my area from month to month a little bit (see bill) - this makes the estimation a little bit difficult --> an input method within esp to enter this value - required
5. the fixed local correction factor (e.g. altitude dependent - see bill) --> an input method within esp to enter this value - if none (=1)
6. gas meter bias (last offical meter value as a reference for the next bill) best with the date as an input method (Esp)
7. The final m³ calculation (pretty easy) = bias + sum_integrator_khw/final_factor
8. a reset functionality to reset all sums (including the integrators/utility meters/...)

What do you think?

[proddy]

@S-Przybylski sounds ok to me and happy to try it out. As long as the ESP doesn't do any heavy calculations and doesn't write to the SPIFFS/EEPROM too often when its collecting data.

[proddy]

To do: add this to the wiki

[lsw271]

hi @S-Przybylski,
at first great work!
could you tell me how yours calculation working? i think about the time when the boiler power is constant for few minutes. Values dont change. And what about boiler efficiency?
I've got Junkers cerapur boiler 24kW heating (minimal power 29%: 6,96kW) and 30kW warm water heating, but my boiler working with 131% boiler power when warm water is preparing. So probably however its working with 31,44kW. For now, I can't hit the same results (counter = calculations) after giving my parameters in sensor. I will be testing more :)

[S-Przybylski]

Dear @lsw271 ,
you found an additional more or less weak point: The home assistant integrators sum the energy consumption if an update of the power consumtion (change) occurs. If a boiler doesn't change the power consumtion for a longer time (e.g. more than 60 min), the values aren't updated. More worse is that statistics could be affeced by this effect (e.g. next day problematic). I saw in the forum that the was a discussion about this (e.g. https://community.home-assistant.io/t/integration-sensor-component-needs-reset-service/98054/6). There was is also an workaround posted https://github.com/gb53smith/Hass . I haven't checked that up to now.
The heating cure is pretty easy: The calculated value based on your values is always 240. Therefore the heating curve is only a line with the gradient 240! I am not quite sure if this is a correct curve.
Please re-check the values if the are valid. I found mine in one of the descriptions of the boiler. It could be that you have to calulate the values using the efficiency factor of the different operation points, if you don't find the gas side power consumption values. My boiler has an gas side power consumption of 20,6 kW for 20kW (water side) - 100% if believe.
Since November my calculation is about 10cbm wrong. I think for a first try its ok!

Calculation:
29% = 6960 W --> 6960W/29% = 240 per power_% (0%-29%)
100% = 24000 W (assumption) --> (24000-6960) / (100 - 29) = 240 per power_% (29%-100%)
131% = 31440 W -->( 31440 -24000) / (131 - 100) = 240 per power_% (>100%)

[swa72]

I'm interested in approximating my gas and energy consumption with the available data of the EMS-EPS (in my case all data ends up in influxdb/grafana via HA). I've read the above posts but could not figure out how/where sensor.energy_consumption_boiler is calculated. I assume it is fed in by a meter?

Initially, I'm looking for a simple formula which gives me an approximation (not an exact measurement). I would like to compare this with the energy consumption data that my RC300 shows.

Input variables:

• Boiler dimension (in my case a whopping 35kW)
• Boiler Burner max power
• Boiler Burner current power
• Runtime (derived from "Boiler Service code number" states and changes/duration)

Is this doable?

[S-Przybylski]

Hi @swa72
this was an approach to calculate the energy consumption not by using an external meter.
This calculation runs since 2019-08 and has a deviation about 10% for my heating. I haven't spend much time in making it more precise...
Regarding your question: energy_consumption_boiler - is an integration sensor. This calculates the kwh out of the energy used.

• platform: integration
  source: sensor.heating_power_boiler
  name: energy_consumption_boiler
  unit_time: h
  unit_prefix: k
  method: left
  round: 3

Please keep in mind that different boilers have different kind of heating curves. In my case it seems like this:
GB172-20 2014:
25% = 5,2kW
100% = 20,6kW
123% = 23,8kW (used only for Water heating)

[toleabivol]

For GB172-20 now I get a sensor natively with total/dhw/heating energy . This can then be added to the energy dashboard and track costs of gas.

[proddy]

For GB172-20 now I get a sensor natively with total/dhw/heating energy . This can then be added to the energy dashboard and track costs of gas.

if you have time it would be great if you could write this up, with screenshots so we can post on the Wiki for others to learn. 🥇

[toleabivol]

Sure thing.
Since the sensor comes directly from the ems there's not much one must do, just go to the energy dashboard and select it for the gas consumption.

I am now looking into calculating the cost/kW for liquified gas so I can insert it. I have a tank burried in the ground and fill it up from time to time. The cost depends on the market value when I am carrying out the tank fill-up order. The complicated part is that I am charged with L (liters).
If I am right we can calculate how many kW can a Liter of liquified gas produce and thus get a static number for the cost of gas.

[toleabivol]

If one wants to see separate values for water and heating then just use the other 2 sensors:

[toleabivol]

Hmm, not sure why the 2 grapsh don't add up, the 3 sensors look to be adding up in history .
Could be due to the annomaly at ~11:30 when the dhw energy went down (could be due to me restarting ems?).

[proddy]

I think it's better if you open a separate GitHub issue for this so we can track and comment. the values should be stable and linear