Daily bucket calculation is not correct when using a solar radiation sensor #363

jata1 · 2024-05-09T22:12:16Z

What happened?

Everything working fine until early April 2024 - so likely when I updated to HA 2024.4.

Since then buckets are no longer updating so something is not working correctly.

My weather station data is all good and nothing has changed so I am not sure what is happening.

How to reproduce

setup smart irrigation as normal with the default so that buckets are updated daily at 11pm
after 11pm I notice that the bucket doesn't change or if it does it is a tiny amount

Relevant log output

Logger: homeassistant.util.loop
Source: util/loop.py:99
First occurred: 8:07:13 AM (14 occurrences)
Last logged: 8:07:18 AM

Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues

Which version are you running?

2024.5.2

Diagnostics file

I have attached a diagnostics file

Additional information

please find diagnostic file here. Happy to help with any further info and troubleshooting.

config_entry-smart_irrigation-7f7127f4c22a04d1cbd475295c54538c.json

jata1 · 2024-05-09T22:17:32Z

Taking a look at the diagnostic file myself, I see that the temperature sensor in all of my sensor groups does not have a temperature unit in the file - but I can confirm that they are all set to C in the GUI.

jata1 · 2024-05-09T22:58:35Z

and I turned on debug logging for the integration and here are the logs for smart_irrigation. Hope this is helpful.

2024-05-10 08:07:12.795 WARNING (SyncWorker_4) [homeassistant.loader] We found a custom integration smart_irrigation which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-10 08:07:13.486 INFO (MainThread) [homeassistant.bootstrap] Setting up stage 2: {'smart_irrigation', 'config', 'wake_word', 'schedule', 'panel_custom', 'roborock', 'script', 'energy', 'intent', 'file_upload', 'weatherflow', 'climate', 'linktap', 'hacs', 'media_source', 'weatherflow_forecast', 'automation', 'default_config', 'mobile_app', 'hue', 'browser_mod', 'diagnostics', 'homeassistant_alerts', 'sensor', 'input_button', 'thread', 'scheduler', 'broadlink', 'input_boolean', 'input_select', 'lovelace', 'conversation', 'min_max', 'backup', 'logbook', 'system_health', 'time_date', 'my', 'trace', 'application_credentials', 'twinkly', 'input_datetime', 'traccar_server', 'python_script', 'cert_expiry', 'rest_command', 'person', 'device_automation', 'blueprint', 'template', 'map', 'assist_pipeline', 'moon', 'onboarding', 'image_upload', 'bureau_of_meteorology', 'history', 'search', 'hardware', 'persistent_notification', 'composite', 'analytics', 'stream', 'stt', 'utility_meter', 'duckdns', 'scene', 'ffmpeg', 'timer', 'tts', 'homekit', 'asusrouter', 'icloud3', 'input_text', 'counter', 'switch', 'tag', 'sun', 'input_number', 'zone'}
2024-05-10 08:07:13.663 INFO (MainThread) [homeassistant.setup] Setting up smart_irrigation
2024-05-10 08:07:13.663 INFO (MainThread) [homeassistant.setup] Setup of domain smart_irrigation took 0.00 seconds
2024-05-10 08:07:13.669 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:07:13.685 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:07:13.688 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:07:13.696 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:07:13.698 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:07:13.699 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:07:13.700 DEBUG (MainThread) [custom_components.smart_irrigation] calling register start event from init
2024-05-10 08:07:13.701 INFO (MainThread) [custom_components.smart_irrigation] Start irrigation event smart_irrigation_start_irrigation_all_zones will fire at 34 seconds before sunrise
2024-05-10 08:07:13.711 INFO (MainThread) [custom_components.smart_irrigation] Scheduled auto calculate for 23:00
2024-05-10 08:07:13.711 INFO (MainThread) [custom_components.smart_irrigation] Scheduled auto clear of weatherdata for 23:59
2024-05-10 08:07:13.713 INFO (MainThread) [custom_components.smart_irrigation] Updating weather data for all automatic zones
2024-05-10 08:07:13.713 DEBUG (MainThread) [custom_components.smart_irrigation] check_mapping_sources for mapping_id 0 returns OWM: False, sensor: True, static: False
2024-05-10 08:07:13.714 DEBUG (MainThread) [custom_components.smart_irrigation] async_update_all for mapping 0 new mapping_data: {'retrieved': datetime.datetime(2024, 5, 10, 8, 7, 13, 714033)}
2024-05-10 08:07:13.714 DEBUG (MainThread) [custom_components.smart_irrigation] check_mapping_sources for mapping_id 1 returns OWM: False, sensor: True, static: True
2024-05-10 08:07:13.714 DEBUG (MainThread) [custom_components.smart_irrigation] merge_weatherdata_and_sensor_values, adding Precipitation value 0.0 from sensors
2024-05-10 08:07:13.714 DEBUG (MainThread) [custom_components.smart_irrigation] async_update_all for mapping 1 new mapping_data: {'Precipitation': 0.0, 'retrieved': datetime.datetime(2024, 5, 10, 8, 7, 13, 714625)}
2024-05-10 08:07:13.714 INFO (MainThread) [custom_components.smart_irrigation] Scheduled auto update time interval for each 1:00:00
2024-05-10 08:07:15.977 INFO (MainThread) [homeassistant.components.sensor] Setting up smart_irrigation.sensor
2024-05-10 08:07:15.977 DEBUG (MainThread) [custom_components.smart_irrigation.sensor] sensor.smart_irrigation_front is added to hass
2024-05-10 08:07:15.980 DEBUG (MainThread) [custom_components.smart_irrigation.sensor] sensor.smart_irrigation_back is added to hass
2024-05-10 08:54:46.013 INFO (MainThread) [custom_components.smart_irrigation] Calculating all zones
2024-05-10 08:54:46.014 INFO (MainThread) [custom_components.smart_irrigation] Calculating all automatic zones
2024-05-10 08:54:46.018 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.021 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.023 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.026 DEBUG (MainThread) [custom_components.smart_irrigation] apply_aggregates_to_mapping_data returns {'data_multiplier': 0.04642906837962963, 'Dewpoint': 14.40775323050375, 'Humidity': 93.38499999999999, 'Pressure': 1024.62, 'Solar Radiation': 2.808, 'Maximum Temperature': 15.8, 'Minimum Temperature': 15.14, 'Temperature': 15.47, 'Windspeed': 0.04350000000000003, 'Precipitation': 0.0}
2024-05-10 08:54:46.031 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.033 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.036 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.044 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.047 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.050 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.052 DEBUG (MainThread) [custom_components.smart_irrigation] apply_aggregates_to_mapping_data returns {'data_multiplier': 0.04642909190972222, 'Dewpoint': 14.40775323050375, 'Humidity': 93.38499999999999, 'Precipitation': 0.0, 'Pressure': 1024.62, 'Solar Radiation': 2.808, 'Maximum Temperature': 15.8, 'Minimum Temperature': 15.14, 'Temperature': 15.47, 'Windspeed': 0.04350000000000003}
2024-05-10 08:54:46.056 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.059 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.061 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to import_module inside the event loop by custom integration 'smart_irrigation' at custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module( (offender: /config/custom_components/smart_irrigation/helpers.py, line 436: mod = importlib.import_module(), please create a bug report at https://github.com/jeroenterheerdt/HASmartIrrigation/issues
2024-05-10 08:54:46.067 DEBUG (MainThread) [custom_components.smart_irrigation] calling register start event from async_calculate_all
2024-05-10 08:54:46.067 INFO (MainThread) [custom_components.smart_irrigation] Start irrigation event smart_irrigation_start_irrigation_all_zones will fire at 35 seconds before sunrise
2024-05-10 08:54:46.068 DEBUG (MainThread) [custom_components.smart_irrigation] calling register start event from async_update_zone_config
2024-05-10 08:54:46.068 INFO (MainThread) [custom_components.smart_irrigation] Start irrigation event smart_irrigation_start_irrigation_all_zones will fire at 35 seconds before sunrise

jeroenterheerdt · 2024-05-10T22:00:05Z

the unit is not interesting - it is set automatically to Celcius if not set to anything else. I am not seeing the blocking call issues on my side, but I will try to resolve it.

jeroenterheerdt · 2024-05-10T22:56:30Z

the warning is not stopping the integration from working - I don't see what is not working here. Are you not getting any other errors?

jata1 · 2024-05-10T23:27:10Z

No errors. Nothing to indicate it is not working.

Watering stopped (bucket not updating) in early April. No idea why. Nothing changed at my end.

I might setup a HA test environment and see if I can reproduce.

Anything else I can do to see what is going on with the bucket calculation?

jata1 · 2024-05-11T01:58:38Z

@jeroenterheerdt - I have setup a new test environment (another docker container) so my test and prod are on the exact same baseline HA config.

In my test environment I have added only my weather station and smart irrigation and I have created the setup exactly the same as in prod.

So I will monitor over the next few days to see if the bucket calcs are the same on both - they should be.

FYI - I did notice something a bit strange when setting up new. The default sensor group has config for max and min temp but any new sensor group does not have these. So I think that might be a little bug.

I created new sensor groups and then deleted the default in both prod and test so both setups are identical.

jeroenterheerdt · 2024-05-11T16:32:58Z

there should not be a default sensor group at all..

jata1 · 2024-05-12T02:38:30Z

But there was a default sensor group for me with a clean install of the integration on a clean/new HA setup.

Today I see that my test setup is having the same issue as prod. bucket value of -0.3 from a full day (with precipitation set at 0 so rain is not taken into consideration).

I will keep monitoring and update you.

jeroenterheerdt · 2024-05-12T19:22:58Z

The only thing I can think of is that in early April I fixed a pretty big bug in which the solrad wasn't converted before storing it. That has indeed reduced values tremendously, but is now correct (unless I have inadvertently introduced new bugs). So take a look at the values in the storage file and compare them with what your sensors provide. Do some calculations to check what is in the storage file is correct. I don't see the issue, but I might be missing something. Default sensor group I will look at.

jata1 · 2024-05-13T05:43:48Z

Thanks. Where can I find storage file to look at the data?

My weather station does measure solar radiation so I do have this in my setup. I think solrad might be a clue as my setup stopped working when you made some changes to this.

FYI my solrad is in W/m2 - peaking at around 800-1000

jata1 · 2024-05-13T08:50:35Z

Looking at the data in the diagnostic file, solrad is definitely converted to a much smaller number but I am not sure what conversion is happening.

Can you explain what W/m2 is being converted to for the calculation?

I have attached a new diagnostic file with a fair amount of weather data

config_entry-smart_irrigation-7f7127f4c22a04d1cbd475295c54538c (1).json

jata1 · 2024-05-15T10:24:56Z

So I have been monitoring prod and test for 3-4 days now and I get exactly the same results on both with around -1mm added to the bucket a day. This translates to about 10secs of watering in the front garden and 30secs in the back garden per day.

When this was all working correctly, it was around 3mins watering in the front and 10mins at the back per day.

So I do think something is wrong with how solrad is being handled.

jeroenterheerdt · 2024-05-15T15:39:57Z

thanks for your efforts so far. I have pulled up the code and added a wiki page about the units in the diag / storage file: https://github.com/jeroenterheerdt/HAsmartirrigation/wiki/reading-the-diagnostic-and-storage-file-(v2).

Can I ask you with this in hand to do the comparison to see if you can spot any issues? Is the data indeed converted properly at storage from your input sensors? I don't see issues on my end, but I am also kind of blind to my own code by now.

The solrad value is expected to be way smaller in the storage file than what your input would be in W/m2 as it's stored in MJ/day/m2. 1 watt is equal to 0.0036 megajoules per hour. Since there are 24 hours in a day, you would multiply 0.0036 MJ/h by 24 hours to find the megajoules per day. I think that is what is happening, but I might have made a mistake here.

jata1 · 2024-05-16T07:35:05Z

Sure thing. I’ll take a look at all the info over the next few days and let you know if I find anything.

jata1 · 2024-05-17T01:00:02Z

@jeroenterheerdt - Have done some research and I agree with your maths for the conversion from W/m2 to MJ/day/m2. I see in the integration the conversion factor of 0.0864 (0.0036 x 24) is applied.

I have been looking at my sensor data and I can see that the W/m2 is being converted using the 0.0864 factor into the data file.

So everything looks good at this point. I assume the integration is using the pyETO API - pyeto.fao56_penman_monteith to calculate evapotranspiration for each data point and then adds up these up to calculate the total.

I am not clever enough to understand more than this...

The only idea I have is that I have the coastal boolean set to true as I live right on the coast. However as I am using a sensor to provide solrad directly there is no need to have this flag set as it should not be needed.

I'm thinking maybe that somehow this coastal flag is forcing the integration to calculate solrad rather than use the sensor data.

So I will switch off the coastal boolean and see if that changes anything.

It would be good to see the dataset that is being passed into the API and how these are then added up to provide a total ETo.

jata1 · 2024-05-17T06:56:32Z

The other thought I have is that my watering is way lower than it should be - but it is calculating something. I think it could be out by a factor of around 24. This is making me think that maybe the API needs the solrad for the day or there is something happening where the integration is not adding all the data together,

jeroenterheerdt · 2024-05-17T13:59:21Z

I will add extra debugging statements around the calls to pyeto and the aggregation of data. I will create a beta build so we can hone in on this.

jata1 · 2024-05-17T20:53:38Z

Thanks. Much appreciated and let me know what I can do to help once you have the debug beta ready

…rk on #363

jeroenterheerdt · 2024-05-17T22:51:01Z

ok, here we go. Install https://github.com/jeroenterheerdt/HAsmartirrigation/releases/tag/v2024.5.1. You can do this through HACS assuming you have "show beta versions" on. Also, in your config, enable debug logging for custom_components.smart_irrigation:

logger:
  default: info
  logs:
    custom_components.smart_irrigation: debug

Your log will be flooded, so keep an eye on it. There will be lots of data to parse through when calculating but hopefully it will help to find any issues.

jata1 · 2024-05-17T23:34:50Z

Thanks. I have this all setup in prod and test so will do a 'calc all zones' manually later today in test and let prod run automatically tonight at 11pm.

Will update you over the weekend. Thanks for all your help with this. Hope we find the bug (if there is one)...

jeroenterheerdt · 2024-05-17T23:40:01Z

can I suggest to first test with a calculation per zone so you don't get too buried in messages? :)

jeroenterheerdt · 2024-05-17T23:45:19Z

please pull the new version of 5.1: https://github.com/jeroenterheerdt/HAsmartirrigation/releases/tag/v2024.5.1

jata1 · 2024-05-18T00:30:36Z

I think I have found a possible clue...

There is a function that aggregates data (apply_aggregates_to_mapping_data) before being pushed into the pyeto API. This looks like it uses an average for all elements and uses a multiplier that I think works out the proportion of a day included in the data.

The issue with this approach is that solrad needs to be added together not the average. See below the solrad being used is 1.36 (this is an average of night time solrad plus a early morning data where solrad is above zero but increasing fast.

So I think all that potentially is needed is to update the function so that solrad is added not the average. This will make a big difference I think.

2024-05-18 10:12:09.654 DEBUG (MainThread) [custom_components.smart_irrigation] apply_aggregates_to_mapping_data returns {'data_multiplier': 0.4038520889467592, 'Dewpoint': 10.580810992807177, 'Humidity': 81.53, 'Pressure': 1015.7644444444445, 'Solar Radiation': 1.3632000000000002, 'Maximum Temperature': 15.94, 'Minimum Temperature': 11.83, 'Temperature': 13.72, 'Windspeed': 0.09044444444444452, 'Precipitation': 0.0}
2024-05-18 10:12:09.654 DEBUG (MainThread) [custom_components.smart_irrigation] calculate_module for zone: {'id': 0, 'name': 'Front', 'size': 1.2, 'throughput': 8.0, 'state': 'automatic', 'bucket': -0.10947098986656402, 'old_bucket': 0, 'delta': 0, 'duration': 1, 'module': 0, 'multiplier': 1.0, 'explanation': None, 'mapping': 2, 'lead_time': 0.0, 'maximum_duration': 3600.0, 'maximum_bucket': 50, 'last_calculated': '2024-05-17T23:00:00.317711', 'last_updated': datetime.datetime(2024, 5, 18, 10, 10, 26, 298852), 'number_of_data_points': 12}, weatherdata: {'data_multiplier': 0.4038520889467592, 'Dewpoint': 10.580810992807177, 'Humidity': 81.53, 'Pressure': 1015.7644444444445, 'Solar Radiation': 1.3632000000000002, 'Maximum Temperature': 15.94, 'Minimum Temperature': 11.83, 'Temperature': 13.72, 'Windspeed': 0.09044444444444452, 'Precipitation': 0.0}, forecastdata: None

jata1 · 2024-05-18T00:45:14Z

I have found something else - but not sure if it is relevant

I see a number of solrad calculations that are taking place in pyETO: calculate_et_for_day - not sure if any of these are being used in the final eto calculation but...

In my setup I have a solrad sensor so there is no need to do any calculations to estimate solrad. Just need to pass in the correct (aggregated) solrad data into the model.

I hope this is helpful info for you

2024-05-18 10:12:09.663 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] weather_data: {'data_multiplier': 0.4038520889467592, 'Dewpoint': 10.580810992807177, 'Humidity': 81.53, 'Pressure': 1015.7644444444445, 'Solar Radiation': 1.3632000000000002, 'Maximum Temperature': 15.94, 'Minimum Temperature': 11.83, 'Temperature': 13.72, 'Windspeed': 0.09044444444444452, 'Precipitation': 0.0}
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] et_radvar: 18.708621038752977
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] cs_radvar: 14.031465779064732
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] estimated sol_rad from sunhours: 12.160603675189435
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] sol_rad passed to net_in_sol_radvar: 12.160603675189435
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] net_in_sol_radvar: 9.363664829895866
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] avp_from_tdew: 1.276581163515211 for tdew 10.580810992807177
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] net_out_lw_radvar: 4.96352000613332
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] net_radvar: 4.400144823762546
2024-05-18 10:12:09.664 DEBUG (MainThread) [custom_components.smart_irrigation.calcmodules.pyeto] [pyETO: calculate_et_for_day] eto: 1.105820250259664

jeroenterheerdt · 2024-06-16T14:37:06Z

Well, yes. We have the timestamps of each sensor update and now that we're calculating as soon as the sensor updates, we have a timestamp for the delta. I don't store it currently but it's there. But the point is that pyeto assumes the info it gets is for 24 hours. So it will return a Et based on the sensor values passed in for every sensor update. Hence, we need to handle that otherwise the bucket grows way to fast. So my thinking is: take the diff between the current time and the last time the sensor was updated. Calculate what fraction of an hour that is (used to be not a fraction really, but now with continuous updates it most likely is) and then divide that by 24 since there are 24 hours in a day. That number is then used to multiply the Et value pyeto returns which is for 24 hours as I said above.

Writing this out I think it makes sense, but curious to your thoughts.

jata1 · 2024-06-16T21:17:21Z

I see. Yes this makes sense in terms of working out the ETo for the day and then dividing this into the fraction of the day that the data relates to. So overall I think it makes sense.

What do you mean by doing calculations 'as soon as the sensors update'? Should you not just do the calculation based on sensor data every x minutes?

For my setup I am getting almost real-time data from my weather station but calculating ETo more than every 10mins is probably overkill.

jata1 · 2024-06-16T21:20:18Z

in other news - I am not getting anything since updating and resetting yesterday afternoon. I will turn on debug logging and send you info later today.

It was a sunny afternoon yesterday so this does not look correct.

jata1 · 2024-06-16T21:30:14Z

Just looking at my logs on prod (not test) and I see this. Prod is also running latest beta of smart irrigation

This error originated from a custom integration.

Logger: homeassistant
Source: custom_components/smart_irrigation/calcmodules/pyeto/__init__.py:236
integration: Smart Irrigation (documentation, issues)
First occurred: 6:07:52 AM (379 occurrences)
Last logged: 7:27:51 AM

Error doing job: Task exception was never retrieved (None)
Traceback (most recent call last):
  File "/config/custom_components/smart_irrigation/__init__.py", line 443, in async_sensor_state_changed
    await self.async_continuous_update_for_mapping(
  File "/config/custom_components/smart_irrigation/__init__.py", line 478, in async_continuous_update_for_mapping
    await self.async_calculate_zone(
  File "/config/custom_components/smart_irrigation/__init__.py", line 895, in async_calculate_zone
    data = await self.calculate_module(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/smart_irrigation/__init__.py", line 969, in calculate_module
    delta = modinst.calculate(
            ^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/smart_irrigation/calcmodules/pyeto/__init__.py", line 109, in calculate
    deltas.append(self.calculate_et_for_day(weather_data))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/smart_irrigation/calcmodules/pyeto/__init__.py", line 236, in calculate_et_for_day
    delta = precip - eto
            ~~~~~~~^~~~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

jeroenterheerdt · 2024-06-17T01:51:18Z

I see. Yes this makes sense in terms of working out the ETo for the day and then dividing this into the fraction of the day that the data relates to. So overall I think it makes sense.

What do you mean by doing calculations 'as soon as the sensors update'? Should you not just do the calculation based on sensor data every x minutes?

For my setup I am getting almost real-time data from my weather station but calculating ETo more than every 10mins is probably overkill.

You wanted to continuous update duration and bucket, right? For that we need to calculate ET values continously.

jeroenterheerdt · 2024-06-17T01:52:20Z

in other news - I am not getting anything since updating and resetting yesterday afternoon. I will turn on debug logging and send you info later today.

It was a sunny afternoon yesterday so this does not look correct.

What kind of value do you expect?

jeroenterheerdt · 2024-06-17T01:53:32Z

Just looking at my logs on prod (not test) and I see this. Prod is also running latest beta of smart irrigation

This error originated from a custom integration.

Logger: homeassistant
Source: custom_components/smart_irrigation/calcmodules/pyeto/__init__.py:236
integration: Smart Irrigation (documentation, issues)
First occurred: 6:07:52 AM (379 occurrences)
Last logged: 7:27:51 AM

Error doing job: Task exception was never retrieved (None)
Traceback (most recent call last):
  File "/config/custom_components/smart_irrigation/__init__.py", line 443, in async_sensor_state_changed
    await self.async_continuous_update_for_mapping(
  File "/config/custom_components/smart_irrigation/__init__.py", line 478, in async_continuous_update_for_mapping
    await self.async_calculate_zone(
  File "/config/custom_components/smart_irrigation/__init__.py", line 895, in async_calculate_zone
    data = await self.calculate_module(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/smart_irrigation/__init__.py", line 969, in calculate_module
    delta = modinst.calculate(
            ^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/smart_irrigation/calcmodules/pyeto/__init__.py", line 109, in calculate
    deltas.append(self.calculate_et_for_day(weather_data))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/smart_irrigation/calcmodules/pyeto/__init__.py", line 236, in calculate_et_for_day
    delta = precip - eto
            ~~~~~~~^~~~~
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

Interesting. If there is no rain does your weatherstation report unknown when there is no precipitation?

jata1 · 2024-06-17T03:25:29Z

No but this is the one weather sensor that I use a cloud based integration rather than local data for. So perhaps this is a one off error due to the cloud based sensor not being available at the time. I will monitor.

jata1 · 2024-06-17T22:49:40Z

Just a quick update that I am no longer getting any bucket updates at all so something in the latest beta has stopped calculations.

I turned on debug logs and have included some of these log entries (way too much to share the whole lot).

smart irrigation.log

jata1 · 2024-06-26T05:57:51Z

Hello @jeroenterheerdt - Hope all is well. Just checking in with you on this.

I know you are busy and this is not really urgent so no pressure :-)

I have switched my watering to my original 'dumb' smart watering system and it is doing fine. haha

jeroenterheerdt · 2024-06-27T10:27:07Z

Sorry busy with family emergency. What do you mean 'dumb'?

jata1 · 2024-06-27T22:36:11Z

Oh no - hope all OK with family stuff.

Before I started using smart irrigation I wrote my own system (dumb - compared to smart irrigation) that uses weather and rain to control the watering using different schedules and watering time. It has four scenarios:

no watering (when it is raining or rained a lot the day before)
low watering (dry but cold in the winter - max temp below 19 C)
medium (dry and quite hot - max temp 19-28 C)
high (dry and very hot - max temp above 28 C)

It works well and I used ETo (when working) to help me tune watering durations for the above scenarios.

I call it dumb but it is quite smart TBH :-)

wauswaus · 2024-06-28T10:49:06Z

Hi all, just went through all the messages.
At a certain point in time it was mentioned just to 'sum' all the measured radiation values. You can't just do that as a watt is a measurement per second unless you measure every second. Lets compare this to a light bulb that uses 100 watt (= 100 joules per second).
If i would measure the usage every minute and just add them would give me a value of 6000 (what unit has this?)
while the actual energy used is 100 watt (joules/s) times 3600 seconds = 3600000 joules.

I believe if you want to convert from loose watt/s measurements to a total you have to use a Riemann sum integral.
See https://www.home-assistant.io/integrations/integration/
Such an intergral can create a watts/day if I read it correctly.

I believe each measurement was always converted to MJ/M2/day when stored by this integration. The only proper thing you can do is average this out... unless a Riemann sum integral is used as this incorporates the time factor as well.

If i would now select 'sum' the integration sums all the calculated MJ/M2/day and this explains the gigantic bucket values you saw.
In my opinion the only proper way to make this work is to use a Riemann sum integral which could gave you a watt/m2/day value.
i see the integration also accepts MJ/M2/day so I can try to create a Riemann sum integral which spits out this value and then select maximum or last in the sensor group

jata1 · 2024-06-30T23:00:37Z

Thank you @wauswaus. Very nice of you to join this discussion :-)

I think you might be onto something with this line of thinking based on the prod version of the integration. It was massively underestimating watering duration - but not sure if Riemann sum integral would resolve things.

The current beta integration is doing almost real-time calculations but it is not working (yet) for my setup. So need to wait for the dev @jeroenterheerdt to comment when he has time.

wauswaus · 2024-07-01T15:31:21Z

I also use have a radiation sensor (in w/m2) and with the latest production version I don't have any big issues. I do let the integration update all the sensors every 5 minutes to get better granularity and have the radiation sensor set to average.

In the meanwhile I made Riemann sum integral sensor and that thing is only counting up... so I also created a statistics sensor that gives me the difference between latest measurement of the Riemann sum and the one 24 hours ago. I'm monitoring if that goes well. If so i'll feed that sensor in the smart irrigation and will use the latest value for the calculations.
I'll keep you posted.

jata1 · 2024-07-01T21:31:03Z

I am also playing around with a Riemann sum integral sensor and also noticed the issue with it not resetting daily but this is easy to resolve with a stats sensor or utility meter helper.

Interesting that you are getting good results with a 5 minute update period. Mine is default so every hour. Another thing to look at I think.

Keep me posted. Keen to get this all working again by next spring/summer so I have a few months to work on it.

wauswaus · 2024-07-02T18:04:17Z

ok, I created a Riemann sum integral sensor with as method 'left' and has as unit w/m2/d. That feeds in a statistics sensor with state_characteristic 'change' and max_age of 24h. That gives me a rolling amount of radiation of the last 24h.
This I convert to MJ/m2/d using a template sensor and feed this into the integration and use the 'last' for calculation

Now lets see what happens :)

Based on all the messages in this issue I have the feeling that the instant calculation that is now in the beta version is also incorrect due to the fact the time factor is not properly incorporated.

jata1 · 2024-07-02T21:24:12Z

That's great progress. Can you explain a couple of things to me please?

why 'left' method for reimann?
how did you configure for w/m2/d as I have mine in w/m2min so I'm doing something wrong?
can you give me the w/m2/d total you are getting so I can compare to my setup?
where are you located. Northern Hemisphere or Southern?

wauswaus · 2024-07-03T19:51:48Z

why 'left' method for reimann?
The integration page says for the left method "extremely accurate at estimating rectangular functions which are very stable for long periods of time and change very rapidly"
So i thought, that fits for solar radiation,

how did you configure for w/m2/d as I have mine in w/m2min so I'm doing something wrong?
You have use the time unit 'd'. You can also specify unit_prefix 'M', then it automatically also converts to MW. make sure you set the precision high enough!

can you give me the w/m2/d total you are getting so I can compare to my setup?
Sure... but is an always increasing value so no idea what you want to do with it...

This is from the last 2 days...

the w/m2 radiation for these 2 days:

where are you located. Northern Hemisphere or Southern?
I live in The Netherlands so Northern Hemisphere

jata1 · 2024-07-03T23:22:53Z

thank you @wauswaus - I will update my setup to the same as this and see how the bucket calcs look over the next week or so.

jeroenterheerdt · 2024-07-06T21:41:06Z

thanks both, I am back and will be working on the bugs in the latest beta. Also following the discussion on the reimann integral with interest, looks very promising. please let me know what you find!

jata1 · 2024-07-07T00:12:37Z

@jeroenterheerdt - welcome back. Hope all ok with the family stuff you have been dealing with.

I have 2 integral sensors configured. One using one minute intervals as this is the frequency of the source sensor data. I setup another using a daily interval like @wauswaus

Looking at the daily interval data, I think it looks too low/small to make a meaningful contribution to the ETO calc (so I will test both ways and report back.

jeroenterheerdt · 2024-07-07T19:21:52Z

@jata1 unfortunately a close family member has passed away, that was the emergency.

jeroenterheerdt · 2024-07-07T19:24:13Z

Just a quick update that I am no longer getting any bucket updates at all so something in the latest beta has stopped calculations.

I turned on debug logs and have included some of these log entries (way too much to share the whole lot).

smart irrigation.log

unfortunately these logs don't tell me what is going on. do you also see these #421 maybe?

wauswaus · 2024-07-08T06:04:55Z

@jeroenterheerdt - welcome back. Hope all ok with the family stuff you have been dealing with.

I have 2 integral sensors configured. One using one minute intervals as this is the frequency of the source sensor data. I setup another using a daily interval like @wauswaus

Looking at the daily interval data, I think it looks too low/small to make a meaningful contribution to the ETO calc (so I will test both ways and report back.

I don't see how you made 2 different ones. The integral is an ever increasing sensor so the only way is to also configure also a utility sensor which resets daily so you know the MW/m2/d.
I have it setup and it is working quite well for my setup. I have the feeling it works better then it did when just using the average of the raw measured values.

jeroenterheerdt · 2024-07-12T16:34:54Z

Hey, I started a discussion now that the feature is in experimental state in production. Can I ask you to move the discussion about how to process the solar radiation input there? Thanks! #437

jata1 added the bug Something isn't working label May 9, 2024

jata1 assigned jeroenterheerdt May 9, 2024

jeroenterheerdt added awaiting user verification and removed bug Something isn't working labels May 10, 2024

jeroenterheerdt added the bug Something isn't working label May 12, 2024

jeroenterheerdt added a commit that referenced this issue May 17, 2024

v2024.5.1 fix #356 (hopefully) and adding debug statements to help wo…

7234310

…rk on #363

jeroenterheerdt added a commit that referenced this issue Jul 7, 2024

v2024.7.0 fix #419, #401, progress for #363

b1f7284

jeroenterheerdt added the continuous_updates label Jul 19, 2024

jeroenterheerdt removed the bug Something isn't working label Jul 30, 2024

ChrisHomeAssistant mentioned this issue Jul 30, 2024

Solar Radiance is scaled when read from sensor even if already given in target unit #461

Closed

1 task

Daily bucket calculation is not correct when using a solar radiation sensor #363

Daily bucket calculation is not correct when using a solar radiation sensor #363

Comments

jata1 commented May 9, 2024

What happened?

How to reproduce

Relevant log output

Which version are you running?

Diagnostics file

Additional information

jata1 commented May 9, 2024

jata1 commented May 9, 2024

jeroenterheerdt commented May 10, 2024

jeroenterheerdt commented May 10, 2024

jata1 commented May 10, 2024

jata1 commented May 11, 2024

jeroenterheerdt commented May 11, 2024

jata1 commented May 12, 2024

jeroenterheerdt commented May 12, 2024

jata1 commented May 13, 2024

jata1 commented May 13, 2024

jata1 commented May 15, 2024

jeroenterheerdt commented May 15, 2024 • edited Loading

jata1 commented May 16, 2024

jata1 commented May 17, 2024

jata1 commented May 17, 2024

jeroenterheerdt commented May 17, 2024

jata1 commented May 17, 2024

jeroenterheerdt commented May 17, 2024 • edited Loading

jata1 commented May 17, 2024

jeroenterheerdt commented May 17, 2024

jeroenterheerdt commented May 17, 2024

jata1 commented May 18, 2024

jata1 commented May 18, 2024

jeroenterheerdt commented Jun 16, 2024

jata1 commented Jun 16, 2024

jata1 commented Jun 16, 2024

jata1 commented Jun 16, 2024

jeroenterheerdt commented Jun 17, 2024 • edited Loading

jeroenterheerdt commented Jun 17, 2024

jeroenterheerdt commented Jun 17, 2024

jata1 commented Jun 17, 2024

jata1 commented Jun 17, 2024

jata1 commented Jun 26, 2024

jeroenterheerdt commented Jun 27, 2024

jata1 commented Jun 27, 2024

wauswaus commented Jun 28, 2024 • edited Loading

jata1 commented Jun 30, 2024

wauswaus commented Jul 1, 2024 • edited Loading

jata1 commented Jul 1, 2024

wauswaus commented Jul 2, 2024

jata1 commented Jul 2, 2024

wauswaus commented Jul 3, 2024

jata1 commented Jul 3, 2024

jeroenterheerdt commented Jul 6, 2024

jata1 commented Jul 7, 2024

jeroenterheerdt commented Jul 7, 2024

jeroenterheerdt commented Jul 7, 2024

wauswaus commented Jul 8, 2024

jeroenterheerdt commented Jul 12, 2024

jeroenterheerdt commented May 15, 2024 •

edited

Loading

jeroenterheerdt commented May 17, 2024 •

edited

Loading

jeroenterheerdt commented Jun 17, 2024 •

edited

Loading

wauswaus commented Jun 28, 2024 •

edited

Loading

wauswaus commented Jul 1, 2024 •

edited

Loading