day-ahead optimisation fails since 0.9.0

[ragg987]

Describe the bug
Getting 2 errors when I run dayahead optimisation and the published results do not update. I think this issue started after release 0.9.0. I have not made any change to Emhass config for over a week. It last ran successfully yesterday 12/05/2024.

To Reproduce
Run day-ahead optimisation. My automation call is (have obfuscated some of my sensor names):
shell_command:
dayahead_optim: "curl -i -H "Content-Type:application/json" -X POST -d '{}' http://localhost:5001/action/dayahead-optim"
publish_data: "curl -i -H "Content-Type:application/json" -X POST -d '{}' http://localhost:5001/action/publish-data"
octopusagile_optim: "curl -i -H "Content-Type:application/json" -X POST -d '{"load_cost_forecast":{{
[states('sensor.electricity_xxx_xxx_current_rate') |float(0)]
+ ( state_attr('event.octopus_energy_electricity_xxx_xxx_current_day_rates', 'rates')|selectattr('start','gt',utcnow()) | map(attribute='value_inc_vat')|list
+ state_attr('event.octopus_energy_electricity_xxx_xxx_next_day_rates', 'rates')|selectattr('start','gt',utcnow()) | map(attribute='value_inc_vat')|list ) [:48]}}}'
http://localhost:5001/action/dayahead-optim"

Expected behavior
Dayahead completes

Screenshots
Error log:
2024-05-13 19:48:24,882 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2024-05-13 19:48:24,882 - web_server - INFO - Home Assistant data fetch will be performed using url: http://supervisor/core/api
2024-05-13 19:48:24,882 - web_server - INFO - The data path is: /data
2024-05-13 19:48:24,884 - web_server - INFO - Using core emhass version: 0.9.0
waitress INFO Serving on http://0.0.0.0:5000
2024-05-13 19:48:29,647 - web_server - INFO - EMHASS server online, serving index.html...
2024-05-13 19:49:00,054 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.196245, 0.185535, 0.16758, 0.15876, 0.134715, 0.1323, 0.11466, 0.12285, 0.112875, 0.10584, 0.095235, 0.085995, 0.08799, 0.085995, 0.077175, 0.077805, 0.077175, 0.085575, 0.08022, 0.10941, 0.121275, 0.147735, 0.147945, 0.1323, 0.16317, 0.18963, 0.1764, 0.183435, 0.182805, 0.187425, 0.18522, 0.17556, 0.17283, 0.17199, 0.17283, 0.147945, 0.13188, 0.134505, 0.12789, 0.125475, 0.134505, 0.28266, 0.288225, 0.30408, 0.319935, 0.341565, 0.341565, 0.2205, 0.21609]}
2024-05-13 19:49:00,055 - web_server - INFO - >> Setting input data dict
2024-05-13 19:49:00,055 - web_server - INFO - Setting up needed data
2024-05-13 19:49:00,105 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2024-05-13 19:49:00,643 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 109, in action_call
input_data_dict = set_input_data_dict(emhass_conf, costfun,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 110, in set_input_data_dict
P_PV_forecast = fcst.get_power_from_weather(df_weather)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 420, in get_power_from_weather
cec_modules = bz2.BZ2File(self.emhass_conf['root_path'] / 'src/emhass/data/cec_modules.pbz2', "rb")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/bz2.py", line 81, in init
self._fp = _builtin_open(filename, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/app/src/emhass/data/cec_modules.pbz2'

Home Assistant installation type

• Home Assistant OS

Your hardware

• OS: HA OS
• Architecture: amd64

EMHASS installation type

• Add-on

Additional context
The other change to my system has been update of HAOS to 2024.5.2. I have tried restating EMHASS as well as rebooting hAOS, neither have been successful.

[davidusb-geek]

Hi @GeoDerp, any idea why is this happening.
The code is trying to look for those files at: /app/src/emhass/data/
But actually they are in: /usr/local/lib/python3.11/dist-packages/emhass/data/
A quick but ugly solution would be to add them manually by copying, for example on line 69 of Dockerfile:

COPY cec_modules.pbz2 /app/src/emhass/data/
COPY cec_inverters.pbz2 /app/src/emhass/data/

But how come that the other files are found without problem, for example: static, template, etc.
Any ideas?

[davidusb-geek]

This error comes from using this as the root path: self.emhass_conf['root_path']
This was changed recently. I just lost track of where do we define that root_path parameter.
I'm releasing a patched version where I'm directly using pathlib.Path(__file__).parent to build a relative path.

[davidusb-geek]

@ragg987, this is solved with patch v0.9.1

[ragg987]

Thank you. Confirmed it runs now.

[davidusb-geek]

Great! closing...

[GeoDerp]

Sorry @davidusb-geek , I will look at this when I have some free time next.