Unexpected error running initialize() for pv_opt: operands could not be broadcast together with shapes

[mergwyn]

Describe the bug
From about 03:30 this morning pv_opt started to fail to initialise with an error "operands could not be broadcast together with shapes". This looks similar to #104 but I don't havealt_tarrifsenabled. As pv_opt never gets to the end of the initialisation, the charging plan is not produced/updated.

I was on 3.7.2 to try to test #105 but have reverted to 3.6.2.
I'm running HA core (2024.1.6) and appdaemon (v4.4.2) in containers

Logs
The key part of the log appears to be:

024-02-10 08:53:29.870420 INFO pv_opt: 
2024-02-10 08:53:29.875473 INFO pv_opt: Previous message repeated 1 times
2024-02-10 08:53:29.875599 INFO pv_opt: Found Octopus Savings Events entity: event.octopus_energy_**********_octoplus_saving_session_events
2024-02-10 08:53:29.877202 INFO pv_opt: 
2024-02-10 08:53:29.877753 INFO pv_opt: No upcoming Octopus Saving Events detected or joined:
2024-02-10 08:53:29.878218 INFO pv_opt: 
2024-02-10 08:53:29.878710 INFO pv_opt: Starting Opimisation with discharge disabled
2024-02-10 08:53:29.879181 INFO pv_opt: --------------------------------------------
2024-02-10 08:53:29.879613 INFO pv_opt: 
2024-02-10 08:53:29.880112 INFO pv_opt: Checking tariffs:
2024-02-10 08:53:29.880590 INFO pv_opt: -----------------
2024-02-10 08:53:29.885497 INFO pv_opt:   Import: E-1R-AGILE-23-12-06-F                    Start: 2024-01-31 13:00:00+0000 End: 2024-02-10 23:00:00+0000 
2024-02-10 08:53:29.890381 INFO pv_opt:   Export: E-1R-AGILE-OUTGOING-19-05-13-F           Start: 2024-01-31 13:00:00+0000 End: 2024-02-10 23:00:00+0000 
2024-02-10 08:53:29.890858 INFO pv_opt: AGILE tariff detected. Rates will update at 16:00 daily
2024-02-10 08:53:29.891319 INFO pv_opt:   - Tariffs OK
2024-02-10 08:53:29.891768 INFO pv_opt: 
  index=pd.date_range(
2024-02-10 08:53:29.897114 INFO pv_opt: Solcast forecast loaded OK
2024-02-10 08:53:29.897663 INFO pv_opt: Getting expected consumption data
  index = pd.date_range(start, end, inclusive="left", freq="30T")
  df.diff(-1).fillna(0).clip(upper=0).cumsum().resample("30T")
  time_value = pd.to_datetime(state, errors="ignore", format="%H:%M")
2024-02-10 08:53:29.914358 WARNING pv_opt: ------------------------------------------------------------
2024-02-10 08:53:29.914517 WARNING pv_opt: Unexpected error running initialize() for pv_opt
2024-02-10 08:53:29.914636 WARNING pv_opt: ------------------------------------------------------------
2024-02-10 08:53:29.915853 WARNING pv_opt: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/appdaemon/app_management.py", line 162, in initialize_app
    await utils.run_in_executor(self, init)
  File "/usr/local/lib/python3.10/site-packages/appdaemon/utils.py", line 304, in run_in_executor
    response = future.result()
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home-assistant/appdaemon/apps/pv_opt/pv_opt.py", line 325, in initialize
    self.optimise()
  File "/usr/local/lib/python3.10/site-packages/appdaemon/adbase.py", line 35, in f_app_lock
    return f(*args, **kw)
  File "/home-assistant/appdaemon/apps/pv_opt/pv_opt.py", line 1316, in optimise
    consumption = self.load_consumption(
  File "/home-assistant/appdaemon/apps/pv_opt/pv_opt.py", line 1943, in load_consumption
    x = temp.to_numpy() + y.to_numpy()
ValueError: operands could not be broadcast together with shapes (96,) (12,) 

2024-02-10 08:53:29.916030 WARNING pv_opt: ------------------------------------------------------------
2024-02-10 08:53:29.916583 INFO AppDaemon: App initialization complete
2024-02-10 08:54:29.828304 INFO pv_opt: >>> Agile Callback Handler

I've restarted HA and appdaemon. This is the appdaemon container log post the most recent recent restart (excluding the FutureWarnings):
appdaemon.log

My config.yaml;
config.yaml.log

I have the separate main.log, error.log and pv_opt.logs for the last few days if this helps (I turned the filter off this morning as I throught it was easier to see the logs in one place).

Many thanks for your help.

[fboundy]

I've added some additional logging to 3.7.4 which should help diagnose this

[fboundy]

Can you please add debug: True to your config.yaml:

     prefix: pvopt
     debug: True

[mergwyn]

Thanks Francis: Here is a a new log (updated):

appdaemon.3.7.4.log

[fboundy]

Can you send the error.log please?

…

On 10 Feb 2024 at 13:41 +0000, mergwyn ***@***.***>, wrote: Thanks Francis: Here is a a new log: appdaemon.3.7.4.log — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[mergwyn]

HI - here is my error.log. Note that just after this log was produced I turned off the filtering such that all the logs were combined, so this log stops at 0850 this morning. The last error "operands could not be broadcast together with shapes" is the same as the last error in the appdaemon.log I posted earlier.
error.log

[fboundy]

OK - please can you run 3.7.5 which is a pre-release with (yet) more logging of the consumption calculation which is where this is falling down.

[mergwyn]

Here you go:
appdaemon.3.7.5.log

Thanks again for your help!

[SzosszeNET]

As a hunch @mergwyn make a copy of your configuration file and download the clean configuration file from the repo and start fresh? Think I have encountered similar issue and ended up an indentation issue.

[mergwyn]

Worth a punt but no joy:
appdaemon.3.7.5-fresh-config.log

config.yaml-fresh.log

[fboundy]

Thanks for the log - it’s something to do with the amount of consumption data that you are downloading. You should be getting 7 days but I’m not seeing that much. I’ll add some extra QC and logging to the load_consumption method.

…

On 11 Feb 2024 at 09:35 +0000, mergwyn ***@***.***>, wrote: Worth a punt but no joy: appdaemon.3.7.5-fresh-config.log config.yaml-fresh.log — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[fboundy]

Please try 3.8.4 which has more logging and some additional checking.

[mergwyn]

All working again on 3.8.4 - many thanks for such a quick fix