datetime.now() method consequences for submitted forecasts

[purcell-lab]

The publish-data command will load the latest csv file and look for the closest timestamp that match the current time using the datetime.now() method in Python. This means that if EMHASS is configured for 30min time step optimizations, the csv will be saved with timestamps 00:00, 00:30, 01:00, 01:30, … and so on. If the current time is 00:05, then the closest timestamp of the optimization results that will be published is 00:00. If the current time is 00:25, then the closest timestamp of the optimization results that will be published is 00:30.

One (unintended) consequence of the datetime.now() method results when submitting forecast data.

Consider the data from my energy provider:

Which is represented in HA as the following entities and attributes:

sensor.amber_general_price: 0.36
sensor.amber_general_forecast: 0.37
'forecasts.per_kwh': [0.37, 0.38, 0.38, 0.38, 0.38, 0.38, 0.38, 0.38, 0.39, 0.43, 0.45, 0.72, 0.73, 0.73, 0.72, 0.45, 0.7, 0.46, 0.45, 0.39, 0.36, 0.32, 0.32, 0.33, 0.7, 0.7, 0.51, 0.43, 0.51, 0.61, 0.46, 0.47, 0.61, 15.12, 15.08, 0.74, 0.52, 0.61, 0.52, 0.45, 0.44, 0.43, 0.45, 0.44, 0.38, 0.36, 0.43, 0.39]

• If I conduct an optimisation at 00:35 the optimisation results will be published for 00:30, so the first value needs to be the now value, followed by the forecast for 01:00, then 01:30 and so on.

"load_cost_forecast":(sensor.amber_general_price + forecasts.per_kwh)[:48]
"load_cost_forecast": [0.36, 0.37, 0.38, 0.38, ...]

[:48] is to truncate to 48 elements as that list will have 49 elements

• If I conduct an optimisation at 00:55 the optimisation results will be published for 01:00, so the first value needs to be the forecast value for 01:00, the 01:30 and so on. The Now value isn't used.

"load_cost_forecast":(forecasts.per_kwh)
"load_cost_forecast":[0.37, 0.38, 0.38, 0.38, ...]

Whilst not impossible to calculate the changing load_cost_forecast, it is confusing as the data it requires changes depending on which 15 minute block it is generated in.

Additionally I would submit the first optimisation including the now value is superior as it includes more data, to base the optimisation on and this is also important for the MPC as it will be called with high frequency (maybe every 5 minutes).

Could I propose that the optimisation should always include the now value as the first element of the forecast list and that for consistency the published optimisation shouldn't map to the closest timestamp (which maybe ahead of behind), but that the published optimisation is always published with the start of the current time step. So if I publish at either 00:05 or 00:25 it will be published for 00:00, if I publish at either 00:35 or 00:55 it will be published for 00:30.

[davidusb-geek]

I guess that the best thing to do is to pass an extra parameter to the publish command to choose the type of handling of the timestamps from the csv file. Something like a parameter called method that can take values nearest, first, last. However for the list of passed forecast values is up to the user to arrange the list to contain the first element as the current now element.

You could contour these issues with the prediction_horizon parameter. If you fix it to a relative small value (say the default 10) this can now give you the possibility to use a smaller optimization time step, maybe 10min. A consequence of reducing the prediction horizon is that the number of constraints in the LP formulation is also reduced. So you can reduce your time step for better precision when publishing the data.

So if I publish at either 00:05 or 00:25 it will be published for 00:00, if I publish at either 00:35 or 00:55 it will be published for 01:00.

I don't get this last bit. According to your explanation if you publish at either 00:35 or 00:55, then you would want to publish the values stored at timestamp 00:30 and not 01:00 as you said? This would be using method=first. Or I'm missing something?

[purcell-lab]

So if I publish at either 00:05 or 00:25 it will be published for 00:00, if I publish at either 00:35 or 00:55 it will be published for 01:00.

I don't get this last bit. According to your explanation if you publish at either 00:35 or 00:55, then you would want to publish the values stored at timestamp 00:30 and not 01:00 as you said? This would be using method=first. Or I'm missing something?

Sorry, my error, I am meaning 00:30, edited original post.

[purcell-lab]

I guess that the best thing to do is to pass an extra parameter to the publish command to choose the type of handling of the timestamps from the csv file. Something like a parameter called method that can take values nearest, first, last. However for the list of passed forecast values is up to the user to arrange the list to contain the first element as the current now element.

That would be great a great solution having options for nearest, first & last.

[davidusb-geek]

In the new version a new parameter was added to the configuration. It is named method_ts_round and it can take options: first, last and nearest. By default it is fixed to nearest so you will want to change that to first.

[purcell-lab]

method_ts_round addresses this issue.