Incorrect Tariff using Intelligent Octopus

[ColinRobbins]

Describe the bug

If using Intellilgent Octopus, and Octopus decide to charge your EV during peak time, the tariff changes to the off-peak tariff.
However, the tariff shown in this integration is still the peak tariff.

To Reproduce

Signup to Intelligent Octopus and observer sensor behaviour.

Expected behavior

The tariff should accurately reflect the current rate.

Home Assistant Version

The version of Home Assistant.

Integration Version

2022.12.8

[BottlecapDave]

Thanks for raising this. Unfortunately, I'm not sure where this information would come from as this doesn't look like Octopus exposes this information in their API. If you are able to point me in the correct direction, then I'll be able to look at it.

[ColinRobbins]

I'm not sure either!

This integration: https://github.com/megakid/ha_octopus_intelligent is able to find out if "intelligent" has switched to the off-peak rate. But does not have the tariff.
The tariff is the same as the overnight offpeak rate.

[github-actions]

This issue has become stale because it has been open for 30 days with no activity. If you still think it's an issue, please respond soon.

[megakid]

Author of the above octopus intelligent integration here. @BottlecapDave I was considering posing the question of merging my integration into yours. What are your thoughts on this? I may be able to do the work on the my hour long commutes but would obviously want your blessing first given the size of the task (there's quite a lot of different sensors and other read/write controls with Intelligent tariff).

The Octopus Intelligent tariff seems popular with EV owners so it would be nice if we teamed up to create the one-stop-shop of Octopus Energy Home Assistant Integration for Home Assistant. Thoughts?

[BottlecapDave]

Hi there @megakid. The main reason why I haven't done anything around intelligent tariff is that I'm unable to test the scenarios (and therefore fine the solution in some cases). I also didn't want to be stepping on toes of other creators like yourself. If you're wishing to contribute your work to this integration, I'm all for it :)

I've looked through your integration and here's my current thoughts.

1. The switches all look like great additions. We'll only want these to appear for users where applicable (similar to the mini, but ideally we'll want to determine it based on their tariff/owning product or by making an "intelligent" based call and checking if data is returned).
2. I try and have as little configuration as possible and refer config from the data provided by OE. It looks like your integration has the offpeak times configured by the user. We can probably calculate these from the standing rates. It looks like OE has gone against their convention of 1R/2R in the tariff code to determine off peak :(.
3. I also try to use data coordinators as much as possible to share data across sensors and reduce API calls. It might be good to look into some of those.
4. We'll want as much automated testing as possible. We probably won't be able to include integration tests, but unit tests are a must - especially around any calculations. I've tried to keep my clients dumb and move all calculations to external functions. Testing can always be improved, but what's there at the moment is good enough.
5. It would be good to know the use cases around some of your sensors and which ones might be able to be covered with something else. Below is my quick thoughts.

• sensor.octopus_intelligent_next_offpeak_start and sensor.octopus_intelligent_next_offpeak_end - Could this be covered by a target rate sensor? Or perhaps we need an "off peak" sensor which has this data available as attributes?
• binary_sensor.octopus_intelligent_slot_next_x_hour - These might be covered by target rate sensors?
• binary_sensor.octopus_intelligent_planned_dispatch_slot - This seems important and useful to keep
• binary_sensor.octopus_intelligent_slot - would this be covered by an "off peak" sensor or a target rate sensor?

6. It would be good to fix the issue that originally raised this ticket.
7. Finally, I think we can release these features slowly. They don't need to be all at once.

[tomfielder]

+1 on this. Thank you both @BottlecapDave @megakid for your work on these fantastic integrations.

I've hit the same issue today as OP. I've plugged my car in this morning and OE have set my smart charging schedule (i.e. off peak rate) to be from midday today to 4pm, continuing at 11pm through to the following 7:30am. I use the current_rate entity on my Energy Dashboard, which 90% of the time is fine - it's just scenarios like this one where it's showing me the peak rate instead of off peak.

[BottlecapDave]

@megakid have you decided if you want to port intelligent features into this integration? I was going to try and look at the original bug for within the next week, but obv will have trouble testing the fix. I also don't mind looking at other parts (e.g. the switches)

[BottlecapDave]

@ColinRobbins @tomfielder @megakid I've started looking at this issue and intelligent support at a wider level.

I've created a beta release that hopefully fixes this issue as well as laying the ground work for future intelligent support. However I've done this a bit blind as I'm not on this tariff.

It would be good to know the following

• Does it fix the initial issue around rates (this might take days/weeks to confirm)
• Does the new sensor and device work?
• What data do you get for planned and complete dispatches?
• Do you get any other errors?

The release can be found at https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/intelligent-beta-1

[megakid]

@megakid have you decided if you want to port intelligent features into this integration? I was going to try and look at the original bug for within the next week, but obv will have trouble testing the fix. I also don't mind looking at other parts (e.g. the switches)

Feel free to take some of the code etc from my integration - my free time has taken a massive hit in the last few weeks (baby arrived) so unlikely to get to it myself.

[BottlecapDave]

Feel free to take some of the code etc from my integration - my free time has taken a massive hit in the last few weeks (baby arrived) so unlikely to get to it myself.

No problem. My main priority is to fix the issue at hand, and then potentially provide some of the other app/integration features in the future. Congratulations on the new addition to your family!

[andyvee69]

I can help test this with IO - I'm on the Ohme integration so it behaves very differently in terms of plannedDispatches to a vehicle integration. I've installed the beta version so will see how it 'behaves' on my next charge tonight.

[megakid]

I can also test this - I have a Tesla so it will test different paths to @andyvee69's Ohme Octopus Intelligent functionality. I can perform GraphQL requests and give you the output too

[BottlecapDave]

@andyvee69 @megakid Has anyone had a chance to check on this fix? The part I'm mainly interested in is what is being returned by plannedDispatches and completedDispatches as there's no filtering ability in the API.

[RPCHome]

@andyvee69 @megakid Has anyone had a chance to check on this fix? The part I'm mainly interested in is what is being returned by plannedDispatches and completedDispatches as there's no filtering ability in the API.

I would love to be able to provide feedback regarding what is returned by plannedDispatches and completedDispatches; however, as I'm still fairly new to HA I'm unsure how to get this info for you.
If it help I have set up a few automation that uses the 'plannedDispatches' using the sensor binary_sensor.octopus_intelligent_planned_dispatch_slot. I don't seem to have a completedDispatches entity

If you point me in the right direction I'll be more than happy to help test any upgrades/beta

[BottlecapDave]

@andyvee69 @megakid Has anyone had a chance to check on this fix? The part I'm mainly interested in is what is being returned by plannedDispatches and completedDispatches as there's no filtering ability in the API.

I would love to be able to provide feedback regarding what is returned by plannedDispatches and completedDispatches; however, as I'm still fairly new to HA I'm unsure how to get this info for you. If it help I have set up a few automation that uses the 'plannedDispatches' using the sensor binary_sensor.octopus_intelligent_planned_dispatch_slot. I don't seem to have a completedDispatches entity

If you point me in the right direction I'll be more than happy to help test any upgrades/beta

@RPCHome The mentioned attributes are present in the beta release of this integration, it isn't present on the octopus intelligent integration. The plan is to add more support for intelligent in this integration, but as I don't have access to the tariff I'm relying on users for support. If you wish to help, then you'll need to install the beta and confirm back what you have in those attributes after a couple of days. This will help me work out if there is more work on my end to do for that part. Once added into the main integration I can start adding the other pieces.

[megakid]

@BottlecapDave I can't see the beta release you released in the drop down in HACS:

[BottlecapDave]

Looks like HACs only support so many versions behind. I've created a new beta release which should have the feature enabled.

[megakid]

I'm not sure how to enable the new intelligent tariff sensors. I've updated to the latest beta and restarted but don't get any new config options or sensors.

[BottlecapDave]

There should be a sensor called binary_sensor.octopus_energy_intelligent_dispatching. What is your reported tariff under your electricity current_rate sensor as I might not be looking for the correct tariff?

[megakid]

Here's the tariff code. The rates are correct

[BottlecapDave]

That tariff should be supported and only differs by region in my testing. I'll have to take another look. I assume there's no errors in your HA logs?

[BottlecapDave]

I'm really sorry it looks like the last build didn't work correctly. Please try https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/intelligent-beta-3

[megakid]

Ok tried it. Charger device now appears.

Only thing to show at the moment is that is goes unavailable quite often. The other sensors do not so perhaps there's a low timeout on the API here or something.

[BottlecapDave]

Glad it's finally appearing. Interesting though that it's going offline a lot and that you don't have anything in your attributes (which suggests the two are probably linked). Do you have any errors in your HA instance that might hint at the issue?

[megakid]

Yeh good point. I scheduled a charge on my car and was waiting for it to appear but nothing yet so I guess the unavailability is linked.

Will check the logs

[BottlecapDave]

hmm yeah I've double checked the code and nothing has changed around that sensor. I've released a new version, but again this shouldn't have any effect. I assume your tariff hasn't changed recently which might stop the sensor from being added? This can be determined by following this guide and pasting the results here. I also assume if you reload the integration the sensor doesn't come back?

[RPCHome]

Updated to beta 10, still unavailable (I have reloaded as well) My tariff hasn't changed yet. Will be changing next Sunday on the 11th

  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2023.5.4",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.11",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "6.1.21-v8",
    "supervisor": "2023.04.1",
    "host_os": "Home Assistant OS 10.2",
    "docker_version": "23.0.6",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "octopus_intelligent": {
      "version": "1.0.0",
      "requirements": [
        "gql==3.2.0"
      ]
    },
    "hacs": {
      "version": "1.32.1",
      "requirements": [
        "aiogithubapi>=22.10.1"
      ]
    },
    "hikvision_next": {
      "version": "1.0.5",
      "requirements": [
        "xmltodict==0.13.0",
        "hikvisionapi==0.3.2",
        "requests-toolbelt==0.10.1"
      ]
    },
    "gardena_smart_system": {
      "version": "1.0.0",
      "requirements": [
        "py-smart-gardena==1.3.7"
      ]
    },
    "pirateweather": {
      "version": "1.1.2",
      "requirements": [
        "python-forecastio==1.4.0"
      ]
    },
    "skyq": {
      "version": "v2.10.14",
      "requirements": [
        "pyskyqremote==0.3.25"
      ]
    },
    "fordpass": {
      "version": "0.1.46",
      "requirements": []
    },
    "dyson_local": {
      "version": "0.16.4-4",
      "requirements": [
        "libdyson==0.8.11"
      ]
    },
    "whatspie": {
      "version": "1.0.0",
      "requirements": []
    },
    "nest_protect": {
      "version": "0.3.12",
      "requirements": []
    },
    "alarmo": {
      "version": "v1.9.9",
      "requirements": []
    },
    "browser_mod": {
      "version": "2.2.2",
      "requirements": []
    },
    "octopus_energy": {
      "version": "7.4.2",
      "requirements": []
    },
    "miele": {
      "version": "0.1.13",
      "requirements": [
        "flatdict==4.0.1",
        "pymiele==0.1.6"
      ]
    }
  },
  "integration_manifest": {
    "domain": "octopus_energy",
    "name": "Octopus Energy",
    "codeowners": [
      "@bottlecapdave"
    ],
    "config_flow": true,
    "dependencies": [
      "repairs",
      "recorder"
    ],
    "documentation": "https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/",
    "homekit": {},
    "iot_class": "cloud_polling",
    "issue_tracker": "https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/issues",
    "ssdp": [],
    "version": "7.4.2",
    "zeroconf": [],
    "is_built_in": false
  },
  "data": {
    "electricity_meter_points": [
      {
        "mpan": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "is_export": false,
            "is_smart_meter": true,
            "device_id": "**REDACTED**",
            "manufacturer": "1125 - Shenzhen Kaifa Metering",
            "model": "31324131",
            "firmware": "17101617"
          }
        ],
        "agreements": [
          {
            "valid_from": "2022-06-18T23:00:00+00:00",
            "valid_to": "2023-06-10T23:00:00+00:00",
            "tariff_code": "E-1R-INTELLI-22-03-29-J",
            "product_code": "INTELLI-22-03-29"
          },
          {
            "valid_from": "2023-06-10T23:00:00+00:00",
            "valid_to": null,
            "tariff_code": "E-1R-INTELLI-VAR-22-10-14-J",
            "product_code": "INTELLI-VAR-22-10-14"
          }
        ]
      }
    ],
    "gas_meter_points": [
      {
        "mprn": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "consumption_units": "m\u00b3",
            "is_smart_meter": true,
            "device_id": "**REDACTED**",
            "manufacturer": "1192 - George Wilson",
            "model": "11111111",
            "firmware": "01C800D1"
          }
        ],
        "agreements": [
          {
            "valid_from": "2023-03-31T23:00:00+00:00",
            "valid_to": null,
            "tariff_code": "G-1R-VAR-22-11-01-J",
            "product_code": "VAR-22-11-01"
          }
        ]
      }
    ]
  }
}```

[wralb]

@RPCHome - Could it be that something has gone wrong with your beta 10 installation? It seems from the code you pasted in the previous post that you might be running production version 7.4.2 of Dave's excellent component.

@BottlecapDave - Dave, I read this thread with interest at the weekend. I have an Ohme charger and Intelligent Octopus so I thought I'd see whether I could give you some feedback. I installed your Beta 10 via HACS yesterday. Just after I plugged my car in at 15:07 this afternoon a planned dispatch appeared; see picture.

I see it is for 23.00 BST to 05.00. Could it be out by half an hour and really represents the normal off-peak period of 23.30 to 05:30? Just wondering... otherwise I'm getting an extra half hour of off-peak.

Let me know if you want any further and/or different feedback.

[BottlecapDave]

@wralb Welcome to the beta and any additional feedback is welcome. I think you might be right with what you suggested to @RPCHome based on his diagnostics data.

In terms of the planned/completed dispatches in that sensor, the data comes straight from OE without any modifications (other than some minor transformations in shape). I then use that data to 1) turn on/off the sensor and 2) adjust the rate/previous accumulative cost sensors to move any times within planned/completed dispatches to off peak rates.

From what I've read with the tariff, planned dispatches can fall outside of the guaranteed hours if OE want to balance the grid. However again I've read (but could be wrong) that you will only get the off peak rates outside of those hours if your car is charging (which is why I'm unsure if it's right to adjust the rate sensor with planned dispatches as this might provide false data)

[BottlecapDave]

The main things I want to test for this release are

1. Does the previous accumulative cost look right in terms of off peak rates for completed dispatches
2. Does the previous/current/next sensors look right in terms of planned/completed dispatches (as mentioned, this I'm unsure if it's correct behaviour to do this as we might not have all of the data available to correctly determine this)
3. Knowing how far back the completed dispatches go (not that I have any control over this)

Once these have been confirmed, I'll look to do a separate beta release with the other missing intelligent related sensors

• Binary sensor to turn on/off intelligent charging
• Binary sensor to turn on/off bump charging
• Number sensor to set desired charge percentage
• Time sensor (new in upcoming release) to set desired charge time

[RPCHome]

Thank you @wralb for point out the error of my ways. I'm not sure what went wrong but beta 10 didn't install correctly. Just re-done it and it now reporting the sensor correctly.

@BottlecapDave To answer a few of the things you have mentioned in your post 2 above. You are correct in thinking that planned dispatched can and do fall outside of the guaranteed hours (this happens a lot when we charge and we make use of the cheap rate)

You do only get the cheap rate if the car is plugged in and charging. We have found that sometimes we have had to unplug and use the car. When this happens, the OE app updates at the end of the half hour window and reverts back to the Peak rate (the other Octopus intelligent integration also reflect this, I haven't tried it on the beta tests yet) We have crossed checked this on the bill a number of times as we were unsure if we were going to be charged peak rate

In addition to this, when the planned dispatch kicks in mid-way through the half-hour window, ie 7 mins pas the hour, the off-peak rate is back-dated to the half hour window start time (ie xx:00 or xx:30). (hope that all make sense) So adjusting the rate with the planned dispatches is okay and doesn't cause false data.

Moving on to your questions

1. The previous accumulative cost at the current time looks correct. However my bill is printed around the 28th of the month so Let me run this for a month and cross reference it in order to give you a 100% yes or no answer.
2. I don't recall looking at previous/current/next sensors during my last charging session so will look at it next time
3. I have updated the beta so again I can't see this info so can't comment on it. However I'm unsure what use completed dispatches info will be. Happy to be educated otherwise.

All the other sensors will be welcomed, once all the above has been confirmed.

Any more info or questions just ask.

[BottlecapDave]

The completed dispatches is required in order to calculate the previous accumulative cost due to planned dispatches disappearing once they are in the past.

[RPCHome]

Ah okay makes sense about the completed dispatched.

Just plugged in this morning to get a cheeky top-up change before 1100 and found that the beta 10 isn't showing any planned dispatches. But the other integration is. in the below screen show the beta is on the left and the other is on the right.

[wralb]

It's going to take a little time for me to get my head fully around what the data I'm seeing means.

Firstly, evidence from my bills indicates that the car does NOT have to be charging for me to benefit from additional off-peak periods. It also seems that the car does NOT need to be plugged in either. Here is a snippet of a bill for 16th April. I know for certain that I was out in the car between 9 AM and approx. 11 AM. The bill shows several additional off-peak sessions in the morning, for which I only have baseline electricity usage and the charges for those periods are clearly at the off-peak rate.

These off-peak additional period during the day seem quite rare, but it does seem that Octopus quite frequently add an extra half hour before and/or after the "guaranteed off-peak" period.

Now for more confusing data.

• I reported seeing a planned_dispatch after plugging in my car yesterday around 3PM.
• I had to then unplug the car for an unplanned outing at 6 PM to 7.30.
• When I looked at the dispatch sensor a little after my return, the planned_dispatch had vanished, but there were FIVE completed_dispatch entries covering the time my car had been plugged in.
• The binary_sensor.octopus_energy_intelligent_dispatching has never been in the true state according to Home Assistant history
• I was up before 5 AM today and so checked the dispatch data. Most of the previous afternoon's completed dispatch data were still there. They gradually disappeared through the next couple of hours, so it seems (from one data point) that completed_dispatch data may disappear after approximately 12 hours.

[RPCHome]

It's going to take a little time for me to get my head fully around what the data I'm seeing means.

Firstly, evidence from my bills indicates that the car does NOT have to be charging for me to benefit from additional off-peak periods. It also seems that the car does NOT need to be plugged in either. Here is a snippet of a bill for 16th April. I know for certain that I was out in the car between 9 AM and approx. 11 AM. The bill shows several additional off-peak sessions in the morning, for which I only have baseline electricity usage and the charges for those periods are clearly at the off-peak rate.

These off-peak additional period during the day seem quite rare, but it does seem that Octopus quite frequently add an extra half hour before and/or after the "guaranteed off-peak" period.

This is very confusing indeed. I've never had it where I'm getting the off peak rate while the car isn't plugged in. However you are using the Ohme charger which OE recommend, so it might work slightly differently to my charger.

Now for more confusing data.

• I reported seeing a planned_dispatch after plugging in my car yesterday around 3PM.
  This happens often when I plug in and there is 'green' energy they want to use.

• I had to then unplug the car for an unplanned outing at 6 PM to 7.30.
• When I looked at the dispatch sensor a little after my return, the planned_dispatch had vanished, but there were FIVE completed_dispatch entries covering the time my car had been plugged in.

Again this is correct, I find the planned_dispatch does vanish as the car isn;t plugged in or charging. However then it doesn't make sense with you bill screenshot you provided. Would be interesting to see what you bill shows for yesterday.

• The binary_sensor.octopus_energy_intelligent_dispatching has never been in the true state according to Home Assistant history
• I was up before 5 AM today and so checked the dispatch data. Most of the previous afternoon's completed dispatch data were still there. They gradually disappeared through the next couple of hours, so it seems (from one data point) that completed_dispatch data may disappear after approximately 12 hours.

Good to know. I've set an alarm to check my completed_dispatch 12hr after they finished. @BottlecapDave I'll post my results here

[BottlecapDave]

Ah okay makes sense about the completed dispatched.

Just plugged in this morning to get a cheeky top-up change before 1100 and found that the beta 10 isn't showing any planned dispatches. But the other integration is. in the below screen show the beta is on the left and the other is on the right.

@RPCHome Do you have any errors in your logs that might indicate why you don't have any planned dispatches? It might be the sensor isn't updating enough.

[RPCHome]

Ah okay makes sense about the completed dispatched.
Just plugged in this morning to get a cheeky top-up change before 1100 and found that the beta 10 isn't showing any planned dispatches. But the other integration is. in the below screen show the beta is on the left and the other is on the right.

@RPCHome Do you have any errors in your logs that might indicate why you don't have any planned dispatches? It might be the sensor isn't updating enough.

Nope nothing but it might be baucase I had not long reloaded beta 10. As the completed ones are showing up now.

[wralb]

More data.

1. This morning I wrote a HASS automation to send me a message any time binary_sensor.octopus_energy_intelligent_dispatching or one of its attributes changes (and send me an easy to read summary of the data).
2. Car was plugged in at the time and the dispatch attributes were blank
3. I Unplugged the car and drove off at 11:25
4. At 12.00 I got my first message with the following data:

Dispatch state: off
Planned:
 6 Jun 22:00 -> 7 Jun 05:30 (source = smart-charge)

Completed:
 6 Jun 10:00 -> 6 Jun 10:30 (source = None)
 6 Jun 09:30 -> 6 Jun 10:00 (source = None)
 6 Jun 09:00 -> 6 Jun 09:30 (source = None)
 6 Jun 08:30 -> 6 Jun 09:00 (source = None)
 6 Jun 08:00 -> 6 Jun 08:30 (source = None)
 6 Jun 07:30 -> 6 Jun 08:00 (source = None)
 6 Jun 07:00 -> 6 Jun 07:30 (source = None)
 6 Jun 06:30 -> 6 Jun 07:00 (source = None)
 6 Jun 06:00 -> 6 Jun 06:30 (source = None)
 6 Jun 05:30 -> 6 Jun 06:00 (source = None)
 6 Jun 05:00 -> 6 Jun 05:30 (source = None)
 6 Jun 04:30 -> 6 Jun 05:00 (source = None)
 6 Jun 04:00 -> 6 Jun 04:30 (source = None)
 6 Jun 03:30 -> 6 Jun 04:00 (source = None)
 6 Jun 03:00 -> 6 Jun 03:30 (source = None)
 6 Jun 02:30 -> 6 Jun 03:00 (source = None)
 6 Jun 02:00 -> 6 Jun 02:30 (source = None)
 6 Jun 01:30 -> 6 Jun 02:00 (source = None)
 6 Jun 01:00 -> 6 Jun 01:30 (source = None)
 6 Jun 00:30 -> 6 Jun 01:00 (source = None)
 6 Jun 00:00 -> 6 Jun 00:30 (source = None)
 5 Jun 23:30 -> 6 Jun 00:00 (source = None)

5. Since then I've been getting updated attributes every half hour where, each time, the oldest of the completed_dispaches disappears so that now I got the following at 16:30:

Dispatch state: off
Planned:
 6 Jun 22:00 -> 7 Jun 05:30 (source = smart-charge)

Completed:
 6 Jun 10:00 -> 6 Jun 10:30 (source = None)
 6 Jun 09:30 -> 6 Jun 10:00 (source = None)
 6 Jun 09:00 -> 6 Jun 09:30 (source = None)
 6 Jun 08:30 -> 6 Jun 09:00 (source = None)
 6 Jun 08:00 -> 6 Jun 08:30 (source = None)
 6 Jun 07:30 -> 6 Jun 08:00 (source = None)
 6 Jun 07:00 -> 6 Jun 07:30 (source = None)
 6 Jun 06:30 -> 6 Jun 07:00 (source = None)
 6 Jun 06:00 -> 6 Jun 06:30 (source = None)
 6 Jun 05:30 -> 6 Jun 06:00 (source = None)
 6 Jun 05:00 -> 6 Jun 05:30 (source = None)
 6 Jun 04:30 -> 6 Jun 05:00 (source = None)
 6 Jun 04:00 -> 6 Jun 04:30 (source = None)
 6 Jun 03:30 -> 6 Jun 04:00 (source = None)

Once I get my bill on the 18th, I'll be able to see whether the planned_dispatch matches the off-peak tariff.

[BottlecapDave]

Thanks for the update. I'll try and work out why the sensor is not turning on/off. It also looks like I'll need to record a local record of the completed dispatches in order to properly calculate the previous cost.

[RPCHome]

Update on `completed dispatches'

Here 3 screenshots showing my completed dispatches' for today and how they are slowly dropping off the list. It looking like they stay on for approx 13/14hours (obvs times are in GMT but I'm screenshotting in BST.

[BottlecapDave]

I've released a new beta https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/intelligent-beta-11 which stores the completed dispatches locally beyond the reported 12 hours. It should store a maximum of 3 days worth of data.

[RPCHome]

Beta 11 is working as you have planned. Completed dispatches is still reporting from the 10th of June for me

[wralb]

My latest beta feedback...

1. Just after @BottlecapDave announced beta11 I went in to HACS which prompted me explicity to upgrade to beta 11 from beta 10; I did this and then checked the version of OctopusEnergy in my HASS - it was production version 7.4.2, so I had to then do a manual upgrade to beta11. I guess this matches the experience of @ColinRobbins and HACS is not handling beta upgrades as it claims.
2. I currently have 2 planned dispatches (overnight 13th/14th June), but my most recent completed dispatch is now over two days ago. 07.0 to 07.30 on the 11th. Maybe this is correct??? I previously had completed dispatches whenever my car was plugged in.
3. In the last week I have two periods when the Intellient Dispatches binary sensor turned on (10.30 to 12.00 on 7th and 9.30 to 12 on the 9th). I will check these against the PDF bill in a couple of weeks.

[BottlecapDave]

Thanks for the update.

When data is downloaded, the completed dispatches should merge the results between the newest data and the cached data and throw away anything that's more than 3 days old. Based on previous reports completed dispatches disappear quite quickly so the integration might have never had a chance to see them? Something to keep an eye on.

Binary sensor should turn on every time a planned or completed dispatch is active. This is built in a similar way to my other sensors, but I've had sproadic reports of sensors not turning on. Not sure if this is a HA thing. I'm also not sure if a planned dispatch drops off the list once it's active or once it's complete.

[wralb]

I may have found a pointer to why my planned_dispatches are no-longer updating. I'm getting the following error in the home-assistant.log file every half hour. Note that the four DEBUG lines at 15:00:37.511 come from four additional logging lines I added to the beta11 code in case they help.

2023-06-14 15:00:37.350 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.previous_consumption_and_rates] Finished fetching rates_1900001048577_22L4339954 data in 0.000 seconds (success: True)
2023-06-14 15:00:37.511 DEBUG (MainThread) [custom_components.octopus_energy.intelligent] dispatch["start"]: 2023-06-13T18:30:00+00:00
2023-06-14 15:00:37.511 DEBUG (MainThread) [custom_components.octopus_energy.intelligent] dispatch["end"]: 2023-06-13T19:00:00+00:00
2023-06-14 15:00:37.511 DEBUG (MainThread) [custom_components.octopus_energy.intelligent] rate["valid_from"]: 2023-06-14 04:30:00+00:00
2023-06-14 15:00:37.511 DEBUG (MainThread) [custom_components.octopus_energy.intelligent] rate["valid_to"]: 2023-06-14 05:00:00+00:00
2023-06-14 15:00:37.511 ERROR (MainThread) [custom_components.octopus_energy.coordinators.electricity_rates] Unexpected error fetching rates data: '<=' not supported between instances of 'str' and 'datetime.datetime'
Traceback (most recent call last):
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 283, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/homeassistant/lib/python3.11/site-packages/homeassistant/helpers/update_coordinator.py", line 242, in _async_update_data
    return await self.update_method()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/.homeassistant/custom_components/octopus_energy/coordinators/electricity_rates.py", line 60, in async_update_electricity_rates_data
    rates[key] = adjust_intelligent_rates(new_rates,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/.homeassistant/custom_components/octopus_energy/intelligent/__init__.py", line 36, in adjust_intelligent_rates
    if __get_dispatch(rate, planned_dispatches, "smart-charge") is not None or __get_dispatch(rate, completed_dispatches, None) is not None:
                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/homeassistant/.homeassistant/custom_components/octopus_energy/intelligent/__init__.py", line 22, in __get_dispatch
    if (expected_source is None or dispatch["source"] == expected_source) and dispatch["start"] <= rate["valid_from"] and dispatch["end"] >= rate["valid_to"]:
                                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'str' and 'datetime.datetime'
2023-06-14 15:00:37.514 DEBUG (MainThread) [custom_components.octopus_energy.coordinators.electricity_rates] Finished fetching rates data in 0.220 seconds (success: False)

For info, here is my diagnostic data (note that the manifest below claims version 7.4.2, but I confirmed by inspection that I have the changes introduced in beta11).

  "home_assistant": {
    "installation_type": "Home Assistant Core",
    "version": "2023.6.1",
    "dev": false,
    "hassio": false,
    "virtualenv": true,
    "python_version": "3.11.4",
    "docker": false,
    "arch": "x86_64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.0-73-generic",
    "run_as_root": false
  },
  "custom_components": {
    "ics_calendar": {
      "version": "3.1.7",
      "requirements": [
        "ics>=0.7.2",
        "recurring_ical_events>=1.0.2b0",
        "icalendar>=4.1.0"
      ]
    },
    "places": {
      "version": "2",
      "requirements": []
    },
    "holidays": {
      "version": "1.9.6",
      "requirements": [
        "python-dateutil>=2.8.2",
        "holidays>=0.14.2"
      ]
    },
    "octopus_energy": {
      "version": "7.4.2",
      "requirements": []
    },
    "yahoofinance": {
      "version": "1.2.2",
      "requirements": []
    },
    "pirateweather": {
      "version": "1.0.2",
      "requirements": [
        "python-forecastio==1.4.0"
      ]
    },
    "hacs": {
      "version": "1.32.1",
      "requirements": [
        "aiogithubapi>=22.10.1"
      ]
    },
    "virgintivo": {
      "version": "0.1.26",
      "requirements": []
    },
    "feedparser": {
      "version": "0.1.7",
      "requirements": [
        "feedparser==6.0.8"
      ]
    },
    "upnp_availability": {
      "version": "0.0.3",
      "requirements": [
        "async_upnp_client>=0.33"
      ]
    },
    "nextcloudtalk": {
      "version": "2021.12",
      "requirements": [
        "requests"
      ]
    },
    "grocy": {
      "version": "v4.10.0",
      "requirements": [
        "pygrocy==1.5.0"
      ]
    }
  },
  "integration_manifest": {
    "domain": "octopus_energy",
    "name": "Octopus Energy",
    "codeowners": [
      "@bottlecapdave"
    ],
    "config_flow": true,
    "dependencies": [
      "repairs",
      "recorder"
    ],
    "documentation": "https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/",
    "homekit": {},
    "iot_class": "cloud_polling",
    "issue_tracker": "https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/issues",
    "ssdp": [],
    "version": "7.4.2",
    "zeroconf": [],
    "is_built_in": false
  },
  "data": {
    "electricity_meter_points": [
      {
        "mpan": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "is_export": false,
            "is_smart_meter": true,
            "device_id": "**REDACTED**",
            "manufacturer": "1063 - Landis and Gyr",
            "model": "00050204",
            "firmware": "38040404"
          }
        ],
        "agreements": [
          {
            "valid_from": "2023-04-09T23:00:00+00:00",
            "valid_to": null,
            "tariff_code": "E-1R-INTELLI-VAR-22-10-14-J",
            "product_code": "INTELLI-VAR-22-10-14"
          }
        ]
      }
    ],
    "gas_meter_points": [
      {
        "mprn": "**REDACTED**",
        "meters": [
          {
            "serial_number": "**REDACTED**",
            "consumption_units": "m\u00b3",
            "is_smart_meter": true,
            "device_id": "**REDACTED**",
            "manufacturer": "1063 - Landis and Gyr",
            "model": "47720101",
            "firmware": "03033254"
          }
        ],
        "agreements": [
          {
            "valid_from": "2023-03-31T23:00:00+00:00",
            "valid_to": null,
            "tariff_code": "G-1R-VAR-22-11-01-J",
            "product_code": "VAR-22-11-01"
          }
        ]
      }
    ]
  }
}

[BottlecapDave]

Ok thanks for that. I know what the issue is, so I'll look at fixing it when I get a spare moment.

[BottlecapDave]

New release (https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/intelligent-beta-12) should fix your issues @wralb

[wralb]

Thanks Dave, Beta 12 installed at 6.15 today; no errors since, and I have more recent completed_dispatches.

One thought, which may or may-not be helpful since I don't understand all the use cases here... I have a lot of half-hour completed dispatches that are contiguous, would it make sense to merge them in to a smaller number of entries spanning a longer time? This would be consistent with the planned_dispatches which show longer periods:

Planned:
 10 Jun 22:00 -> 10 Jun 22:30 (source = smart-charge)
 10 Jun 23:00 -> 11 Jun 05:30 (source = smart-charge)

It would save memory, reduce the size of the attributes and make the completed_dispatches attribute more readable.

[BottlecapDave]

I'm glad it's now working :) If others agree, I think this fix might be ready?

The 30 minute chunks is how OE report them. I assume because you could stop charging during a planned dispatch and therefore cut it short. I'm trying to do minimal transformations on the data due to the nature of me being unable to test is thoroughly.

I plan to try and work on the other features over the few weeks, but I'll create a new ticket to keep the discussions focused aand tag everyone in it.

[RPCHome]

I've upgraded to Beta 12 and it is working well for me.

The only issue I have is the automation
{% if state_attr('binary_sensor.octopus_energy_intelligent_dispatching', 'planned_dispatches') | length > 0 %} {% for dispatch in state_attr('binary_sensor.octopus_energy_intelligent_dispatching', 'planned_dispatches') %} ⏰{{ (dispatch.start | as_datetime | as_local).strftime('%H:%M') }} to {{ (dispatch.end | as_datetime | as_local).strftime('%H:%M') }} {%- endfor %} {% else %} ⏰ None {%- endif %}

It returns this error message TypeError: float() argument must be a string or a real number, not 'datetime.datetime'

It stopped working a few betas ago but unsure why.

For reference, I use this automation from the Octopus Intelligent integration and it working fine.

{% if state_attr('binary_sensor.octopus_intelligent_slot', 'plannedDispatches') | length > 0 %} {% for dispatch in state_attr('binary_sensor.octopus_intelligent_slot', 'plannedDispatches') %} ⏰{{ (dispatch.startDtUtc | as_datetime | as_local).strftime('%H:%M') }} to {{ (dispatch.endDtUtc | as_datetime | as_local).strftime('%H:%M') }} {%- endfor %} {% else %} ⏰ None {%- endif %}

More than happy to continue testing the beta as when you other on the other features.

[BottlecapDave]

@RPCHome As mentioned before your issue is your trying to cast a datetime into a datetime which doesnt work. You template needs to be

{% if state_attr('binary_sensor.octopus_energy_intelligent_dispatching', 'planned_dispatches') | length > 0 %} {% for dispatch in state_attr('binary_sensor.octopus_energy_intelligent_dispatching', 'planned_dispatches') %} ⏰{{ (dispatch.start | as_local).strftime('%H:%M') }} to {{ (dispatch.end | as_local).strftime('%H:%M') }} {%- endfor %} {% else %} ⏰ None {%- endif %}

[RPCHome]

Thanks @BottlecapDave I must have missed that before.

[BottlecapDave]

@RPCHome @wralb @megakid @tomfielder @ColinRobbins Thanks for the assistance in trying to fix this issue. I've released a new beta which adds various sensors exposed in @megakid's integration. Please join the discusssion at #282 and raise any related issues there.

I'll keep this issue open until I've released the fix for the original issue.

[BottlecapDave]

This fix is now available within https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy/releases/tag/v7.4.4 so closing this issue. Thanks for all of the assistance :)