Melcloud integration- Reports wrong energy value for the total consumption

[maxime1992]

The problem

For about a week now, since the issue where MelCloud started to impose a rate limiting (see #109728) there's been other issues.

First of all, the daily consumption is gone. I've searched a bit and it looks like it was a deliberate move to get less calls for the polling. CF 5521a39. I don't consider this a big issue, it's easy enough to recreate with a helper using the total consumption.

Now, what's more of a problem at the moment, is that since this all started (~ last week), the total consumption appears to be wrong. It's a lot lower than what it should be. To give an idea, it's roughly the same weather since last week and you can look at the break down of my consumption in the following picture. The orange bar is the consumption of the heatpump, notice how it's super low for a week now:

Here are some value comparison between cloud data from the MelCloud app and my HA data, in kWh:

Date	HA	MelCloud	Ratio
14/02/24	2.0	14.4	7.20
13/02/24	2.4	14.9	6.20
12/02/24	1.9	11.5	6.05
11/02/24	1.8	13.8	7.66

Looking at the ratio, it look like it's always between ~6 and ~8 times less on HA than MelCloud.

As the polling period has been extended, if there isn't a poll made right before midnight, I'm guessing we could have some of the HA value slightly different for the day compared to MelCloud, which might explain that we're not getting the same ratio. Or it's totally unrelated, unsure.

While the main feature to manage the heat pump temperature is working again (which is great, thanks for reacting so quickly!), it's still quite problematic in my case as I was hoping to get good quality stats after buying the heat pump right when winter started to be able to make long term stats difference pre heat pump VS now (not using HA, using Grafana but it gets the data from HA still).

I have looked for similar issues and found this one #109864 but I think it was wondering about the daily consumption and as stated at the beginning of this issue, it was a deliberate move and it can be fixed easily so this issue is really about fixing the total consumption.

I don't get to create loads of issue on HA, so I'll take this one to say a massive thank you for the work you're doing and I'm quite impressed with the pace at which HA ships new feature. Top 3 of my favourite open source projects for sure.

What version of Home Assistant Core has the issue?

2024.2.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

MelCloud

Link to integration documentation on our website

https://www.home-assistant.io/integrations/melcloud

Diagnostics information

Not sure it's needed in that case but let me know otherwise

Example YAML snippet

No response

Anything in the logs that might be useful for us?

I'm afraid not

Additional information

No response

[home-assistant]

melcloud documentation
melcloud source

[madd0x2000]

Mine reports a too low value. It feels like it should be 10 times higher. They are consuming 1000-2000 W per hour, according to other measurments I have on them, and still reports using like 100-200w. Really strange.

[membersound]

We would first have to find out: does the API return wrong values, or is HA processing and aggregating the values in a wrong manner? I mean, it could be possible that HA is working correctly (means: displaying simply the value it receives), but MelCloud is giving wrong values after their update.

[maxime1992]

We would first have to find out: does the API return wrong values, or is HA processing and aggregating the values in a wrong manner? I mean, it could be possible that HA is working correctly (means: displaying simply the value it receives), but MelCloud is giving wrong values after their update.

Is there any debug we can use to see that ? I'm happy to take a look

[maxime1992]

Interesting...

Source.

I do wonder if anything changed upstream

[maxime1992]

Ok I dug into this a little bit...

I had a look at the repo HA is using: https://github.com/vilppuvuorinen/pymelcloud

I then created 3 files:

requirements.txt (as per the one defined in HA)

aiohttp
pymelcloud==2.5.9

main.py

import aiohttp
import asyncio
import pymelcloud

async def main():

    async with aiohttp.ClientSession() as session:
        # call the login method with the session
        token = await pymelcloud.login("EMAIL_HERE", "PASSWORD_HERE", session=session)

        # lookup the device
        devices = await pymelcloud.get_devices(token, session=session)
        print(devices)
        device = devices[pymelcloud.DEVICE_TYPE_ATA][0]
        print(device)

        # perform logic on the device
        await device.update()

        print(device.total_energy_consumed)
        await session.close()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Dockerfile

FROM python:3.10

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD [ "python", "./main.py" ]

And then ran

docker build -t test-melcloud-python . && docker run --rm test-melcloud-python:latest

The call to the API gives me... 🥁 The wrong answer 😞 ...

So it looks like HA is doing the right thing but most likely the API changed upstream and this isn't computing the correct value. As per my previous message, I guess this changed.

The repo has been archived 10 days ago... So we're unlikely to get a fix unless there's a fork made by someone I guess. I don't know what HA does in those circumstances but maybe instead of importing the lib it could have the code defined in HA core directly? Or make an HA fork so that it can be owned by the community? Because right now there's no way to make a fix upstream, can't even create an issue to ask if anyone has done a fork because the repo is readonly...

[membersound]

The call to the API gives me... 🥁 The wrong answer 😞 ...
Does this mean the meldcloud API offers the correct value, but it gets transfered wrong through the pymeldcloud integration?

[maxime1992]

Lol, you took this picture from a post I made 3 days ago, and just like me initially I think you didn't notice that the date was 2022 😄

[maxime1992]

Does this mean the meldcloud API offers the correct value, but it gets transfered wrong through the pymeldcloud integration?

I haven't dug this deep... But my guess is that melcloud api returns a value, and the conversion ran by the python script is wrong. I'd guess that maybe they changed upstream the unit of what's being sent... But I haven't checked.

[maxime1992]

I took another look today.

When I open up the MelCloud app, by default it takes a year worth of data, here 2023/2024:

This shows the correct total consumption, as I've had my heat pump for less than a year.

Now, looking into what PyMelCloud used to do here:

from_str = (datetime.today() - timedelta(days=2)).strftime("%Y-%m-%d")
to_str = (datetime.today() + timedelta(days=2)).strftime("%Y-%m-%d")

It was taking 2 days prior and 2 days later compared to now.

Which I've mimicked from the MelCloud web app and I can see the payload I'd expect:

And this gives me the following report:

So the TotalHeatingConsumed is relative to the date passed as param, it's not the one since forever.

Testing out a code change from the master branch (which still has energy report compared to the latest 2.5x branch...). I have changed the start date to be the 1st of January 1970:

And I correctly get the total consumption of the device:

@vilppuvuorinen if you see this, thanks a lot for the work you've provided on the repo! It looks like it's now archived so there's no way to open an issue there but also it looks like the 2.5x branch has been active lately and that you have disabled the energy report.

Based on the little investigation I've made above, would there be any way for you to revert your commit and simply change the from date to be from_str = "1970-01-01" so that it'd give the correct total consumption again?

Thanks!

[vilppuvuorinen]

This is the endpoint explicitly mentioned by those temporary account suspension messages. The calls to it were removed to avoid prompting harsher actions from the service provider.

Please don't build any REST sensors based on this endpoint and especially with a full history of the universe time range like this.

[maxime1992]

This is the endpoint explicitly mentioned by those temporary account suspension messages. The calls to it were removed to avoid prompting harsher actions from the service provider

I think this was done way more frequently than it is now. Isn't that enough?

Please don't build any REST sensors based on this endpoint and especially with a full history of the universe time range like this

So what do you recommend to get the total consumption reported properly again to HA?