Netatmo authentication breaking change again !

[philippelt]

We had some quiet time these last weeks. Netatmo has decided to restart with breaking changes.
Credentials created through their user interface are no longer long lived, the refresh token is now refreshed also. No more long lived credentials.

So there will be definitively NO WAY to keep credentials data READ ONLY.

All existing applications will stop to work immediately.
Congratulations to Netatmo

[mnin]

Can confirm this, the refresh token get's refreshed now and needs to be stored dynamically if it get's refreshed.

May this line needs to be extended: https://github.com/philippelt/netatmo-api-python/blob/7d3006830b9870472b13cfd86121382e5d704ca0/lnetatmo.py#L264C6-L264C6 and store the token in a file and reused it next run.

[mnin]

I solved it for now by adding a refresh token file called netatmo_refresh_token, reading initially the REFRESH_TOKEN from this file with:

file = open("/netatmo_refresh_token", "r")
REFRESH_TOKEN = file.readline().splitlines()[0]
file.close()

And pass it (with CLIENT_ID and CLIENT_SECRET) to the ClientAuth call:

authorization = lnetatmo.ClientAuth(
        clientId=CLIENT_ID,
        clientSecret=CLIENT_SECRET,
        refreshToken=REFRESH_TOKEN
        )

And adding some new lines to the renew_token method:

    def renew_token(self):
        postParams = {
                "grant_type" : "refresh_token",
                "refresh_token" : self.refreshToken,
                "client_id" : self._clientId,
                "client_secret" : self._clientSecret
                }
        resp = postRequest("authentication", _AUTH_REQ, postParams)
        if self.refreshToken != resp['refresh_token']:
            print("New refresh token:", resp['refresh_token'])
            file = open("/netatmo_refresh_token", "w")
            file.write(resp['refresh_token'])
            file.close()
        self._accessToken = resp['access_token']
        self.refreshToken = resp['refresh_token']
        self.expiration = int(resp['expire_in'] + time.time())

to save the new token into the netatmo_refresh_token file. And use it next time again.

[mnin]

Thanks for the fix in 4.0.0 @philippelt !

[philippelt]

I pushed a quick fix e11d226
I choose the path of rewriting .netatmo-credentials as this file has the most chances to already exist.
The problem will be for people using small appliances like raspberry who hard coded their credentials in the library source or people using containers and embedding credentials in images (changes would not be persistent)

[rvk01]

Apparently this change is not implemented wide-spread (or globally) by Netatmo. I have my own implementation where I already save the resulting json (which contains the new access token but also any "new" refresh token) but looking back in the log files of those older json I see that the refresh token still hasn't changed for me (while the access token needs to be refreshed 8 times a day). Note. This refresh token is generated during an API call. Not via the dashboard.

And for another implementation (where I do use lnetatmy.py) the refresh token was invalidated. I recreated one via the dashboard and I will see if that one expires. Maybe only the dashboard created ones expire. But saving a potentially changed returned refresh token is best practice anyway.

[Boulder08]

Can you please explain in simple steps how to make things work again? I just updated to v4.0.0 and I've never used any credential file before.
My Python code just dumps: code=400, reason=, body=b'{"error":"invalid_grant"}'

EDIT: hmm.. looks like PyCharm didn't update the package even if it installed successfully??

[mnin]

@Boulder08 think these are the possible steps:

1. Create a file called '.netatmo.credentials` with this content (replace with your tokens and yoy need a new refresh token from https://dev.netatmo.com/):

{
  "CLIENT_ID" : "xxx",
  "CLIENT_SECRET" : "xxx",
  "REFRESH_TOKEN" : "xxx"
}

This file needs to be available in the home directory from the perspective the python script is running. Like /home/user/.netatmo.credentials.

2. Update and install lnetatmo version 4.0.0

• Edit requirements.txt file
• Install new version with pip3 install -r requirements.txt

3. Maybe remove all credentials from your script and just call ClientAuth() without arguments, it will fetch the credentials automatically and update the .netatmo.credentials file if the API returned a new refresh token.

[mnin]

@Boulder08 the version 4.0.0 is available on https://pypi.org/project/lnetatmo/ and it worked for me to install the new version via pip.

[Boulder08]

@mnin , thanks, I got it working now. pip3 found only the v3.3.0 initially, but I uninstalled it and ran install again, which then found v4.0.0. I guess there might be a mix up between the PyCharm installed packages and pip3.

[poet-of-the-fall]

Thanks for the quick fix! I already was wondering this afternoon, what they messed up now agin...
Updating to 4.0.0 helped, but for my project (raspi + display), my code looks like something like this:

while True:
    # Initiate Netatmo client
    try:
        authorization = lnetatmo.ClientAuth()
        weatherData = lnetatmo.WeatherStationData(authorization)
    except:
        logging.warning('Fetching data failed! Waiting 20 Minutes.')
        time.sleep(1200)
        continue
    ...
    time.sleep(600)

In this case I always get the exception ERROR: code=400, reason=, body=b'{"error":"invalid_grant"}' after the second loop cycle. But stopping the script and running it again works. Is it a bug or is my code shitty?

[philippelt]

Hello @poet-of-the-fall , just avoid to include lnetatmo.ClientAuth() inside your loop. The refresh_token is here to request a new access token when it will be required due to expiration

[philippelt]

Netatmo is playing with our nerves.

Their Web service is only capable of producing a SINGLE refresh token even if you change your scope, the new generated token will REPLACE the previous one.

The consequence is that, if you have multiple homes in a single account, and are using, for example, two raspberry to monitor data in each home, the first that will get a new refresh token and save it on its own file system will kill the refresh token of the other machine.

Is there ANYONE at Netatmo using their products ? Do they review their code to avoid regression (the answer is NO)

I can't do ANYTHING to work around this Netatmo nonsense. Until they understand that home AUTOMATION is mainly about BATCH ACCESS, life will continue to be hard with Netatmo.

[rvk01]

Netatmo is playing with our nerves.

Their Web service is only capable of producing a SINGLE refresh token even if you change your scope, the new generated token will REPLACE the previous one.

Aaaaargh. Yes. One refresh token per account/device per client-id/app registration. Even when getting the tokens via web OAuth. I could use the same account with a different client-id though.

[rvk01]

@philippelt I still have a question about your saving of the tokens. Why are you not also saving the access token?? The access token is valid for about 3 hours. I have a script which runs this code every 5 minutes. As it is now, the access token is refreshed EVERY time, every 5 minutes (because it doesn't take the old, still valid, access token).

If done correctly it should take the stored access token and use that. If it gets an error "2" back (access token expired) it can get a new fresh one with the refresh token. That how I do it in another custom script of mine in PHP (the returned json from the call gets stored which contain both the new access token and new refresh token, and client-id and secret isn't even needed anymore !!!) Otherwise you would be hammering the netatmo servers with refreshing the access token EACH time.

So the script should get its client_id and secret from the ClientAuth call, which it only uses ONE time to get the refresh_token/access_token. It can save the returned json and with that refresh/access token it can live on forever (until netatmo screws up again :) ).

[hermannx5]

Thanks for the quick fix! I already was wondering this afternoon, what they messed up now agin... Updating to 4.0.0 helped, but for my project (raspi + display), my code looks like something like this:

while True:
    # Initiate Netatmo client
    try:
        authorization = lnetatmo.ClientAuth()
        weatherData = lnetatmo.WeatherStationData(authorization)
    except:
        logging.warning('Fetching data failed! Waiting 20 Minutes.')
        time.sleep(1200)
        continue
    ...
    time.sleep(600)

In this case I always get the exception ERROR: code=400, reason=, body=b'{"error":"invalid_grant"}' after the second loop cycle. But stopping the script and running it again works. Is it a bug or is my code shitty?

Hi, I have the same problem...Did you manage to fix this ?

[poet-of-the-fall]

Hi, I have the same problem...Did you manage to fix this ?

Yes, as mentioned right below just take the authorization out of the loop:

authorization = lnetatmo.ClientAuth()

while True:
    # Initiate Netatmo client
    try:
        weatherData = lnetatmo.WeatherStationData(authorization)
    except:
        logging.warning('Fetching data failed! Waiting 20 Minutes.')
        time.sleep(1200)
        continue
    ...
    time.sleep(600)

[hermannx5]

Hi, I have the same problem...Did you manage to fix this ?

Yes, as mentioned right below just take the authorization out of the loop:

authorization = lnetatmo.ClientAuth()

while True:
    # Initiate Netatmo client
    try:
        weatherData = lnetatmo.WeatherStationData(authorization)
    except:
        logging.warning('Fetching data failed! Waiting 20 Minutes.')
        time.sleep(1200)
        continue
    ...
    time.sleep(600)

Ah, thanks...didn't see this, sorry :)

[ultima-originem]

My code also stopped working:

2023-12-05 12:40:37 ERROR  MainThread  code=400, reason=, body=b'{"error":"invalid_grant"}'
--- Logging error ---
Traceback (most recent call last):
  File "/home/pi/scripts/rain.py", line 106, in main
    rain_measurements = fetch_netatmo()
  File "/home/pi/scripts/rain.py", line 69, in fetch_netatmo
    weather_data = lnetatmo.WeatherStationData(authorization)
  File "/usr/local/lib/python3.10/dist-packages/lnetatmo.py", line 413, in __init__
    self.getAuthToken = authData.accessToken
  File "/usr/local/lib/python3.10/dist-packages/lnetatmo.py", line 240, in accessToken
    if self.expiration < time.time() : self.renew_token()
  File "/usr/local/lib/python3.10/dist-packages/lnetatmo.py", line 251, in renew_token
    if self.refreshToken != resp['refresh_token']:
TypeError: 'NoneType' object is not subscriptable

I've installed 4.0.0 and I'm using .netatmo.credentials. The code runs via crontab every 4 minutes and authorization = lnetatmo.ClientAuth() is only called once (no looping).

Is there anything I need to change that I'm unaware of? Any help will be appreciated as I'm stuck.

[mnin]

@ultima-originem did you also recreate the refresh token on https://dev.netatmo.com?

[hermannx5]

@ultima-originem did you also recreate the refresh token on https://dev.netatmo.com?

Yes, I did. It did not work before I recreated the refreh token on their site

[egnappahz]

Hello, first of all thank you for all your work and very fast response. I could not figure out yesterday the correct flow of authentication. Infact, I still really can't, but it seems your code is managing the new refreshtoken correctly.

My only question is, does this guarantee the refreshtoken stays valid, even if we stop polling? You see, I have had 1 situation already where i got the famous grant error (after happely polling for some hours every 5 minutes), and I had to get a new refreshtoken with scope on the dev plesks . Did something go wrong on my end? Is there a way to be 100% sure the refreshtoken stays valid?

Is there maybe a way to get a new, valid scope refreshtoken via the CLI when the refresh goes wrong, so I can make a failsafe?

Dont know if it makes any sense what I am saying. I am reaaaaaaaaaaaally struggeling with the expected flow here.

Thank you again for everything.

[ultima-originem]

@mnin : thanks for pointing this out; all seems fine now.

[mnin]

@ultima-originem so you forgot to recreate the refresh token?

[ultima-originem]

Yes, I must have missed that somehow.

[egnappahz]

OK I think I fixed on my end, but I would still like to know if there is an automated way to get a new refreshtoken with scope whenever the refreshchain is lost/broken/expired, for whatever reason . You know, as a failsave -- I like my things as reliable and automated as possible. This refreshtoken chaining is clunky and thus prone to error, client AND serverside.

[rvk01]

OK I think I fixed on my end, but I would still like to know if there is an automated way to get a new refreshtoken with scope whenever the refreshchain is lost/broken/expired, for whatever reason . You know, as a failsave -- I like my things as reliable and automated as possible. This refreshtoken chaining is clunky and thus prone to error, client AND serverside.

Once the refresh token is invalid (for whatever reason) you can't fix it without getting a new one. A new one can be generated at the dev dashboard or can be obtained by the user OAuth flow (i.e. user permission screen via https://api.netatmo.com/oauth2/authorize).

[egnappahz]

@rvk01 yeah not exactly my idea of true automation.

I am researching a way to headlessly automate the 4step scope authorisation (Authorization code grant type) without user intervention when the chain is broken, any help and tips are appreciated. The user prompt to authorize the application might need some browser spoofing (puppeteer maybe?), im looking into it.

[philippelt]

Thanks to all for mutual help in searching for solutions.

As I said, it was a "quick fix" but I need to do some rework to cleanup the code.

@poet-of-the-fall your code is better with clientAuth out of the loop but it definitively should have been able to work without change. It is the residue of the old static authentication method, implemented at module level, that is no longer useful. It's one of the cleanup task I have to implement for a better code.

@rvk01 you are absolutely right, considering that now I am obliged to rewrite the credentials file, I could add the access_token and rewrite it as well. It will improve the dialog with the netatmo server for people with a lot of calls in the access_token live timeframe.

@egnappahz Ensure that you have only one program/task using the credentials or that the credentials file is effectively shared among all tasks. A single task using the refresh_token without storing a new one, may be with due to an outdated version of the library, is enough to kill the stored refresh_token.

The other improvement I am going to add is to pass the credentials file name as a clientAuth parameter. Now that it must be a writable file, I prefer to let user choose the name and location. Of course, by default, the current name and location will continue to be used : I hate breaking changes.

I just need to find some quiet time to release a 4.1.0 !

[egnappahz]

@philippelt indeed it was exactly that as I found out, thank you for the info. But it still makes me concerned about other potential interuptions though...

[HaikoKurt]

@philippelt please do not read the credential file at the loading time of the module, but only when the ClientAuth object is generated. Then it is possible to create the file programmatically before the module is used.

I want to use the module in a Docker container and not have to specify the credentials at build time, but only at deploy time

Thank you, for your great work! Why makes it netatmo so complicated?

[HaikoKurt]

Right now, my code looks like this:

import os
import json

CRED_PATH = os.path.expanduser("~/.netatmo.credentials") # module internals should be better hidden?
if not os.path.isfile(CRED_PATH) :  # at restart container, file will exist
    credentials = {
        "CLIENT_ID" : env.client_id, # get credentials from container environment 
        "CLIENT_SECRET" : env.client_secret,
        "REFRESH_TOKEN" : env.refresh_token
    }
    with open(CRED_PATH, "w") as f:
        json.dump(credentials, f)

import lnetatmo

authorization = lnetatmo.ClientAuth()
devList = lnetatmo.WeatherStationData(authorization)
print(json.dumps(authorization.accessToken))
print(json.dumps(devList.lastData()))

[philippelt]

Thanks for the suggestion. This is an old decision to allow for static authentication credentials that no longer males sense now with short lived refresh tokens.

It's on my to do list of code cleanup ! If only I could find some time to do all planned changes. Probably next week...

[pulsebreaker]

For the people who are running a python script on a Synology NAS through the Task Scheduler (like me), I finally got it working by moving the .netatmo.credentials file to the standard /home folder on the NAS. In my case the Netatmo script is somewhere else on the NAS but that doesn't seem to matter.

using

authorization = lnetatmo.ClientAuth()

try:
    ws = lnetatmo.WeatherStationData(authorization)
    etc.

in the script works like a charm.

It's probably worth checking the file permissions.

[philippelt]

Just pushed a new version with authentication refactoring.

• credential file is no longer read at module import but only when ClientAuth initializer is called
• Credential file name is now accepted as a parameter of ClientAuth initializer. Of course, the previous file name and location is still the default so no changes are required to what works but it will allow to manage the credential file location for specific use cases (behind web server for example :-)

I do not publish immediately to pypi to continue testing of this release. Thanks to report any regression you found.

[edennede]

Hi,
I'm using the ThermostatData class to obtain the thermostat setpoint as well as the room temperature.
I get a result in JSON example: 'measured': {'time': 1704981541, 'temperature': 22.5, 'setpoint_temp': 21}}]
but do you know how to get a live value? The data updates every how many minutes?
Thanks for your feedback

[philippelt]

Hello @edennede,

sorry for the (long) delay,

As far as I know, data is refreshed every 10 minutes or so. Requesting it more frequently is useless as you will be returned with the same previous data.

Not a big deal, it's hard to imagine a significant room temperature change in less that 10 minutes 🙂

[rvk01]

@philippelt Wouldn't that depend on the endpoint. The normal sensor stations do have an interval of 10 minutes (although I thought it could be changed). But a thermostat would need to have live real-time data. Otherwise it couldn't do its job.

As you can notice from the table below, data can be obtained from 2 endpoints: /gestationsdata 3 & /homestatus 6. The difference between these 2 endpoints is that /getstationsdata 3 retrieves data from the cloud where data are updated every 10min whereas /homestatus 6 retrieves data from the devices and data are near realtime data (<10s).

https://community.home-assistant.io/t/netatmo-poll-refresh-time-is-currently-10-minutes-ideally-should-be-5-minutes/44524/19

So the endpoint /homestatus would be near realtime.

[philippelt]

@rvk01 Correct, it certainly depends of the endpoint which itself depends of the device upload.

As far as I know, there is no direct access to a device from the Netatmo API, nor locally, nor remotely except for the camera live or recorded streams. But I do not own any thermostat device thus I can't check what precisely happen with them.

I can easily imagine that an alarm detection would trigger an immediate upload of data for action triggering but for slow moving data, such as room temperature, it's likely only a time poll interval that is used.

It's not easy to find precise informations about the data collection by Netatmo...