Netatmo OAUTH2 change to auth_code grant (from password)

[kiddigital]

Small PR addressing issue #5766

Not the most user friendly (option 1 as discussed in the issue) but not to difficult to set up. Should be done only once as long as domoticz is able to keep refreshing the tokens.

step 1:

Setup an App under MyApps in the Netatmo developers portal

• The Redirect_URI has to contain http://localhost/netatmo

Once configured correctly, the Client_ID and Client_Secret will be generated and shown.

step 2:

In the domoticz hardware setup:

• Create/modify the Netatmo hardware
• the username should contain <client_id>:<client_secret>
• while the password should contain the retrieved <code>

To retrieve the code (a.k.a. authorisation_code), use the browser and type the following URL. Make sure to fill in the CLIENTID with your clientid!

https://api.netatmo.com/oauth2/authorize?client_id=<YOURCLIENTID>&redirect_uri=http%3A%2F%2Flocalhost%2Fnetatmo&state=teststate&scope=read_station%20read_thermostat%20write_thermostat%20read_homecoach%20read_smokedetector%20read_presence%20read_camera

This URL brings you to the My Netatmo page where you have to identify yourself and are asked for permission to grant access to certain devices and types (based on the scope provided).

After confirmation, your browser will try to redirect you to a page it cannot find so you will get an error. But the URL of that page is visible in the browser and looks like http://localhost/netatmo?state=teststate&code=<SOMECODE>.

You need to pass this SOMECODE as the password in the domoticz hardware screen.

NOTE:

• At the moment, the scopes are fixed to read_station read_thermostat write_thermostat read_homecoach read_smokedetector read_presence read_camera
• The <SOMECODE> you retrieved is only valid for a few minutes, so if you take too long and domoticz says (check the logs) it cannot login, retrieve a new code and be a little quicker :)
• Proper documentation (on the Wiki) has to be created/updated

[gizmocuz]

Thank you!

[jcbraem]

Ready for me !
Many thanks.

[gizmocuz]

@jcbraem , just double check, ready=working?

[Fill1234]

I've added the scope for the co2 detector and i could successfully added the hardware.
But no sensor (smokedetector/co2 detector visible.

[kiddigital]

I've added the scope for the co2 detector and i could successfully added the hardware. But no sensor (smokedetector/co2 detector visible.

Which scope is that? Currently the Netatmo module uses a fixed lists of scopes (see above). Maybe we need to add an additional scope? And not sure if the module contains the logic to handle the smokedetector/co2 module (yet).

[Fill1234]

Scope: read_carbonmonoxidedetector
Not sure if it works with only your scopes but i was asked for authorisation and the co2 sensor was added in the list.

[Onshuis]

I wanted to let you know, that everything works like it did before.
My only scope is the (basic) weather station, 2 smart indoor units and one outdoor unit.
So thank you very much for changing the login procedure. :-)

[kiddigital]

I've added the scope for the co2 detector and i could successfully added the hardware. But no sensor (smokedetector/co2 detector visible.

Which scope is that? Currently the Netatmo module uses a fixed lists of scopes (see above). Maybe we need to add an additional scope? And not sure if the module contains the logic to handle the smokedetector/co2 module (yet).

Scope: read_carbonmonoxidedetector Not sure if it works with only your scopes but i was asked for authorisation and the co2 sensor was added in the list.

hmm... according to the docs that scope does not exist. Any clue if/where to find better documentation? Maybe about the API calls for the co2 sensor?

[Fill1234]

Should i try again without the co2 detector?

[JurgenLB]

By this message you can't generate a key with 'Redirect URL' in use

I'm still in the process of updating and testing my setup
But I think this will work.

[kiddigital]

By this message you can't generate a key with 'Redirect URL' in use

I'm still in the process of updating and testing my setup

But I think this will work.

You don't have to generate a token yourself. The client_id and client_secret is enough.

[jcbraem]

@gizmocuz

@jcbraem , just double check, ready=working?

Yes, I have all values retrieve in Domoticz for Temp, CO2, humidity, wind, rain, ...

[jcbraem]

Hello,
I had a breakdown of Internet during the night. Result : need to take a new token manually.
Do you know the validity time of token ?
The option 2 descripted below with and generation of first token is planned ?
Thank-you

[kiddigital]

No idea how long the token is valid (could be an hour or less which is not uncommon) but the refresh token is usually valid for a few hours or day(s).

What do you have in your log? How long was the internet connection down? Did the module retry several times?

Option 2 is not planned as it is not possible to implement that in the current setups.

[JurgenLB]

maybe we need to discus this on a different thread but it seems that Netatmo.h only uses NETYPE_WEATHER_STATION = 0, NETYPE_HOMECOACH, NETYPE_ENERGY

So I aspect the Homecoach and the thermostat valves will work, but this is not.
The other scopes are not in use from my point of view
What do we need for this development of the hardware?

@Fill1234
for the CO2 sensor according the documentation it uses;
Home + security
read_smokedetector for Smart Smoke Detector
Product - - - - - - - - - - - - - - - - - - - - - - Product Type - - - - - Endpoint
Smart Smoke Detector - - - - - - - - - - - - - NSD - - - - - - - - - - /homestatus
Smart Carbon Monoxide Alarm - - - - - - - - NCO - - - - - - - - - - /homestatus

[jcbraem]

What do you have in your log? How long was the internet connection down? Did the module retry several times?

Sorry I have don't check the log and no idea of downtime.

[chaudron]

My internet was down for a couple of hours, and now even when regenerating the code it returns authentication errors. I'll try it again tomorrow, but has anyone seen this also?

Not sure what solved it, but I did a couple of deactivate/activate, generating some codes. it started to work again.

[Phil353556]

@kiddigital
FYI,
1/ reading the page https://dev.netatmo.com/apidocumentation/oauth#authorization-code
"expires_in":10800,
so three hours
2/ I opened a ticket in june on this topic and the answer was (I translated from french using deepl.com)
"If you refresh your access_token every 3 hours with refresh_token, you'll never be asked for the "code" value again."
hope this can help

[kiddigital]

@Phil353556 , thx. And that mechanism is used by domoticz. Each time the access token expires (after 3 hours), it will use the refreshtoken to request new tokens and receive a new access token AND new refreshtoken.

The refreshtoken is valid for a longer time, but will expire as well after some time.

[Niekvanderpas]

Hello all,

Sorry but I am a bit lost. As Fill1234 previously stated when you enter a redirect URI I get a warning and cannot generate a Token.
If I leave the redirect URI empty and save the App Technical Parameters I can use the button Generate Token.
But entering either the Access token or the Refresh Token in the Password: field in Domoticz has no results.
The Error log gives: No access granted, check credentials...

It would be verry helpful is one of you can help me a step in the right direction.

[Phil353556]

Hi,
forget token generator, ie don't use it.
The most important, based on my experience is to use the complete URL
https://api.netatmo.com/oauth2/authorize?client_id=<CLIENT_ID>&redirect_uri=http%3A%2F%2Flocalhost%2Fnetatmo&state=teststate&scope=read_station%20read_thermostat%20write_thermostat%20read_homecoach%20read_smokedetector%20read_presence%20read_camera
My mistake was to, for example, remove read_homecoach
once the full URL is used with your client ID, the webpage display an error, because http://localhost/netatmo is not found, this is not a problem, the most important is to catch (copy/paste) the code=xxxxxxxxxxxxxxx displayed in the URL field
and to use it as password in domoticz
hope this helps

[Niekvanderpas]

@Phil353556 , you saved my day.
It works Thanks a lot!

[Phil353556]

You're welcome 👍
experience shared: benefit for all
Tell me then if you notice the same behavior as me.
I have netatmo devices like smart door sensors, rain gauge, external device, all of them have battery
What I notice: the level( %) charge for all the devices is only got at the start, no more then

[Niekvanderpas]

My Netatmo devices are not battery operated so I have no experience and my two Sonoff battery powerd devices are just 2 weeks old and do give 100% battery level.

[Phil353556]

ok
thank you for the feedback

[JurgenLB]

New breaking change from Netatmo ! 04/12/2023
#philippelt/netatmo-api-python#73

[gizmocuz]

@JurgenLB , we are already using the refresh token, do you experience an issue?

[dgilbert2]

I have just updated to the new Netatmo login procedure after my setup finally had trouble logging in after many years without issues. All working again thanks to this work, thank you.

The only issue I've spotted is that if I reboot Domoticz or restart the Netatmo worker, the access "code" will be out of date so I need to rerun the URL to generate a fresh one and update it in Domoticz. I assume that if I experience an internet issue, the same access code issue will also occur. At least for now I know how to get things working again.

[Phil353556]

Am I alone with this behavior?
"What I noticed: the level( %) charge for all the devices is only got at the start, no more then"

[JurgenLB]

@JurgenLB , we are already using the refresh token, do you experience an issue?

currently rebuilding my system. But will look in to it.
Kind regards

[Phil353556]

Hi, I am testing the last version of netatmo plugin with domoticz beta [2023.2 (build 15667)]
Great job, it works with the latest authentification implemented by netatmo. Thank you!
However the behavior is that percentage level, for netatmo devices with battery, is tested only one time at startup. (in /tmp/domoticz.txt reference is Netatmo: General/Percentage)
Is it normal?
How to enhanced that? one time per day could be good enough.
Thank you

[gizmocuz]

what do you mean with "percentage level' what device type is this? What is the default label?

[Phil353556]

Hi,
"percentage level" : the current level of battery for each device
devices impacted : Smart Rain Gauge, smart door and window sensor, Smart outdoor Air Quality Monitor
Sorry in french :
"Pluviomètre" = Smart Rain Gauge
"Entrée" or "garage" or "porte fenêtre" = each is smart door and window sensor
"Module extérieur" = Smart outdoor Air Quality Monitor

[JurgenLB]

I'm running build 2023.2 '15727'
I'm only getting 16 devices;
2023-12-15 22:54:38.603 Status: Netatmo: Worker started... 2023-12-15 22:54:44.502 Netatmo: Temp + Humidity (Buiten) 2023-12-15 22:54:44.505 Netatmo: Temp + Humidity (Boven ) 2023-12-15 22:54:44.507 Netatmo: Air Quality (Boven ) 2023-12-15 22:54:44.509 Netatmo: Temp + Humidity (Badkamer ) 2023-12-15 22:54:44.511 Netatmo: Air Quality (Badkamer ) 2023-12-15 22:54:44.514 Netatmo: Rain (Regenmeter) 2023-12-15 22:54:44.516 Netatmo: Wind (Windmeter) 2023-12-15 22:54:44.518 Netatmo: Temp + Humidity + Baro (Weather Station) 2023-12-15 22:54:44.520 Netatmo: Air Quality (Weather Station) 2023-12-15 22:54:44.522 Netatmo: General/Sound Level (Weather Station) 2023-12-15 22:54:44.764 Netatmo: Temp (Thermostaat Tuinhuis) 2023-12-15 22:54:44.766 Netatmo: Setpoint (Thermostaat Tuinhuis - SetPoint ) 2023-12-15 22:54:44.768 Netatmo: Light/Switch (Thermostaat Tuinhuis - Mode) 2023-12-15 22:54:44.771 Netatmo: Temp (Woonkamer) 2023-12-15 22:54:44.773 Netatmo: Setpoint (Woonkamer - SetPoint ) 2023-12-15 22:54:44.776 Netatmo: Light/Switch (Woonkamer - Mode) 2023-12-15 22:56:08.362 Status: New sensors allowed for 5 minutes... 2023-12-15 22:59:45.171 Netatmo: Temp + Humidity (Buiten) 2023-12-15 22:59:45.172 Netatmo: Temp + Humidity (Boven ) 2023-12-15 22:59:45.173 Netatmo: Air Quality (Boven ) 2023-12-15 22:59:45.174 Netatmo: Temp + Humidity (Badkamer ) 2023-12-15 22:59:45.175 Netatmo: Air Quality (Badkamer ) 2023-12-15 22:59:45.176 Netatmo: Rain (Regenmeter) 2023-12-15 22:59:45.177 Netatmo: Wind (Windmeter) 2023-12-15 22:59:45.178 Netatmo: Temp + Humidity + Baro (Weather Station) 2023-12-15 22:59:45.179 Netatmo: Air Quality (Weather Station) 2023-12-15 22:59:45.179 Netatmo: General/Sound Level (Weather Station) 2023-12-15 22:59:45.434 Netatmo: Temp (Thermostaat Tuinhuis) 2023-12-15 22:59:45.434 Netatmo: Setpoint (Thermostaat Tuinhuis - SetPoint ) 2023-12-15 22:59:45.435 Netatmo: Light/Switch (Thermostaat Tuinhuis - Mode) 2023-12-15 22:59:45.437 Netatmo: Temp (Woonkamer) 2023-12-15 22:59:45.437 Netatmo: Setpoint (Woonkamer - SetPoint ) 2023-12-15 22:59:45.438 Netatmo: Light/Switch (Woonkamer - Mode)

So no smoke sensors, no thermostat Valves and no Homecoach or door-sensors / camera's.
The camera's are the bridge for the door-sensors.

The Homecoach can not work together with a Weatherstation in Domoticz because of the IFNOT-statement invoid CNetatmo::GetMeterDetails() on line 807.
if (!ParseStationData(sResult, false)) { // User doesn't have a weather station, so we check if it has // a homecoach device (only once) if (m_bFirstTimeWeatherData)

I believe the battery is only running on startup and is not called again.

[gizmocuz]

Okey, but do you see a battery level for each normal device? Because the battery level is taken from the API result
Maybe we should therefor ignore these battery/signal sensors and not add them to the system

[JurgenLB]

each device have a battery indicator.

Do you want to ignore the battery levels or the battery operated sensors ?
This last one is not preferable.

The API (July 2023) changed also the handling of the devices and there are more endpoints.

For so far I can see Netatmo.h only uses NETYPE_WEATHER_STATION = 0, NETYPE_HOMECOACH, NETYPE_ENERGY on this moment.

I don't know if you want to support all devices from Netatmo or just a selection

`

 case NETYPE_MEASURE:
      sstr << "https://api.netatmo.net/api/getmeasure";
      break;
 case NETYPE_WEATHER_STATION:
      sstr << "https://api.netatmo.net/api/getstationsdata";
      break;
 case NETYPE_HOMECOACH:
      sstr << "https://api.netatmo.net/api/gethomecoachsdata";
      break;
 case NETYPE_THERMOSTAT:
      sstr << "https://api.netatmo.net/api/getthermostatsdata";
      break;
 case NETYPE_HOME:
      sstr << "https://api.netatmo.net/api/homedata";
      break;
 case NETYPE_ENERGY:         // Not only Energy with new API so better to change to NETYPE_DATA
      sstr << "https://api.netatmo.net/api/homesdata";
      break;
 case NETYPE_STATUS:
      sstr << "https://api.netatmo.net/api/homestatus";
      break;
 case NETYPE_CAMERAS:
      sstr << "https://api.netatmo.net/api/getcamerapicture";
      break;
 case NETYPE_EVENTS:
      sstr << "https://api.netatmo.net/api/geteventsuntil";
      break;

`

[Phil353556]

If it is possible to have the battery level not only at startup but one per day, it would be useful.
It is then possible to set an alert when the level reaches 10%, for example.
Thank you

[gizmocuz]

What I was asking, the battery level reported in the devices tab should be the lastest battery level being reported from the API

The percentage sensor is only being updated in the function ParseHomeStatus, the function ParseStationData is only taking the battery level, using it downstream, but is not creating a percentage sensor
I don't know the person who hacked this battery percentage sensor into the code, but I would not have done this.
This is not done anywhere ... should not be needed, the battery percentage you can see in the devices tab/eventsystem

Edit: https://www.domoticz.com/wiki/Battery_level_check

So, does the battery level in the devices tab reflect the correct state?

[JurgenLB]

I have checked the battery levels;
When I hover over the battery symbol on the devices tab I get the following numbers;
Buiten 65%
Boven 89%
Badkamer 91%
Regen 68%
Wind 100%
Tuinhuis 0%
Woonkamer 0%

When I run lNetatmo.py the percentages are different ;
Buiten 77 %
Boven 93 %
Badkamer 66 %
Regen 92 %
Wind 67 %
Tuinhuis 66 %
Woonkamer 67 %

There can be a few minutes between the two but the difference is to large.

so the battery level has not the correct state.