Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Authentication failed #38

Closed
WayneManion opened this issue Mar 5, 2024 · 67 comments
Closed

Authentication failed #38

WayneManion opened this issue Mar 5, 2024 · 67 comments

Comments

@WayneManion
Copy link

I have been using this Sensi integration for a few weeks. Today I had a notification that the authorization for the Sensi integration was expired. I went to the Integrations page in Home Assistant and clicked "Re-configure" for the Sensi integeration.

A modal box with no text and just a "sleepy eyeball" appeared. I entered my Sensi account password. Authentication failed. I went to the mobile app on my phone and changed the password. I was able to log into the mobile app with the new password. I used the new password in the Sensi integration and again, authentication failed.

@715Steve
Copy link

715Steve commented Mar 5, 2024

This started for me as well. Removed and reinstalled but cannot get past the credentials screen.
Authentication failed.

@iprak
Copy link
Owner

iprak commented Mar 5, 2024

Most probably Sensi backend changed the verification, this has happened before #28.

@bperry11
Copy link

bperry11 commented Mar 5, 2024

+1 for having this issue pop up today.

@NathanT15
Copy link

Same here, I got the auth request, tried my password, and then removed and reinstalled but cannot get past the credentials screen.

@derousse
Copy link

derousse commented Mar 5, 2024

Same for me...

@sjbarkey
Copy link

sjbarkey commented Mar 5, 2024

Issue started today for me as well.

@KE2EAC
Copy link

KE2EAC commented Mar 5, 2024

Same here.....

@Menz01
Copy link

Menz01 commented Mar 6, 2024

Same issue for me too

@laurenslo
Copy link

laurenslo commented Mar 6, 2024

same.

@stleusc
Copy link

stleusc commented Mar 6, 2024

There is a new header required:

rct: xxxxxxx

It changes with every login based on MITMPROXY and Android app.
The code to create the value seems to be at least partially in com.emerson.sensi.util.recaptcha

So far I don't know yet how exactly it is calculated....

@stleusc
Copy link

stleusc commented Mar 6, 2024

I had filtered to traffic with Sensi only, haha.
There are calls to:

https://www.recaptcha.net/recaptcha/api3/mri
https://www.recaptcha.net/recaptcha/api3/mrr
https://www.recaptcha.net/recaptcha/api3/mlg

It seems like the first one gets static data (mostly) and the response to mrr contains the header we need for rct.
Need some more digging but should be fairly straight forward.

Maybe someone finds docu for that API?

SEEMS to be Google Reccaptcha V3 (maybe)

@jsiemonski
Copy link

Add me to the pile 😁

@squid1127
Copy link

I am having this issue as well

@mondomondoman
Copy link

I am also having this issue.

@stephennutt
Copy link

+1

@jcamm3
Copy link

jcamm3 commented Mar 7, 2024

+1 same issue

@stleusc
Copy link

stleusc commented Mar 7, 2024

I really don't think any more +1 comments are really helping the issue. This is going to happen for EVERYONE!
I am pretty sure the repo owner got that point and maybe we should limit comments to things that are helpful to fixing the issue.

@iprak
Copy link
Owner

iprak commented Mar 7, 2024

The way I understand reCAPTCHA is the app talks to captcha server using an app specific key. The server returns a token based on bot identification and tells the backend server. This token is then passed to the backend server during authentication. I am suspecting the rtc header is this token.

I have been looking at the decompiled app and have not been able to find any key so far. I will keep digging.

https://cloud.google.com/recaptcha-enterprise/docs/instrument-android-apps
https://www.geeksforgeeks.org/how-to-integrate-google-recaptcha-in-android/

@stleusc
Copy link

stleusc commented Mar 7, 2024

Key needed:

6LdS6SYpAAAAAFCti9uo4Yg47fSIGygzTGl_6lEV

This is from Android app...

@stleusc
Copy link

stleusc commented Mar 7, 2024

I can create a fake POST to mri endpoint. It contains a few values needed for mrr endpoint.
mrr takes a total of 7 strings protobuf encoded.
String 1 and 2 are in response from mri, String 3 is the key and String 4 is "login"
String 5-7, so far no clue what they are or where to get them....
Once we have those we can post to mrr and the response will contain the value needed for the auth header that we are missing now.

@troyboy27
Copy link

+1 - but don't understand how to fix it.

@calimansi
Copy link

The integration through homekit still works, but doesn't have as many options.

@omarquis
Copy link

+1 how can I fix the issue?

@iprak
Copy link
Owner

iprak commented Mar 11, 2024

@troyboy27 @omarquis This cannot be addressed without code changes. The authentication at Sensi end has changed.

@macwriter
Copy link

macwriter commented Mar 11, 2024

I am also investigating the {"grant_type":"refresh_token","refresh_token": "some_random_string"} , if the refresh_token is long lived and how to acquire it. More than likely it will rely on the aforementioned recaptcha.

Will report when I know more...



data = {
        "client_id": client_id,
        "client_secret": client_secret,
        "grant_type": "refresh_token",
        "refresh_token": refresh_token
    }

headers = {
        "accept": "*/*",
        "accept-language": "en-US,en;q=0.9",
        "content-type": "application/x-www-form-urlencoded; charset=utf-8"
    }

@stleusc
Copy link

stleusc commented Mar 11, 2024

More than likely it will rely on the aforementioned recaptcha.

You get that as part of the auth process. You generally get the token and a refresh token. Once the token expires you exchange the refresh token for a new pair. That step requires the recaptcha as well.

BUT, Google has libraries that take care of that on Android, not sure if Sensi had to implement sth for iOS or if there is a library or even a different process...
Since it does not matter for us we might get lucky with iOS and the way it's done there.
Can you log traffice between the app and sensi/google? Using sth like MITM proxy or so.

That is how I found out about the recaptcha process...

@macwriter
Copy link

Ok thanks,

I got the "refresh_token" by logging the traffic via MITM proxy and have been using that in the sensi service for HA as a workaround.

I did see the recaptcha as well., but I could not follow the output. I will have another try at it and report back later. Thanks for the tips!

@stleusc
Copy link

stleusc commented Mar 11, 2024

I'll take some of what I said back... once you have the refresh tokoen you do not need recaptcha anymore!

@stleusc
Copy link

stleusc commented Mar 11, 2024

have been using that in the sensi service for HA as a workaround.

I tried that and it is failing... how did you tell HA to use it? I tried to enter it into the box that asks for auth....
but not working.

@macwriter
Copy link

@Pheelix
Copy link

Pheelix commented Mar 12, 2024

https://manager.sensicomfort.com/

It's the same site someone would use if they paid for there subscription to access there thermostats from there site.

username/password is the same on both app and website.

@stleusc
Copy link

stleusc commented Mar 12, 2024 via email

@Pheelix
Copy link

Pheelix commented Mar 12, 2024

nope, not paying, so no functionality. But i figured it was worth a shot to see if it would help at all. Tho atm I am using the Homekit method in HA to control my Sensi Touch. But would be nice to see this up and running again.

@stleusc
Copy link

stleusc commented Mar 12, 2024 via email

@Pheelix
Copy link

Pheelix commented Mar 12, 2024

site says first month is free, cancel any time. so I guess it could be tested without having to pay for it. Just remember to cancel before the first month ends.

@stleusc
Copy link

stleusc commented Mar 12, 2024

Actually, I don't think that is even needed. Just looked at the traffic....
The response looks VERY similar to the app and contains the required tokens.

@macwriter you already have modified sources... I think it would be super easy for you to check if the response tokens work.
Simply log into https://manager.sensicomfort.com/ and look at the traffic to grab the token from the oauth call.
I have a good feeling about this!

@Pheelix you might have solved the mystery for us...

@iprak The website also uses recaptcha but it is fully browser based. I ASSUME that python can emulate a webbrowser in a way that you load the login page and let it do it's thing to take care of the recaptcha?

@stleusc
Copy link

stleusc commented Mar 12, 2024

I can confirm, the tokens work!

Changed auth.py around line 67

    refresh_token = "copied from browser response"
    client_id = "fleet" # instead of android
    client_secret = "JLFjJmketRhj>M9uoDhusYKyi?zUyNqhGB)H2XiwLEF#KcGKrRD2JZsDQ7ufNven" # different secret

    post_data = {
        #"username": config.username,
        #"password": config.password,
        #"client_id": CLIENT_ID,
        #"client_secret": CLIENT_SECRET,
        #"grant_type": "password",
        "client_id": client_id,
        "client_secret": client_secret,
        "refresh_token": refresh_token,
        "grant_type": "refresh_token",
    }

Error in HA goes away after a restart...

I guess if @iprak can simulate the browser login then we are back in action!

I guess worst case solution would be to have the user login to the website and copy the refresh token and have that as an external config value that we have to paste. I guess other add-ons do the same where the user needs to create an account with company X and provide some keys manually.

I guess my manual fix will stop working as soon as my current token expires since I hard-coded the refresh token and it only works once. But if we can feed the initial refresh_token and than the addon keeps managing it like in the past it should work long term.

@Pheelix amazing catch!

@stleusc
Copy link

stleusc commented Mar 12, 2024

@iprak I just looked at the code and noticed that it seems like you never really use the refresh_token. Is that right?
If I understood it right you are sending user/pwd again when the original token expires using "grant_type": "password".
That is not the intention of OAuth... Ideally your addon would never know my user/pwd and only save the 2 tokens.
Once the access token expires you ask for a fresh one using "grant_type": "refresh_token" and the saved refresh_token value.
That will give you a new pair and you repeat.
You only ever need user/pwd ONCE at the very beginning.... Then never again.
With this changed flow the user could provide a single valid refresh_token and no user/pwd and you don't have to worry about recaptcha since that seems to be needed only for the user/pwd exchange!

@iprak
Copy link
Owner

iprak commented Mar 12, 2024

Python cannot emulate reCaptcha, some library attempt to do that but they either implement v2 or are paid. reCaptcha involves a rotating sort of token which is generated by reCaptcha library.

Sensi uses the latest google reCaptcha. In web, this
e.g. https://www.google.com/recaptcha/enterprise.js?render=6LebYk0pAAAAAIUlMrqCdXOq83pY2O4u-CxuTh28
returns an executing block which returns the token.

The ideal way to address would be to change the initial authentication itself. This is how authentication is implemented for 3rd party verification e.g. https://sensiapi.io/authorize

In my experimentation, extracting token from web and passing to Python did not work. There might be additional headers/cookies.

Yes, you are right about refresh_token. I just login in again; I could not confirm when refreshing would be adequate.

@stleusc
Copy link

stleusc commented Mar 12, 2024 via email

@stleusc
Copy link

stleusc commented Mar 12, 2024

I see two possible solutions here:

A: Change the setup flow to ask the user for a refresh_token. Explain that the user can get it by logging into https://manager.sensicomfort.com/ and then copying the value using the browser developer tools. This step is needed ONCE to setup. After that the refresh_token can be exchanged for an auth_token and a new refresh_token. Whenever sensi complains about expired credentials simply exchange again. This is the simplest impementation and would not require much change I think?!

B: Have python emulate a webwroser and login to https://manager.sensicomfort.com/ using username/pwd and then grab the refresh_token from the web traffic. Would be easy in and Android WebView, not sure if Python can do that.
Once you have that token the flow should be like in A. Don't store user/pwd but only refresh_token and renew when needed.

This hardcoded snippet DOES work until the token expires (hard coded refresh_token works only once.
But by replacing the hard-coded token with one initially provided by the user and then updated on every refresh this really works.
I see all my thermostats in HA as we speak!

auth.py around line 68ish

refresh_token = "copied from browser response"
client_id = "fleet" # instead of android
client_secret = "JLFjJmketRhj>M9uoDhusYKyi?zUyNqhGB)H2XiwLEF#KcGKrRD2JZsDQ7ufNven" # different secret

post_data = {
    #"username": config.username,
    #"password": config.password,
    #"client_id": CLIENT_ID,
    #"client_secret": CLIENT_SECRET,
    #"grant_type": "password",
    "client_id": client_id,
    "client_secret": client_secret,
    "refresh_token": refresh_token,
    "grant_type": "refresh_token",
}

@kevinCefalu
Copy link

I was just about to suggest using selenium and chrome webdriver for the browser interactions. I know the workload can be containerized, as well. But honestly, I feel like @stleusc's first suggested solution is far straighter forward.

@akseidel
Copy link

The tokens are JWT. Paste them at jwt.io to see their contents. The expiration date for my refresh_token is way out into 2034, but it does report as an Invalid Signature. Stleusc, how does yours report?

@stleusc
Copy link

stleusc commented Mar 12, 2024

@akseidel learned something new today about JWT :-)

Yes, the refresh_token is valid for 10 years. Which is the idea behind it. The access_token is short lived, the refresh_token is long lived. HOWEVER, you generally can only use it ONCE to convert to a new access_token. You have 10 years to do that, but can do it only one time.

I guess what I stated above depends on the implementation. In other words, if we get a refresh_token in the reponse, I would update to that one (also gives 10 more years). If there is none in the response, keep the old one.
Other sites that I worked with in the past that used OAuth allowed only one exchange...

From https://www.oauth.com/oauth2-servers/access-tokens/refreshing-access-tokens/

If everything checks out, the service can generate an access token and respond. The server may issue a new refresh token in the response, but if the response does not include a new refresh token, the client assumes the existing refresh token will still be valid.

Regading signature: Signature Verified

@stleusc
Copy link

stleusc commented Mar 12, 2024

The tokens are JWT. Paste them at jwt.io to see their contents. The expiration date for my refresh_token is way out into 2034, but it does report as an Invalid Signature. Stleusc, how does yours report?

The access token reports invalid sig, the refresh token valid sig.
I think the page needs to know the secret for checking the sig and we did not enter that!

@macwriter
Copy link

macwriter commented Mar 12, 2024

@macwriter you already have modified sources... I think it would be super easy for you to check if the response tokens work. Simply log into https://manager.sensicomfort.com/ and look at the traffic to grab the token from the oauth call. I have a good feeling about this!

Yes, works for me with the updated information (client_id, client_secret,refresh_token). Great job finding this!

def test(client_id,client_secret,refresh_token):
    params=()
    data = {
            "client_id": client_id,
            "client_secret": client_secret,
            "grant_type": "refresh_token",
            "refresh_token": refresh_token
        }

    headers = {
            "accept": "*/*",
            "accept-language": "en-US,en;q=0.9",
            "content-type": "application/x-www-form-urlencoded; charset=utf-8",
        }

    _response = requests.post("https://oauth.sensiapi.io/token", params=params, data=data, headers=headers)
    print(_response)
    print(_response.json().keys())
      
test(**options_from_Sensi_Manager)
<Response [200]>
dict_keys(['access_token', 'refresh_token', 'token_type', 'expires_in', 'password_reset', 'eula_accepted', 'offers', 'alerts', 'user_id'])

@troyboy27
Copy link

troyboy27 commented Mar 12, 2024 via email

@stleusc
Copy link

stleusc commented Mar 12, 2024

I think we have all we need to get this working again....
My option A from above should be pretty easy to implement and we already tested all the elements that are involved.
Now we just need to hope that @iprak finds the time to work on it.

@bperry11
Copy link

B: Have python emulate a webwroser and login to https://manager.sensicomfort.com/ using username/pwd and then grab the refresh_token from the web traffic.

Ironically, this could be foiled by recaptcha! lol
But maybe not.

@stleusc
Copy link

stleusc commented Mar 12, 2024

@bperry11 using something like an Android WebView would work since that is basically the component used to build browser apps. Of course Python would be different...
I personally prefer A ;-)

@iprak
Copy link
Owner

iprak commented Mar 14, 2024

Sorry for late response, this integration usually only gets my attention on weekends.

When I wrote this extension, my aim was to provide an easy to setup integration with the ultimate goal to have it be integrated in Core release.

While solution "A" makes it work, I do not think it is the right approach. It certainly puts the extension out of non-developers or folks just using mobile device.

I do not think browser emulation is possible in Python without using something like Selenium which makes it a quite heavy integration. This also might not work well not work in the constraints of HomeAssistant.

There are 3rd party subscription based libraries for solving reCaptcha but that again might nor work since reCaptcha is tied to the site.

Sensi does have external integration support e.g. Whoosh Air, SmartThings and electricity providers. I personally have Sensi integrated via the last 2 paths. My hope is that Sensi's external integration could be used to pass back the authentication information via webhook. But I don't know enough yet about the implementation. https://sensiapi.io/authorize however is the site for establishing external integration.

I understand that this issue is a headache for many. I do not think this integration is subtainable due to the additional restrictions imposed at Sensi end. I will try to get refresh_token based authentication out soon but I won't be making any more updates to this extension unless I can establish the authentication to be more standard.

@stleusc
Copy link

stleusc commented Mar 14, 2024

While 'A' is not ideal, there are certainly many other integrations that are more complicated. Eg https://www.home-assistant.io/integrations/smartthings/ or https://www.home-assistant.io/integrations/alexa.smart_home/
And I also don't think that there have been any changes in recent years that would have broken that approach.
I assume you could contact Sensi and see if they allow you to become a partner, but I have a feeling they would not.

That being said, thank you for your continued support. I hope you change your mind and keep supporting this great integration!

@iprak
Copy link
Owner

iprak commented Mar 14, 2024

Those two do have some setup but those are using established public facing workflows. Launching DevTools is at a different level.

One can link Sensi from SmartThings app which establishes authentication via https://sensiapi.io/authorize and this gives me hope that such a possibility might be there. The initial call does pass in a different token and a callback, if that is not locked down then the callback could be HomeAssistant webhook (same as SmartThings) to receive the token.

@bperry11
Copy link

It was Homebridge, but I had to do very similar process to integrate Google Nest. Login, open devtool, grab the cookie / or token, and paste into homebridge. At least for the time being, it wouldn't be too big of a deal IMO. Either way, thanks for your work on this!

@dziban303
Copy link

I tried to do what's shown in this comment but it still wasn't working. I'm guessing the refresh_token I was using was wrong. What exactly does it look like?

@bperry11
Copy link

bperry11 commented Mar 14, 2024

I tried to do what's shown in this comment but it still wasn't working. I'm guessing the refresh_token I was using was wrong. What exactly does it look like?

Mine's a 333 characters of upper and lowercase letters, digits, and symbols. Did you also get the client_secret? You'll need to put that in the auth.py file as well.

Find and change the following in the asnyc def login() function around line 48. (I commented out the old and put the new underneath:

post_data = {
        # "username": config.username,
        # "password": config.password,
        # "client_id": CLIENT_ID,
        # "client_secret": CLIENT_SECRET,
        # "grant_type": "password",
        "client_id": "fleet",
        "client_secret": "<INSERT_YOUR_ NEW_CLIENT_SECRET>",
        "grant_type": "refresh_token",
        "refresh_token": "<INSERT_YOUR_RESFRESH_TOKEN>",
    }

headers = {
        # "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
        # "x-platform":"android",
        # "accept": "*/*",
        "accept": "*/*",
        "accept-language": "en-US,en;q=0.9",
        "content-type": "application/x-www-form-urlencoded; charset=utf-8",
   }

@dziban303
Copy link

Thanks @bperry11, I was copying the wrong refresh_token but I got it figured out now. Cheers mate

@iprak iprak closed this as completed Mar 19, 2024
@iprak
Copy link
Owner

iprak commented Mar 19, 2024

The token can expire. 1.3.2 has better support for renewing the token when data load indicates expiration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests