Skip to content
This repository was archived by the owner on Jul 4, 2024. It is now read-only.

Conversation

drysart
Copy link
Contributor

@drysart drysart commented Oct 10, 2023

See: home-assistant/core#101763

This PR updates the call to the OAuth get token endpoint to be more in line with the OAuth specs, to fix an issue where MyQ is returning 401 errors.

Please review pretty closely, I don't do a lot of coding in Python. ;)

@Lash-L Lash-L changed the title Update OAuth get token call for correctness fix: update oauth get token call for correctness Oct 10, 2023
@Lash-L
Copy link
Member

Lash-L commented Oct 10, 2023

Seemingly good to me - we can ignore the commit lint warning as I can fix that during merge. Give me about ~10 minutes and I'll test it and merge

Copy link
Contributor

@justinlindh justinlindh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

Tested this most recent version in this PR. Connection works as expected and functionality is restored.

@schott12521
Copy link

For Commit message, maybe include some of @drysart's words?


  • Add an authorization header to specify the client ID (base64 encoded value for IOS_CGI_MYQ:)
  • Add an Accept header with /
  • Remove the unnecessary client_id, client_secret, and scope form fields since they're not supposed to be sent on the get token call in the first place
  • Added [0] to the line extracting the code form field to get the code as a string rather than as an array

@Lash-L Lash-L merged commit e8ab829 into Python-MyQ:master Oct 10, 2023
@drysart drysart deleted the feature/20231010-GetTokenFix branch October 10, 2023 21:44
Lash-L added a commit that referenced this pull request Oct 10, 2023
* add an authorization header to specify the client id (base64 encoded value for ios_cgi_myq:)

* add an accept header with /

* remove the unnecessary client_id, client_secret, and scope form fields

* added [0] to the line extracting the code form field to get the code as a string

---------

Co-authored-by: Luke Lashley <conway220@gmail.com>
@Lash-L
Copy link
Member

Lash-L commented Oct 10, 2023

Thanks @drysart - if you want to make the PR to bump the version in Home Assistant too, feel free if you'd like it on your git history or anything. Otherwise I can do it in a little bit -stepping away from a computer for a bit.

@atombombzero
Copy link

I performed the edit on the api.py file and that seems to start generating 403 errors but I am still getting the 429 error after the 403 errors accumulate. I restarted HA completely but NO change.

2023-10-11 12:33:04.059 DEBUG (MainThread) [pymyq.api] Scanning login page for fields to return 2023-10-11 12:33:04.084 DEBUG (MainThread) [pymyq.api] Missing one of the valid keys email: True, password: True, submit: True, verfication: True 2023-10-11 12:33:04.085 DEBUG (MainThread) [pymyq.api] Performing login to MyQ 2023-10-11 12:33:04.085 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/Account/LoginWithEmail?returnUrl=/connect/authorize/callback?client_id%3DIOS_CGI_MYQ%26code_challenge%3D-cHWSYBdfbXPJUUB1CQSquUE8AvLgOIKkYrPeyykEqg%26code_challenge_method%3DS256%26redirect_uri%3Dcom.myqops%253A%252F%252Fios%26response_type%3Dcode%26scope%3DMyQ_Residential%2520offline_access and headers {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'} with connection pooling 2023-10-11 12:33:04.185 DEBUG (MainThread) [pymyq.request] Response: 2023-10-11 12:33:04.186 DEBUG (MainThread) [pymyq.request] Response Code: 302 2023-10-11 12:33:04.186 DEBUG (MainThread) [pymyq.request] Headers: ((b'Date', b'Wed, 11 Oct 2023 17:33:04 GMT'), (b'Content-Length', b'0'), (b'Connection', b'keep-alive'), (b'Cache-Control', b'no-cache'), (b'pragma', b'no-cache'), (b'expires', b'Thu, 01 Jan 1970 00:00:00 GMT'), (b'location', b'/connect/authorize/callback?client_id=IOS_CGI_MYQ&code_challenge=-cHWSYBdfbXPJUUB1CQSquUE8AvLgOIKkYrPeyykEqg&code_challenge_method=S256&redirect_uri=com.myqops%3A%2F%2Fios&response_type=code&scope=MyQ_Residential%20offline_access'), (b'set-cookie', b'idsrv.session=11EE455B8FF62206C4CF213DF28BBAB2; path=/; secure; samesite=none'), (b'set-cookie', b'idsrv=CfDJ8ON7dBOWWjJAuGokEvbbeTmW5hfXEZcfUrGveiKWW4Vz2TOClWJRor_owdKahzLIGW2-dhdPksMAt8fzfcSWzWGk9QYU7nzm3O4zBAjWWui7QE1dcxltCDtmmfWjBcgsYGr7pfD9rEvaekbIRcyPoqCcDCNAOSGrpvluYB5ra8zudDSvKzEwVjfcyIs4l6lLnUfCkwoO_5umu-t7EVVfGHTax4YlxbKz-Qd4rUlp1KvSUduYpaaYIDffqYL03yO3o4j6XcDNSwQ3SvGWQhieh30tRPnsE9FHZjmrogEiMSopa3_IQbGMhIyKuhYuMcmfRNElfwAmLBEbI9wrhNOAZJihnRQwye-g-BJjViR35WqVoCLOnDyloPUNbKLme6mj6rTmzcHyiw31R18S6vUcKoUuChWbFzR_qrzd3gF2PZgWDd1sxsET4jBV2QDktOQRhNECLmRyQg966GL_kN-KxFiGoiukEKsqDVuadSz66sdwRYzQh7ThbnGVfwqzZNffc-XsiRsExR8IHPLp87NOlu1xZz16kqGtuvQ5Zli5Meth0qDIbq6RdbBjdcC2u0tdLA; path=/; secure; samesite=none; httponly'), (b'request-context', b'appId=cid-v1:0e843709-4fd8-4c8a-a52a-7eb45ba4930b'), (b'x-xss-protection', b'1; mode=block'), (b'myq-correlationid', b'fa2fb2f8-4ad5-4234-b317-92b79e9f06ce'), (b'strict-transport-security', b'max-age=15724800; includeSubDomains'), (b'strict-transport-security', b'max-age=63072000'), (b'CF-Cache-Status', b'DYNAMIC'), (b'Server', b'cloudflare'), (b'CF-RAY', b'8148d8948b736303-ORD'), (b'alt-svc', b'h3=":443"; ma=86400')) 2023-10-11 12:33:04.186 DEBUG (MainThread) [pymyq.request] Body: 2023-10-11 12:33:04.187 DEBUG (MainThread) [pymyq.api] Calling redirect page 2023-10-11 12:33:04.187 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize/callback?client_id=IOS_CGI_MYQ&code_challenge=-cHWSYBdfbXPJUUB1CQSquUE8AvLgOIKkYrPeyykEqg&code_challenge_method=S256&redirect_uri=com.myqops%3A%2F%2Fios&response_type=code&scope=MyQ_Residential%20offline_access and headers {'Cookie': 'Set-Cookie: idsrv=CfDJ8ON7dBOWWjJAuGokEvbbeTmW5hfXEZcfUrGveiKWW4Vz2TOClWJRor_owdKahzLIGW2-dhdPksMAt8fzfcSWzWGk9QYU7nzm3O4zBAjWWui7QE1dcxltCDtmmfWjBcgsYGr7pfD9rEvaekbIRcyPoqCcDCNAOSGrpvluYB5ra8zudDSvKzEwVjfcyIs4l6lLnUfCkwoO_5umu-t7EVVfGHTax4YlxbKz-Qd4rUlp1KvSUduYpaaYIDffqYL03yO3o4j6XcDNSwQ3SvGWQhieh30tRPnsE9FHZjmrogEiMSopa3_IQbGMhIyKuhYuMcmfRNElfwAmLBEbI9wrhNOAZJihnRQwye-g-BJjViR35WqVoCLOnDyloPUNbKLme6mj6rTmzcHyiw31R18S6vUcKoUuChWbFzR_qrzd3gF2PZgWDd1sxsET4jBV2QDktOQRhNECLmRyQg966GL_kN-KxFiGoiukEKsqDVuadSz66sdwRYzQh7ThbnGVfwqzZNffc-XsiRsExR8IHPLp87NOlu1xZz16kqGtuvQ5Zli5Meth0qDIbq6RdbBjdcC2u0tdLA\r\nSet-Cookie: idsrv.session=11EE455B8FF62206C4CF213DF28BBAB2'} with connection pooling 2023-10-11 12:33:04.251 DEBUG (MainThread) [pymyq.request] Response: 2023-10-11 12:33:04.251 DEBUG (MainThread) [pymyq.request] Response Code: 302 2023-10-11 12:33:04.251 DEBUG (MainThread) [pymyq.request] Headers: ((b'Date', b'Wed, 11 Oct 2023 17:33:04 GMT'), (b'Content-Length', b'0'), (b'Connection', b'keep-alive'), (b'location', b'com.myqops://ios?code=7D57D588F8B513C324FA5BF3CFD895A2A6C6F2842D8AB923E1568A73D2D2E587&scope=MyQ_Residential%20offline_access&iss=https%3A%2F%2Fpartner-identity.myq-cloud.com'), (b'Cache-Control', b'no-store, no-cache, max-age=0'), (b'pragma', b'no-cache'), (b'expires', b'Thu, 01 Jan 1970 00:00:00 GMT'), (b'set-cookie', b'idsrv=CfDJ8ON7dBOWWjJAuGokEvbbeTkYLr9DIIJ1sKZigr_qG-r-xPttC3KLZWOKzITEiYuKrjSNa3RJxvC00tZXa9jhepUIIYwmst6oY8GDkJsglYufmxfIs7RnAENiRswsOcIL4YwcYbI3Yez0WixkYPl6kjcwQ2ZAEXMTvnADWC8ZjW95LuSxrnI88Z7uecdw-fVtq6zqoOLKB85eXtFKqP-ykbT2R8NU4E51N2nRxHG9QPRptgJloKKFC7iO2rA4U2hz98VzVqTY_mbns67W1mrTdgitF3ceFebZ2X8QEqhoCf86LIFfo3AN2NarrVv_Z0adH6XHwNOkH2OyntNMvuP32OLS0qYpZkd0tqZ3oz6xD5muW-nOdY9nTRsySWqlySOQ7fX4uDGCVZj0-ozm-RSruzjMRlkkox2lVxt4iMm-uMJPd9zFWNcvOne2VuRD9jnbY7ghAa1fUeyyEvyp_079xZAzV4aeLz3_1zzyXb0EOGDqAAmZKzYm5zGx1z-QC1kU9q8iyQuLCS8nLEjNHgm06QqMHLqpZVZURd5T91f_owU8hf-evQcGqzZeG2R6hrQqh3xXVMvNk1X7h-KndoMfarjhCsTsaAOecLAnNyim6vYt; path=/; secure; samesite=none; httponly'), (b'request-context', b'appId=cid-v1:0e843709-4fd8-4c8a-a52a-7eb45ba4930b'), (b'x-xss-protection', b'1; mode=block'), (b'myq-correlationid', b'21b3453b-0c29-4629-a058-dbf9cda21b45'), (b'strict-transport-security', b'max-age=15724800; includeSubDomains'), (b'strict-transport-security', b'max-age=63072000'), (b'CF-Cache-Status', b'DYNAMIC'), (b'Server', b'cloudflare'), (b'CF-RAY', b'8148d8952c246303-ORD'), (b'alt-svc', b'h3=":443"; ma=86400')) 2023-10-11 12:33:04.252 DEBUG (MainThread) [pymyq.request] Body: 2023-10-11 12:33:04.252 DEBUG (MainThread) [pymyq.api] Getting token 2023-10-11 12:33:04.252 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/token and headers {'Authorization': 'Basic SU9TX0NHSV9NWVE6', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'User-Agent': ''} with connection pooling 2023-10-11 12:33:04.287 DEBUG (MainThread) [pymyq.request] Attempt 1 request failed with exception : 403 - Forbidden 2023-10-11 12:33:04.288 DEBUG (MainThread) [pymyq.request] Received error status 403, bad request. Will refresh user agent. 2023-10-11 12:33:04.288 DEBUG (MainThread) [pymyq.request] Retrieving user agent from GitHub. 2023-10-11 12:33:04.377 DEBUG (MainThread) [pymyq.request] Retrieved user agent #RANDOM:5 from GitHub. 2023-10-11 12:33:04.377 DEBUG (MainThread) [pymyq.request] User agent set to randomized value: s1haq. 2023-10-11 12:33:04.377 DEBUG (MainThread) [pymyq.request] Request failed with "403 Forbidden" (attempt #1/5)"; trying again in 2 seconds 2023-10-11 12:33:06.379 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/token and headers {'Authorization': 'Basic SU9TX0NHSV9NWVE6', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'User-Agent': 's1haq'} with connection pooling 2023-10-11 12:33:06.405 DEBUG (MainThread) [pymyq.request] Attempt 2 request failed with exception : 403 - Forbidden 2023-10-11 12:33:06.406 DEBUG (MainThread) [pymyq.request] Request failed with "403 Forbidden" (attempt #2/5)"; trying again in 4 seconds 2023-10-11 12:33:10.407 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/token and headers {'Authorization': 'Basic SU9TX0NHSV9NWVE6', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'User-Agent': 's1haq'} with connection pooling 2023-10-11 12:33:10.441 DEBUG (MainThread) [pymyq.request] Attempt 3 request failed with exception : 403 - Forbidden 2023-10-11 12:33:10.442 DEBUG (MainThread) [pymyq.request] Request failed with "403 Forbidden" (attempt #3/5)"; trying again in 5 seconds 2023-10-11 12:33:15.443 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/token and headers {'Authorization': 'Basic SU9TX0NHSV9NWVE6', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'User-Agent': 's1haq'} with connection pooling 2023-10-11 12:33:15.465 DEBUG (MainThread) [pymyq.request] Attempt 4 request failed with exception : 403 - Forbidden 2023-10-11 12:33:15.466 DEBUG (MainThread) [pymyq.request] Request failed with "403 Forbidden" (attempt #4/5)"; trying again in 5 seconds 2023-10-11 12:33:20.467 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/token and headers {'Authorization': 'Basic SU9TX0NHSV9NWVE6', 'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'User-Agent': 's1haq'} with connection pooling 2023-10-11 12:33:20.492 DEBUG (MainThread) [pymyq.request] Attempt 5 request failed with exception : 403 - Forbidden 2023-10-11 12:33:20.493 DEBUG (MainThread) [pymyq.api] Error requesting data from https://partner-identity.myq-cloud.com/connect/token: 403 - Forbidden 2023-10-11 12:33:20.495 ERROR (MainThread) [pymyq.api] Authentication failed: Error requesting data from https://partner-identity.myq-cloud.com/connect/token: 403 - Forbidden 2023-10-11 12:33:20.496 WARNING (MainThread) [homeassistant.config_entries] Config entry 'myusername@gmail.com' for myq integration not ready yet: Error requesting data from https://partner-identity.myq-cloud.com/connect/token: 403 - Forbidden; Retrying in background 2023-10-11 12:33:25.928 DEBUG (MainThread) [pymyq.api] Performing initial authentication into MyQ 2023-10-11 12:33:25.928 DEBUG (MainThread) [pymyq.api] Scheduling token refresh, last refresh was None 2023-10-11 12:33:25.929 DEBUG (MainThread) [pymyq.api] Initiating OAuth authentication 2023-10-11 12:33:25.929 DEBUG (MainThread) [pymyq.api] Retrieving authentication page 2023-10-11 12:33:25.930 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'} with connection pooling 2023-10-11 12:33:26.063 DEBUG (MainThread) [pymyq.request] Attempt 1 request failed with exception : 429 - Too Many Requests 2023-10-11 12:33:26.064 DEBUG (MainThread) [pymyq.request] Too many request have been made - putting a temporary pause on sending any requests for 60 minutes 2023-10-11 12:33:36.589 DEBUG (MainThread) [pymyq.api] Performing initial authentication into MyQ 2023-10-11 12:33:36.589 DEBUG (MainThread) [pymyq.api] Scheduling token refresh, last refresh was None 2023-10-11 12:33:36.590 DEBUG (MainThread) [pymyq.api] Initiating OAuth authentication 2023-10-11 12:33:36.591 DEBUG (MainThread) [pymyq.api] Retrieving authentication page 2023-10-11 12:33:36.591 DEBUG (MainThread) [pymyq.request] Sending myq api request https://partner-identity.myq-cloud.com/connect/authorize and headers {'redirect': 'follow'} with connection pooling 2023-10-11 12:33:36.723 DEBUG (MainThread) [pymyq.request] Attempt 1 request failed with exception : 429 - Too Many Requests 2023-10-11 12:33:36.724 DEBUG (MainThread) [pymyq.request] Too many request have been made - putting a temporary pause on sending any requests for 59 minutes 2023-10-11 12:33:37.758 WARNING (MainThread) [homeassistant.components.songpal.media_player] [STR-DN1080(http://192.168.40.203:10000/sony)] Unable to connect

I also get this login screen that shows up IN MY LOG VIEWER.

image

@drysart
Copy link
Contributor Author

drysart commented Oct 11, 2023

@atombombzero MyQ has made additional changes to their API since the fix from this PR that have once again broken it. There's no additional fix available yet.

@atombombzero
Copy link

@atombombzero MyQ has made additional changes to their API since the fix from this PR that have once again broken it. There's no additional fix available yet.

Thanks. I was beating my head against this wondering why. I appreciate the input.

I am a UNIX nerf herder by trade but the seven levels of inception in the HAOS dockers had me stuck for a bit. I got this far and hit this wall. I really appreciate the knowledge that this is NOT something that I can control.

I have time and (limited) skills, so if there is anything that I can do that will help benefit myself and/or the community at large - teach me (what I don't know) and I will get as involved/deep as I can.

@atombombzero
Copy link

@drysart this may be a newb question but here goes: is there a way to git updates to the add-on directly from the CLI without the HAOS GUI (uninstall/install)?

@Lash-L
Copy link
Member

Lash-L commented Oct 11, 2023

@atombombzero

Basically we need to figure out all of the correct parameters for the login requests

I've done this in a few ways before for other apps

  1. https://www.linkedin.com/pulse/easy-guide-apple-ios-packet-capture-amit-singh/ Using a mac and ios, I create a network interface and do low level inspection using wireshark Random link I didn't look at but may explain it( https://nimishprabhu.com/analyze-post-and-get-packets-using-wireshark.html)

  2. I use mitm or Charles Proxy to perform a man in the middle attack to get the rest calls. I tried to do this quickly but I couldn't get all of the requests as it seems like MyQ implements certificate pinning which is super normal now(or something else could have been going on, worth retrying probably) - the only way I know of how to get around it is to use a android device (I use android studio to emulate one), then download the apk of the app, and then use something like this: https://github.com/mitmproxy/android-unpinner to remove the certificate pinning from the apk - download the apk on the device, then setup a mitm attack.

I am sure there are other ways, but I haven't had time to try anything further than just a basic mitm attack. But there are a lot of resources online if you would like to learn a new talent

My understanding is when they originally Reverse Engineered the api, there was a web client you could use to connect to myq, but that does not exist anymore.

@hjdhjd
Copy link

hjdhjd commented Oct 12, 2023

You will need to defeat certificate pinning prior to any additional reverse engineering. That’s less and less easy as devices are getting more locked down over the years…but if someone has an older, compromised device, you can then MITM it.

There’s also a tantalizing MQTT entry point built into myQ. Couldn’t get past it, but if someone with more time and creativity wants to go for it, it’s the better long-term vector, as it appears to have a pub/sub API which would alleviate all sorts of things in the current poll-based method we’re stuck with. Same issues with respect to pinning need to be dealt with there as well.

That’s how the app tends to receive its telemetry. Enjoy.

@drysart
Copy link
Contributor Author

drysart commented Oct 12, 2023

@drysart this may be a newb question but here goes: is there a way to git updates to the add-on directly from the CLI without the HAOS GUI (uninstall/install)?

Yes, if you've got a basic knowledge of using the Linux command line, you can get a bash shell in the homeassistant docker image and update files in there yourself; and they'll persist until the next time your homeassistant image as a whole is updated. There are instructions for doing so in another issue related to the fix for today's breakage (the fix I mentioned we didn't have yet earlier in this thread -- we've got it now).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants