Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aiohttp triggers 403 response from Akamai Global Host on certain machines #5643

Closed
alandtse opened this issue Apr 24, 2021 · 33 comments
Closed
Labels
question StackOverflow

Comments

@alandtse
Copy link
Contributor

🐞 Describe the bug

We have noticed inconsistent behavior using aiohttp client's get while accessing a Tesla auth URL.
https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ

Depending on the machine, sometimes the get works and sometimes we receive a 403 but whatever response is consistent for a particular client machine. This may be related to changes in a web application firewall setup by Tesla but the fact that the same code on multiple machines (including virtual) from the same network have inconsistent results may indicate an underlying aiohttp issue.

Using requests with the same headers as aiohttp results in success across all client machines. If we inject a mitmproxy in the middle for the aiohttp code, we avoid the 403 entirely. Curl and web browsers with the same headers also can access the url consistently on the same machines.

We understand that 403 is coming from the Tesla server but the same machine can use other mechanisms with the same headers/cookies and avoid the 403. This behavior also arose recently so it is probably something Tesla did.

💡 To Reproduce

We have not isolated what causes a particular machine to have this behavior but this is test code that receives the 403 only for aiohttp:

#!/usr/bin/env python

import aiohttp
import asyncio
import requests

head = {
        "User-Agent": "python-requests/2.25.1",
        "Connection": "keep-alive"
        }
url = 'https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ'

async def fetch(client):
    async with client.get(
            url,
            headers = head) as resp:
        return resp

async def main():
    async with aiohttp.ClientSession() as client:
        r = await fetch(client)
        print("AIOHTTP")
        print("Request headers:", r.request_info.headers)
        print("Response Headers:", r.headers)
        print("Response code:", r.status)
        print("Full response:", r)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

sess = requests.Session()
req = sess.get(url)
print("REQUESTS")
print("Request headers:", req.request.headers)
print("Response headers:", req.headers)
print("Response code:", req.status_code)


# vim:syntax=python
# vim:sw=4:softtabstop=4:expandtab

💡 Expected behavior

📋 Logs/tracebacks

➜  config git:(e6d94845dd) python3 test.py
AIOHTTP
Request headers: <CIMultiDictProxy('Host': 'auth.tesla.com', 'User-Agent': 'python-requests/2.25.1', 'Connection': 'keep-alive', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate')>
Response Headers: <CIMultiDictProxy('Server': 'AkamaiGHost', 'Mime-Version': '1.0', 'Content-Type': 'text/html', 'Content-Length': '296', 'Expires': 'Sat, 24 Apr 2021 18:05:40 GMT', 'Date': 'Sat, 24 Apr 2021 18:05:40 GMT', 'Connection': 'close')>
Response code: 403
Full response: <ClientResponse(https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ) [403 Forbidden]>
<CIMultiDictProxy('Server': 'AkamaiGHost', 'Mime-Version': '1.0', 'Content-Type': 'text/html', 'Content-Length': '296', 'Expires': 'Sat, 24 Apr 2021 18:05:40 GMT', 'Date': 'Sat, 24 Apr 2021 18:05:40 GMT', 'Connection': 'close')>

/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'auth.tesla.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
REQUESTS
Request headers: {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
Response headers: {'Server': 'nginx', 'Content-Type': 'text/html; charset=utf-8', 'X-DNS-Prefetch-Control': 'off', 'X-Frame-Options': 'DENY', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains', 'X-Download-Options': 'noopen', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-Request-ID': 'c2231d4c-9d8c-47fd-867f-d0368e2ce6b5', 'X-Correlation-ID': 'c2231d4c-9d8c-47fd-867f-d0368e2ce6b5', 'Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-912d33049fe59e048553'; style-src 'unsafe-inline' 'self'", 'X-Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-912d33049fe59e048553'; style-src 'unsafe-inline' 'self'", 'X-WebKit-CSP': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-912d33049fe59e048553'; style-src 'unsafe-inline' 'self'", 'ETag': 'W/"663b-bhX9PEUNTm9iVzhYOI9wS/7mBsA"', 'X-Response-Time': '29.951ms', 'X-EdgeConnect-MidMile-RTT': '51', 'X-EdgeConnect-Origin-MEX-Latency': '85', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Date': 'Sat, 24 Apr 2021 18:05:41 GMT', 'Content-Length': '5337', 'Connection': 'keep-alive', 'Set-Cookie': 'tesla-auth.sid=s%3AE27ghjQ4B_z9zyGR8hnnTyeulNlsZu8R.1BU0G9ht36N8OovcekfbJHzfMZefzNfic4NkPz4sOVU; Path=/; Expires=Tue, 27 Apr 2021 18:05:40 GMT; HttpOnly; Secure; SameSite=Lax'}
Response code: 200

📋 **Your version of the Python**
<!-- Attach your version of the Python. -->
```console
$ python --version
...
Python 3.9.4

📋 Your version of the aiohttp/yarl/multidict distributions

$ python -m pip show aiohttp
...
Name: aiohttp
Version: 3.7.4.post0
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /opt/homebrew/lib/python3.9/site-packages
Requires: attrs, chardet, yarl, multidict, typing-extensions, async-timeout
Required-by:
$ python -m pip show multidict
...
Name: multidict
Version: 5.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /opt/homebrew/lib/python3.9/site-packages
Requires:
Required-by: yarl, aiohttp
$ python -m pip show yarl
...
Name: yarl
Version: 1.6.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Location: /opt/homebrew/lib/python3.9/site-packages
Requires: multidict, idna
Required-by: aiohttp

📋 Additional context

client
M1 macOS Big Sur 11.2.3
Darwin Alans-MBP 20.3.0 Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101 arm64

More analysis can be found here:
zabuldon/teslajsonpy#190

@alandtse alandtse added the bug label Apr 24, 2021
@Dreamsorcerer
Copy link
Member

The threads around figuring out Tesla's API suggest that these problems are pretty common, it's unlikely to be a bug with aiohttp (but, possibly some difference in the way the requests are sent).

Have you tried using netcat or something to see exactly what the requests are sending, in order to see what differences are present between aiohttp/requests/curl?

I'll close this as not a bug, until proven otherwise. But, please post an update if you figure out what is causing the 403, as I need to update tesla_api soon.

@alandtse
Copy link
Contributor Author

Yes, nc says they're identical. I assumed there's something happening low level in aiohttp so raised the issue here. I do not have the expertise to debug the low level in aiohttp unfortunately.

Aiohttp

config git:(e6d94845dd) nc -l 8888
GET /?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ HTTP/1.1
Host: localhost:8888
User-Agent: python-requests/2.25.1
Connection: keep-alive
Accept: */*
Accept-Encoding: gzip, deflate

Requests

config git:(e6d94845dd) nc -l 8888
GET /?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ HTTP/1.1
Host: localhost:8888
User-Agent: python-requests/2.25.1
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

@Dreamsorcerer
Copy link
Member

Well, the 403 comes from the Tesla server, so any difference has to be in the data being sent, looking at the aiohttp code isn't going to help. If the requests are identical, then the server would respond the same way.

Is it possible that you've been hitting the 403 while making many requests during development, then switched to curl/requests just to test if things still work? The most likely cause (based on discussions from other people hitting this issue) is that you've sent too many requests and Tesla is then blocking the client. Then presumably, some difference in the curl/requests allows new requests to go through, but they would likely also get blocked if you kept sending too many requests.

@Dreamsorcerer
Copy link
Member

I don't know if it's possible to monitor all the data passing through as it happens, which could maybe reveal something.

Netcat doesn't show how the connection is made, so maybe (though it seems unlikely) there's a difference in TLS parameters or something like that...

@alandtse
Copy link
Contributor Author

Thanks. I'm aware of the threads on the 403 on tesla due to rate limiting on ip. If it was a rate limit block, then running the test code posted would be a consistent block on both the requests and aiohttp. That is not the case. You can swap the order and only aiohttp has the issue.

We hypothesized it may be the way aiohttp is establishing the connection with the server and perhaps it's being penalized because it's faster. For example, aiohttp is sub 10ms while requests is 40-70ms. I also noticed with mitmproxy that requests apparently is sending a ALPN and aiohttp isn't. But again, we're just users of the library and don't know if there are settings or tracing that could verify it one way or the other.

Again we realize it's something that Tesla is responding to. Tesla is using Akamai GHost. If aiohttp has a non-consistent bug that triggers a 403 on Akamai GHost, we thought it'd be a good idea to at least give other users of aiohttp a heads up.

Please consider trying the test example. If it works fine for you on your host, then we understand it's not something that can be debugged by you. However, if you run into the 403, perhaps you or some other contributor has an insight we don't have and it might make sense to leave it open.

@mattsch
Copy link

mattsch commented Apr 24, 2021

I've been working with @alandtse to figure this out in zabuldon/teslajsonpy#190, and it's a really weird issue. From my home network, I can reproduce the 403 on a host every time but from a container on that same host (using host networking), the request works every time. It also 403's from my laptop but not from an rpi4. Again, all on the same home network using ipv4 so all NAT'd which means it should look like the same source on the Akamai side.

I've done everything short of dropping to tcpdump to figure out what's different but it all comes back to using aiohttp to connect directly. It's completely reproducible so far, but not consistent across environments.

@webknjaz
Copy link
Member

@alandtse both of your nc logs contain User-Agent: python-requests/2.25.1 and are identical. I suppose you've copied the same thing twice (the query produced by requests, not aiohttp).

Also, seeing that you have redirect_uri= param with non-encoded special chars, the problem is probably that the parts that are supposed to be in its argument are interpreted as parts of the outer URI.

@webknjaz
Copy link
Member

Also, I see the whitespaces encoded as + which may also be problematic as aiohttp reencodes the URLs provided as pure strings. If your URL is pre-encode, you may want to use yarl.URL(..., encoded=True) to prevent the reencoding.

@alandtse
Copy link
Contributor Author

@alandtse both of your nc logs contain User-Agent: python-requests/2.25.1 and are identical. I suppose you've copied the same thing twice (the query produced by requests, not aiohttp).

Also, seeing that you have redirect_uri= param with non-encoded special chars, the problem is probably that the parts that are supposed to be in its argument are interpreted as parts of the outer URI.

Thanks for the comments and thoughts. No it's not copied twice. I used the test script where we copied the same headers as used by requests and just changed the target to the machine running nc.

The issue is the same headers and cookies sent in a get to the listed site results in a 403 from aiohttp but not requests/curl/browsers.

I'll look at the url encoding but I don't think that's a problem. Other machines without the 403 problem access the url just fine.

@webknjaz
Copy link
Member

Ah, then it sounds like an auth problem unrelated to aiohttp in any way..

@webknjaz webknjaz removed the bug label Apr 25, 2021
@alandtse
Copy link
Contributor Author

Ah, then it sounds like an auth problem unrelated to aiohttp in any way..

I'm not sure why it would be an auth problem though. We're not passing any authentication and is the end point is an open endpoint.

There is something about the way aiohttp on certain machines is presenting the request to the end point where the end point is flagging aiohttp for a 403. Perhaps aiohttp looks like a DDOS attack or something else. Essentially, Akamai Global Host 'Server': 'AkamaiGHost' is taking over the response instead of nginx. While we found it on a a Tesla endpoint, this could potentially happen for any site which uses Akamai Global Host servers.

@alandtse alandtse changed the title Aiohttp get receives 403 response on different machines Aiohttp triggers 403 response from Akamai Global Host on certain machines Apr 25, 2021
@webknjaz
Copy link
Member

Based on what you've shown, there is no reason to believe that what aiohttp sends is in any way different from what requests presents.
Auth is not only about sending explicit credentials, sometimes it happens on other levels, taking into account things like originating IPs.

@alandtse
Copy link
Contributor Author

Agreed. So if it's identical, why does it fail when requests.get in the same script doesn't? We tried to eliminate everything besides aiohttp before we opened this ticket.

We've already accounted for IP because we include the requests get in the same script. Are there any other parameters you can think of that we can tweak to isolate the issue?

@mattsch
Copy link

mattsch commented Apr 26, 2021

@webknjaz Something is different in how it interacts. If you read through the discussion on the issue @alandtse linked, we're seeing failures only from aiohtto when requests and httpx both work from the same sets of hosts.

@webknjaz
Copy link
Member

Well, without a reproducer, it's just a guessing game.

@alandtse
Copy link
Contributor Author

Well, without a reproducer, it's just a guessing game.

Would you be willing to try the test script? If you get a 403, it will be completely reproduceable to you on that machine. If not, then you can safely ignore this. I appreciate that you've continued to talk to us despite the fact it's closed.

alandtse added a commit to alandtse/auth_capture_proxy that referenced this issue Apr 26, 2021
aiohttp appears to have issues related to Akamai Global Host and developers
do not seem interested in resolving as a bug.
aio-libs/aiohttp#5643

BREAKING CHANGE: API has changed due to use of httpx.
Modifiers, test_url, and other items that access aiohttp ClientResponse
will need to be fixed.
alandtse added a commit to alandtse/teslajsonpy that referenced this issue Apr 26, 2021
aiohttp appears to have issues related to Akamai Global Host and
developers do not seem interested in resolving as a bug.
aio-libs/aiohttp#5643

BREAKING CHANGE: API has changed due to use of httpx. Modifiers, test_url, and other items that access aiohttp ClientResponse will need to be fixed.
Closes zabuldon#190
@Dreamsorcerer
Copy link
Member

Would you be willing to try the test script? If you get a 403, it will be completely reproduceable to you on that machine. If not, then you can safely ignore this. I appreciate that you've continued to talk to us despite the fact it's closed.

I think that's both successful:

AIOHTTP
Request headers: <CIMultiDictProxy('Host': 'auth.tesla.com', 'User-Agent': 'python-requests/2.25.1', 'Connection': 'keep-alive', 'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate')>
Response Headers: <CIMultiDictProxy('Server': 'nginx', 'Content-Type': 'text/html; charset=utf-8', 'X-DNS-Prefetch-Control': 'off', 'X-Frame-Options': 'DENY', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains', 'X-Download-Options': 'noopen', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-Request-ID': '46c82233-2f68-4d0a-975e-c7e038602702', 'X-Correlation-ID': '46c82233-2f68-4d0a-975e-c7e038602702', 'Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-1be2e3c101cf85b5de40'; style-src 'unsafe-inline' 'self'", 'X-Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-1be2e3c101cf85b5de40'; style-src 'unsafe-inline' 'self'", 'X-WebKit-CSP': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-1be2e3c101cf85b5de40'; style-src 'unsafe-inline' 'self'", 'Etag': 'W/"663b-ic6LpL17ZZ5ZYYDMkit6GvdiUHo"', 'X-Response-Time': '20.782ms', 'X-EdgeConnect-MidMile-RTT': '69', 'X-EdgeConnect-Origin-MEX-Latency': '92', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Date': 'Mon, 26 Apr 2021 16:33:54 GMT', 'Content-Length': '5337', 'Connection': 'keep-alive', 'Set-Cookie': 'tesla-auth.sid=s%3Af-zXBH6dmmNrxldjRQMiPmLRSRgr3hGK.IKkIT5zlWb9TwrSUFOLlpVupW6Q2cV%2BNaNLe84A5hjg; Path=/; Expires=Thu, 29 Apr 2021 16:33:54 GMT; HttpOnly; Secure; SameSite=Lax')>
Response code: 200
Full response: <ClientResponse(https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ) [200 OK]>
<CIMultiDictProxy('Server': 'nginx', 'Content-Type': 'text/html; charset=utf-8', 'X-DNS-Prefetch-Control': 'off', 'X-Frame-Options': 'DENY', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains', 'X-Download-Options': 'noopen', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-Request-ID': '46c82233-2f68-4d0a-975e-c7e038602702', 'X-Correlation-ID': '46c82233-2f68-4d0a-975e-c7e038602702', 'Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-1be2e3c101cf85b5de40'; style-src 'unsafe-inline' 'self'", 'X-Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-1be2e3c101cf85b5de40'; style-src 'unsafe-inline' 'self'", 'X-WebKit-CSP': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-1be2e3c101cf85b5de40'; style-src 'unsafe-inline' 'self'", 'Etag': 'W/"663b-ic6LpL17ZZ5ZYYDMkit6GvdiUHo"', 'X-Response-Time': '20.782ms', 'X-EdgeConnect-MidMile-RTT': '69', 'X-EdgeConnect-Origin-MEX-Latency': '92', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Date': 'Mon, 26 Apr 2021 16:33:54 GMT', 'Content-Length': '5337', 'Connection': 'keep-alive', 'Set-Cookie': 'tesla-auth.sid=s%3Af-zXBH6dmmNrxldjRQMiPmLRSRgr3hGK.IKkIT5zlWb9TwrSUFOLlpVupW6Q2cV%2BNaNLe84A5hjg; Path=/; Expires=Thu, 29 Apr 2021 16:33:54 GMT; HttpOnly; Secure; SameSite=Lax')>

REQUESTS
Request headers: {'User-Agent': 'python-requests/2.25.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
Response headers: {'Server': 'nginx', 'Content-Type': 'text/html; charset=utf-8', 'X-DNS-Prefetch-Control': 'off', 'X-Frame-Options': 'DENY', 'Strict-Transport-Security': 'max-age=15552000; includeSubDomains', 'X-Download-Options': 'noopen', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'X-Request-ID': '280a4c8a-ddbb-40fd-87cf-858e674bfc18', 'X-Correlation-ID': '280a4c8a-ddbb-40fd-87cf-858e674bfc18', 'Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-4e18c48abce316c125f6'; style-src 'unsafe-inline' 'self'", 'X-Content-Security-Policy': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-4e18c48abce316c125f6'; style-src 'unsafe-inline' 'self'", 'X-WebKit-CSP': "connect-src 'self'; default-src 'none'; font-src 'self' data: fonts.gstatic.com; frame-src 'self' www.google.com www.recaptcha.net; img-src 'self' data:; script-src www.recaptcha.net 'self' 'nonce-4e18c48abce316c125f6'; style-src 'unsafe-inline' 'self'", 'ETag': 'W/"663b-ylFH+6vp9v3Q3nbfeaCFEOyB96c"', 'X-Response-Time': '16.757ms', 'X-EdgeConnect-MidMile-RTT': '71', 'X-EdgeConnect-Origin-MEX-Latency': '88', 'Vary': 'Accept-Encoding', 'Content-Encoding': 'gzip', 'Date': 'Mon, 26 Apr 2021 16:33:54 GMT', 'Content-Length': '5340', 'Connection': 'keep-alive', 'Set-Cookie': 'tesla-auth.sid=s%3AR8aZLHwPjrxYNsRA6LqmctaP8s3y625k.9VNrexwnyw%2FGsD%2BAu1Qx%2F6BgiJY1oWA2JsrEste0Z3w; Path=/; Expires=Thu, 29 Apr 2021 16:33:54 GMT; HttpOnly; Secure; SameSite=Lax'}
Response code: 200

@Dreamsorcerer
Copy link
Member

Which suggests that it's still something weird going on with the firewall, as others have reported. I suspect some way, some how, the requests from aiohttp were being recognised as different from the other requests. Then, because you've been developing with aiohttp, this has resulted in getting banned with the aiohttp identifier, rather than there being an issue with aiohttp itself.

If I find out any more information while updating tesla_api, I'll report back.

@alandtse
Copy link
Contributor Author

Which suggests that it's still something weird going on with the firewall, as others have reported. I suspect some way, some how, the requests from aiohttp were being recognised as different from the other requests. Then, because you've been developing with aiohttp, this has resulted in getting banned with the aiohttp identifier, rather than there being an issue with aiohttp itself.

If I find out any more information while updating tesla_api, I'll report back.

Thanks for testing. I would normally agree with you on that but I'm only seeing the 403 on two of four machines I tested within the same network.

  • Macbook M1 host - 403
  • Linux docker container inside Macbook - No 403, and this is my main dev machine
  • Rpi4 host - No 403
  • Linux docker container inside Rpi4 - 403, but same code worked two weeks ago without a 403.

@mattsch also had the error inconsistently within his network.

So that's why I raised it here to find out if there's something else being sent by aiohttp about the host that is configurable. I am happy to do more testing using that switch.

alandtse added a commit to alandtse/auth_capture_proxy that referenced this issue Apr 27, 2021
aiohttp appears to have issues related to Akamai Global Host and developers
do not seem interested in resolving as a bug.
aio-libs/aiohttp#5643

BREAKING CHANGE: API has changed due to use of httpx.
Modifiers, test_url, and other items that access aiohttp ClientResponse
will need to be fixed.
alandtse added a commit to alandtse/auth_capture_proxy that referenced this issue Apr 27, 2021
aiohttp appears to have issues related to Akamai Global Host and developers
do not seem interested in resolving as a bug.
aio-libs/aiohttp#5643

BREAKING CHANGE: API has changed due to use of httpx.
Modifiers, test_url, and other items that access aiohttp ClientResponse
will need to be fixed.
@webknjaz
Copy link
Member

@alandtse I tried out your reproducer and it does indeed return 403 for me. Let me take a deeper look.

@webknjaz webknjaz reopened this Apr 27, 2021
@webknjaz
Copy link
Member

It'd be interesting to compare the whole output of pip list in the differing envs.

@webknjaz
Copy link
Member

but same code worked two weeks ago without a 403.

There's a million reasons for this, unfortunately. Starting with the backend behind a load balancer that proxies the requests to different servers that may have different settings/software versions (this may be caused with rolling or feature flagged releases) and ending with different TLS settings selected and then used as a part of auth disallowing some old protocol versions or ciphers, for example.

I noticed that when I use curl, it switches over to HTTP/2 which is unsupported by aiohttp but adding --http1.1 doesn't make it fail. I know that httpx at least supports HTTP/2 but not sure if requests switches over.

I changed the URL to http:// to see what is being sent over normal HTTP using Wireshark and the HTTP requests look identical indeed. But I can't compare what's happening after the switch to TLS.
One difference that I noticed, though, is that after talking with aiohttp, the server sends TCP RST while with requests it's vice versa (the client side resets the connection).

@webknjaz
Copy link
Member

Looking at the TLS handshake, the clients send different extension lists:

--- aiohttp.tls	2021-04-27 15:28:16.552509935 +0200
+++ requests.tls	2021-04-27 15:28:16.552509935 +0200
@@ -2,23 +2,24 @@
     Handshake Type: Client Hello (1)
     Length: 508
     Version: TLS 1.2 (0x0303)
-    Random: 1f34830de3529e1a7fd07f694f26e1fa0d81951572120756bd2fef8544e006f5
+    Random: dd96563543b09503d56b9142905a1cfcf2d3eadec8f20459040454190ebc8b75
     Session ID Length: 32
-    Session ID: 4ed97926f81af2abda58924b6efc79c1ec75d6fdffbcd4e93b222366ab751d8b
-    Cipher Suites Length: 62
-    Cipher Suites (31 suites)
+    Session ID: 5d85fee2d017f765a1c7c37333a290e5345b271439f0714dc2c1c21f1b1b18f3
+    Cipher Suites Length: 86
+    Cipher Suites (43 suites)
     Compression Methods Length: 1
     Compression Methods (1 method)
-    Extensions Length: 373
+    Extensions Length: 349
     Extension: server_name (len=19)
     Extension: ec_point_formats (len=4)
     Extension: supported_groups (len=12)
-    Extension: session_ticket (len=0)
+    Extension: application_layer_protocol_negotiation (len=11)
     Extension: encrypt_then_mac (len=0)
     Extension: extended_master_secret (len=0)
+    Extension: post_handshake_auth (len=0)
     Extension: signature_algorithms (len=48)
     Extension: supported_versions (len=9)
     Extension: psk_key_exchange_modes (len=2)
     Extension: key_share (len=38)
-    Extension: padding (len=197)
+    Extension: padding (len=158)
```
I wonder if the server decides to send 403 based on this...

@webknjaz
Copy link
Member

Oh, and also the ciphers differ. All the ciphers that aiohttp uses are present in the list that requests sends. But the order is different and requests sends more extra ciphers that aiohttp doesn't: TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_256_CCM, TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8, TLS_ECDHE_ECDSA_WITH_AES_128_CCM, TLS_DHE_RSA_WITH_AES_256_CCM_8, TLS_DHE_RSA_WITH_AES_256_CCM, TLS_DHE_RSA_WITH_AES_128_CCM_8, TLS_DHE_RSA_WITH_AES_128_CCM, TLS_RSA_WITH_AES_256_CCM_8, TLS_RSA_WITH_AES_256_CCM, TLS_RSA_WITH_AES_128_CCM_8, TLS_RSA_WITH_AES_128_CCM.

--- aiohttp.tls	2021-04-27 15:32:05.129983943 +0200
+++ requests.tls	2021-04-27 15:32:05.129983943 +0200
@@ -2,37 +2,49 @@
     Handshake Type: Client Hello (1)
     Length: 508
     Version: TLS 1.2 (0x0303)
-    Random: 1f34830de3529e1a7fd07f694f26e1fa0d81951572120756bd2fef8544e006f5
+    Random: dd96563543b09503d56b9142905a1cfcf2d3eadec8f20459040454190ebc8b75
     Session ID Length: 32
-    Session ID: 4ed97926f81af2abda58924b6efc79c1ec75d6fdffbcd4e93b222366ab751d8b
-    Cipher Suites Length: 62
-    Cipher Suites (31 suites)
+    Session ID: 5d85fee2d017f765a1c7c37333a290e5345b271439f0714dc2c1c21f1b1b18f3
+    Cipher Suites Length: 86
+    Cipher Suites (43 suites)
         Cipher Suite: TLS_AES_256_GCM_SHA384 (0x1302)
         Cipher Suite: TLS_CHACHA20_POLY1305_SHA256 (0x1303)
         Cipher Suite: TLS_AES_128_GCM_SHA256 (0x1301)
         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
-        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
-        Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
-        Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
-        Cipher Suite: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xccaa)
         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
+        Cipher Suite: TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9)
+        Cipher Suite: TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
         Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
+        Cipher Suite: TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xccaa)
+        Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8 (0xc0af)
+        Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CCM (0xc0ad)
+        Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 (0xc0ae)
+        Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CCM (0xc0ac)
         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
-        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
-        Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
-        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
         Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
         Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CCM_8 (0xc0a3)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CCM (0xc09f)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CCM_8 (0xc0a2)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CCM (0xc09e)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
+        Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
         Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
         Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
         Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
+        Cipher Suite: TLS_RSA_WITH_AES_256_CCM_8 (0xc0a1)
+        Cipher Suite: TLS_RSA_WITH_AES_256_CCM (0xc09d)
+        Cipher Suite: TLS_RSA_WITH_AES_128_CCM_8 (0xc0a0)
+        Cipher Suite: TLS_RSA_WITH_AES_128_CCM (0xc09c)
         Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
         Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
         Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
@@ -41,7 +53,7 @@
     Compression Methods Length: 1
     Compression Methods (1 method)
         Compression Method: null (0)
-    Extensions Length: 373
+    Extensions Length: 349
     Extension: server_name (len=19)
         Type: server_name (0)
         Length: 19
@@ -68,16 +80,22 @@
             Supported Group: x448 (0x001e)
             Supported Group: secp521r1 (0x0019)
             Supported Group: secp384r1 (0x0018)
-    Extension: session_ticket (len=0)
-        Type: session_ticket (35)
-        Length: 0
-        Data (0 bytes)
+    Extension: application_layer_protocol_negotiation (len=11)
+        Type: application_layer_protocol_negotiation (16)
+        Length: 11
+        ALPN Extension Length: 9
+        ALPN Protocol
+            ALPN string length: 8
+            ALPN Next Protocol: http/1.1
     Extension: encrypt_then_mac (len=0)
         Type: encrypt_then_mac (22)
         Length: 0
     Extension: extended_master_secret (len=0)
         Type: extended_master_secret (23)
         Length: 0
+    Extension: post_handshake_auth (len=0)
+        Type: post_handshake_auth (49)
+        Length: 0
     Extension: signature_algorithms (len=48)
         Type: signature_algorithms (13)
         Length: 48
@@ -173,9 +191,9 @@
             Key Share Entry: Group: x25519, Key Exchange length: 32
                 Group: x25519 (29)
                 Key Exchange Length: 32
-                Key Exchange: 7f077a5f788dc142bad27b849327e1d9cc39de4e6d183a2a2c60b7f6c8ceea55
-    Extension: padding (len=197)
+                Key Exchange: 5962849cf4d8b4ae2471098850903ca6cf7966e7fe96ae96825e0fcdf15f0633
+    Extension: padding (len=158)
         Type: padding (21)
-        Length: 197
+        Length: 158
         Padding Data: 000000000000000000000000000000000000000000000000000000000000000000000000…

@webknjaz
Copy link
Member

OTOH server hello selects TLS_AES_256_GCM_SHA384 in both cases so it's probably not the client cipher support that causes this difference.

@webknjaz
Copy link
Member

One prominent difference on the HTTP level is the order of the headers (I've extracted this from plain HTTP recording in Wireshark):

--- aiohttp.req	2021-04-27 15:45:52.754698720 +0200
+++ requests.req	2021-04-27 15:45:51.061335499 +0200
@@ -1,16 +1,16 @@
 GET /oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ HTTP/1.1
 Host: auth.tesla.com
 User-Agent: python-requests/2.25.1
-Connection: keep-alive
-Accept: */*
 Accept-Encoding: gzip, deflate
+Accept: */*
+Connection: keep-alive
 
 HTTP/1.1 302 Moved Temporarily
 Location: https://auth.tesla.com/oauth2/v3/authorize?client_id=ownerapi&code_challenge=NDgwM2RlMGE1NTEwZWI1NWIzM2Q2NzM3YTRkYTBlZWNjYWMyOGUzZGZiNDJkNmZkNWE3ZDkxNmQ1MzI5YTg0OQ&code_challenge_method=S256&redirect_uri=https://auth.tesla.com/void/callback&response_type=code&scope=openid+email+offline_access&state=9_MVz16nNle7FSB8-O50bKZId0XNAgTrrIg9agarIiPBV9GnMtsw3uAHeC3jXNjLjs4CSYrqQ5EQBIy-_fmoVQ
 Server: BigIP
 Content-Length: 0
 X-EdgeConnect-MidMile-RTT: 15
-X-EdgeConnect-Origin-MEX-Latency: 152
+X-EdgeConnect-Origin-MEX-Latency: 154
 Date: Tue, 27 Apr 2021 13:05:44 GMT
 Connection: keep-alive

@webknjaz
Copy link
Member

I checked that the RFCs explicitly say that the header order does not matter:

However looking at some search results, I'm pretty sure that Akamai uses a lot of heuristics like matching the header order with known user-agent (browser) behaviors, maybe they also match typical standard browser TLS settings, maybe they mix in some geolocation and usage patterns/ML:

If you want to try to replicate the exact TLS settings, look into providing a custom SSLContext instance via https://docs.aiohttp.org/en/stable/client_advanced.html#ssl-control-for-tcp-sockets. Maybe you'll have some luck pretending to be a browser that should get you through their CDN edge servers all the way to the actual back-end.

P.S. It looks like other libs work mostly by accident and you may expect them to stop working in the future too.

@webknjaz webknjaz added the question StackOverflow label Apr 27, 2021
@Dreamsorcerer
Copy link
Member

Maybe you'll have some luck pretending to be a browser that should get you through their CDN edge servers all the way to the actual back-end.

Or, specifically, appear as the official Tesla app, which is the one thing that will always meet the requirements.

Everything else is completely unofficial and not supported by Tesla, which is why we have such a hard time with it. :P

@alandtse
Copy link
Contributor Author

Thanks for taking an additional look and providing the details. Maybe it is the headers (for purpose of identification). Does aiohttp allow header ordering or if not, would an option be possible to expose if it's the root cause?

alandtse added a commit to alandtse/teslajsonpy that referenced this issue Apr 28, 2021
aiohttp appears to have issues related to Akamai Global Host.
aio-libs/aiohttp#5643

Closes zabuldon#190
@webknjaz
Copy link
Member

@alandtse I didn't mention it but I tested it locally and it doesn't help. But yes, you can construct an instance of multidict.CIMultiDict with the order you want.
This is why I suggested to look deeper into the TLS settings.

@Dreamsorcerer
Copy link
Member

I'm seeing increasing amounts of reports from other people not using aiohttp, for example:
timdorr/tesla-api#260 (comment)

@webknjaz
Copy link
Member

webknjaz commented May 4, 2021

Well, that was quite predictable: looks like they make another round of "improvements" for their antibot prevention layer.

@alandtse
Copy link
Contributor Author

alandtse commented May 5, 2021

At least one person reported that in Python downgrading to TLS 1.2 avoided the 403. They did not explain what library they were using but it is consistent with @webknjaz 's thoughts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question StackOverflow
Projects
None yet
Development

No branches or pull requests

4 participants