Description
After some research, I found that httpx.Client and httpx.Proxy may implicit parse wrong URL because of the improper implement of httpx.URL().copy_with.
And this issue may lead to some blacklist bypass.
For example:
httpx.Client
user_input_from_http_request = 'http:////admin-dashboard/secret'
u = httpx.URL(user_input_from_http_request)
assert u.host.lower() != 'admin-dashboard'
resp = httpx.Client(base_url=u).get('/') # SSRF to http://admin-dashboard/secret
print(resp.text) # sensitive data leakhttpx.Proxy
user_input_from_http_request = 'http://x@//internal-proxy:8082/'
u = httpx.URL(user_input_from_http_request)
assert u.host.lower() != 'internal-proxy'
# httpx.Proxy(u).url.netloc == b'internal-proxy:8082'
resp = httpx.Client(proxies=u).get('http://localhost/secret') # will request via http proxy at internal-proxy:8082Main reason:
Line 487 in b07fe7b
copy_with parse self._uri_reference.copy_with(**kwargs).unsplit() before returning the new URL, but the new URL string return by unsplit may make some unintended changes on the new URL.
For example:
print(httpx.URL('http://[invalid string!!!!]//localhost/test!'). _uri_reference.unsplit())
# output: http://localhost/test!
u = httpx.URL('http://[invalid string!!!!]//localhost/test!').copy_with(scheme='https') # I only want to change scheme
print(u.host)
# output: localhost
# oops, host changed!So if a function is using copy_with, it may have the same issue as httpx.Client and httpx.Proxy, too (For example, copy_set_param).
I also made a patch PR for this issue by replacing return URL(self._uri_reference.copy_with(**kwargs).unsplit()) to:
new_url = URL(self)
new_url._uri_reference = self._uri_reference.copy_with(**kwargs)
if new_url.is_absolute_url:
new_url._uri_reference = new_url._uri_reference.normalize()
return URL(new_url)By the way, I think this issue is similar to CWE-172 and CWE-20.
If you want to request a CVE id for this issue to remind httpx's user, you can use these categories.
Updated: This has been assigned as CVE-2021-41945.