URL pattern does not match or fails on URL-encoded special ascii symbols in URL

[alexdrydew]

Hey!

I have faced a bug during an attempt to use respx with hypothesis to mock autogenerated URLs when encoded special symbols are present in the url path.

This code fails with the AllMockedAssertionError:

import httpx
import respx

with respx.mock:
    url = 'http://test.com/%0a'
    respx.get(url=url).respond()
    httpx.get(url)

This code fails with httpx.InvalidURL:

with respx.mock:
    url = 'http://test.com/%08'
    respx.get(url=url).respond()
    assert httpx.get(url).status_code == 200

Full traceback:

Traceback (most recent call last):
  File "...", line 8, in <module>
    respx.get(url=url).respond()
  File ".../python3.9/site-packages/respx/api.py", line 74, in get
    return mock.get(url, name=name, **lookups)
  File ".../python3.9/site-packages/respx/router.py", line 174, in get
    return self.request(method="GET", url=url, name=name, **lookups)
  File ".../python3.9/site-packages/respx/router.py", line 165, in request
    return self.route(method=method, url=url, name=name, **lookups)
  File ".../python3.9/site-packages/respx/router.py", line 132, in route
    route = Route(*patterns, **lookups)
  File ".../python3.9/site-packages/respx/models.py", line 119, in __init__
    self._pattern = M(*patterns, **lookups)
  File ".../python3.9/site-packages/respx/patterns.py", line 544, in M
    extras = parse_url_patterns(value)
  File ".../python3.9/site-packages/respx/patterns.py", line 648, in parse_url_patterns
    bases[Path.key] = Path(url.path, lookup=lookup)
  File ".../python3.9/site-packages/respx/patterns.py", line 92, in __init__
    self.value = self.clean(value)
  File ".../python3.9/site-packages/respx/patterns.py", line 417, in clean
    value = httpx.URL(path).path
  File ".../python3.9/site-packages/httpx/_urls.py", line 113, in __init__
    self._uri_reference = urlparse(url, **kwargs)
  File ".../python3.9/site-packages/httpx/_urlparse.py", line 160, in urlparse
    raise InvalidURL("Invalid non-printable ASCII character in URL")
httpx.InvalidURL: Invalid non-printable ASCII character in URL

I could reproduce this problem only with the url pattern, the following works fine:

from urllib.parse import urlparse

import httpx
import respx

with respx.mock:
    url = 'http://test.com/%08'
    parsed = urlparse(url)
    respx.get(scheme=parsed.scheme, host=parsed.hostname, path=parsed.path, params=parsed.params).respond()
    assert httpx.get(url).status_code == 200

Versions:

• python 3.9.7
• respx==0.20.1
• httpx==0.24.0

[lundberg]

Looking at the stack trace, it's seems to be a httpx "bug". I don't think we should try to fix this in respx, but rather open an httpx issue and hopefully it gets fixed there, and then automatically gets supported in respx.

[alexdrydew]

I believe this is a misuse of the httpx.URL:

respx/respx/patterns.py

Line 417 in b014780

value = httpx.URL(path).path

RESPX attempts to initialize it with the unquoted path (which is indeed an invalid URL in this case), while httpx.URL seems to work correctly with the quoted version of the URL.

[lundberg]

Well, that might be the case. Let's handle the URL instantiation same way as httpx does then, if they support the unparsed url 👍