Repository navigation
Potential issue: URL.host decodes IDNA only when the first label is punycode, and leaks idna exceptions #3818
Replies: 1 comment
|
Both halves of this report check out on version 0.28.1.
httpx.URL with "http://xn--fiqs8s.icom.museum" returns a decoded host. So, I presume, decoding = a prefix check. A punycode label anywhere other than position 1 is returned still encoded, while raw_host keeps the encoded bytes. The .host property is therefore not "IDNA hosts decoded into unicode" as its docstring claims. It is "decoded only when the first label happens to be punycode."
httpx.URL with "http://xn--zzzzzz.com" constructs fine. That is an idna.IDNAError (a UnicodeError subclass) leaking out of an httpx property, not an httpx.InvalidURL. Callers catching InvalidURL around URL parsing never see it, and the URL was already accepted. Fix direction: attempt idna.decode() for every label, or at least every label with the xn-- prefix, rather than only the first. Either fall back to the raw host on IDNAError, or translate it into InvalidURL at parse time so the failure is typed and raised where the URL is constructed. |
Uh oh!
There was an error while loading. Please reload this page.
Raising this here per the contributing guide, as PR creation on this repo is limited to collaborators. Found with property-based testing (per-label IDNA round-trip properties over generated hosts).
URL.host promises "IDNA hosts decoded into unicode", but the decoding only happens when the first label of the host is punycode, because the whole host string is checked with startswith("xn--"):
The existing IDNA tests only ever place the internationalized label first, so this was not covered. The property is consumed internally by URLPattern matching and redirect handling, so an undecoded host can also affect proxy/redirect decisions.
The same accessor has a second defect: for a label that looks like punycode but does not decode, it leaks the idna package's exception types out of a property access:
The fix decodes each xn-- label individually, and leaves a label that fails to decode in its raw form instead of raising. It is ready, with regression tests for both cases, on a fork branch: https://github.com/Sun-Wukhan/httpx/tree/fix/url-host-idna-labels (the full tests/models suite passes with it). Happy for a maintainer to open the PR from it, or to open one myself if the restriction is lifted for it.
All reactions