[tangentially related to CVE-2023-24329] urlparse does not correctly handle schemes that begin with ASCII digits, '+', '-', and '.' characters

[kenballus]

Background

RFC 3986 defines a scheme like this:

• scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )

RFC 2234 defines an ALPHA like this:

• ALPHA = %x41-5A / %x61-7A

The WHATWG URL spec defines a scheme like this:

• "A URL-scheme string must be one ASCII alpha, followed by zero or more of ASCII alphanumeric, U+002B (+), U+002D (-), and U+002E (.)."

The bug

This is the scheme string parsing code from Lib/urllib/parse.py:462-468:

    i = url.find(':')
    if i > 0:
        for c in url[:i]:
            if c not in scheme_chars:
                break
        else:
            scheme, url = url[:i].lower(), url[i+1:]

This is the definition of scheme_chars from Lib/urllib/parse.py:77-80:

scheme_chars = ('abcdefghijklmnopqrstuvwxyz'
                'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
                '0123456789'
                '+-.')

This will erroneously validate schemes that begin with any of ('.', '-', '+', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9'). This behavior is in violation of both specifications.

This bug is reproducible with the following snippet:

>>> from urllib.parse import urlparse
>>> urlparse(".://") # Should error, but doesn't
ParseResult(scheme='.', netloc='', path='', params='', query='', fragment='')

My environment

• CPython versions tested on:
  • 3.12.0a1+ (fb844e1)
  • 3.10.8
• Operating system and architecture:
  • Arch Linux x86_64

• PR: gh-99418: Make urllib.parse.urlparse enforce that a scheme must begin with an alphabetical ASCII character. #99421

• PR: [3.11] gh-99418: Make urllib.parse.urlparse enforce that a scheme must begin with an alphabetical ASCII character. (GH-99421) #99446

[hauntsaninja]

Thanks, looks like this has been fixed

[vstinner]

CVE-2023-24329 was assigned to this issue.

[vstinner]

Python 3.7, 3.8 and 3.9 are affected by this issue and still get security fixes.

@gpshead: Should this fix be backported to Python 3.7-3.9?

[vstinner]

Ah, I don't see a fix for Python 3.10 neither, whereas the issue was reported on Python 3.10.

[vstinner]

I created https://python-security.readthedocs.io/vuln/urlparse-scheme.html to track fixes of this issue.

[gpshead]

Please see #102153..

[gpshead]

(ie: that python-security urlparse-scheme blog text is currently wrong: this is not fixed, and the first report was in July, not November)

[vstinner]

(ie: that python-security urlparse-scheme blog text is currently wrong: this is not fixed, and the first report was in July, not November)

I'm maintaining this page manually and it's quite a lot of work to maintain it. I tried to automate as many things as possible. The source can be found in the YAML file: https://github.com/vstinner/python-security/blob/main/vulnerabilities.yaml#L2134

Free free to propose a PR to fix the entry ;-)

[ngie-eign]

The fix for this CVE should really be backported if applicable.

[gpshead]

The fix for this CVE should really be backported if applicable.

This issue does not contain the fix.

See #102153.

[ngie-eign]

The fix for this CVE should really be backported if applicable.

This issue does not contain the fix.

See #102153.

@gpshead : thank you so very much for the pointer! I'll do some poking around next week to see if some other OS distributions have addressed this and if there aren't any available fixes, try crafting (an) appropriate patch(es) and link it/them to the appropriate issue.

I work on a project that uses 3.8; if it's too much work for 3.7, I'll just look into making the 3.8 patch work.