00366-CVE-2021-3733.patch

[frenzymadness]

I'm gonna test this on RPM level.

00366 #
CVE-2021-3733: Fix ReDoS in urllib AbstractBasicAuthHandler

Fix Regular Expression Denial of Service (ReDoS) vulnerability in
urllib2.AbstractBasicAuthHandler. The ReDoS-vulnerable regex
has quadratic worst-case complexity and it allows cause a denial of
service when identifying crafted invalid RFCs. This ReDoS issue is on
the client side and needs remote attackers to control the HTTP server.

Backported from Python 3.

[vstinner]

LGTM.

[vstinner]

In Python 3, it was decided to not only change the regex to avoid the worst case performance, but also to "fix" the behavior since the old one was not reliable nor correct. See python#18284 discussion about RFC, web browser behavior, etc.

In Python 2.7, we can make a different trade-off: we can only make the regex more efficient without changing the behavior (pick the last realm when the web server offers multiple Basic realms).

[frenzymadness]

Thanks Victor. I've incorporated the efficiency improvement proposed into the patch and I'll test it now. I will let you know about the results.

[vstinner]

LGTM. If possible, the change should be validated with a benchmark.

[frenzymadness]

Simple reproducer:

from urllib2 import AbstractBasicAuthHandler

AbstractBasicAuthHandler.rx.search(
    "basic " + ("," * 25) + "A"
)

Official one using pyperf:

from urllib2 import AbstractBasicAuthHandler
import pyperf


class AuthHandler(AbstractBasicAuthHandler):
    handler = None

    def retry_http_basic_auth(self, host, req, realm):
        self.realm = realm
        return None

realm = 'realm@example.com'
simple = 'Basic realm="{}"'.format(realm)
repeat_10 = '' + ',' * 10 + 'A'
repeat_10_2 = '' + ',' * (10 ** 2)
repeat_10_4 = '' + ',' * (10 ** 4)

class Headers:
    def __init__(self, header):
        self.header = header

    def get(self, *ignored_args):
        return self.header

    def get_all(self, *ignored_args):
        return [self.header]

def func(handler, headers):
    try:
        handler.http_error_auth_reqed("WWW-Authenticate", None, None, headers)
    except ValueError:
        pass


runner = pyperf.Runner()
handler = AuthHandler()

for name, header in (
    ('simple', simple),
    ('repeat 10', repeat_10),
    ('repeat 10^2', repeat_10_2),
    ('repeat 10^4', repeat_10_4),
):
    headers = Headers(header)
    runner.bench_func(name, func, handler, headers)

Old version:

# time python2 simple_redos.py 

real	0m45.528s
user	0m45.209s
sys	0m0.040s

# time python2 python2_redos.py
.....................
simple: Mean +- std dev: 2.82 us +- 0.19 us
.....................
repeat 10: Mean +- std dev: 427 us +- 27 us
^CBenchmark interrupted: exit

real	25m49.137s
user	25m35.931s
sys	0m1.769s

There is not a single dot for 10² in the benchmark above in more than 25 minutes.

New version:

# time python2 simple_redos.py 

real	0m0.039s
user	0m0.026s
sys	0m0.010s

# time python2 python2_redos.py 
.....................
simple: Mean +- std dev: 2.72 us +- 0.22 us
.....................
repeat 10: Mean +- std dev: 5.76 us +- 0.55 us
.....................
repeat 10^2: Mean +- std dev: 524 us +- 39 us
.....................
repeat 10^4: Mean +- std dev: 5.65 sec +- 0.48 sec

real	8m44.457s
user	8m36.923s
sys	0m2.129s

Benchmark is not as fast as it is for the new Pythons but it's able to finish testing in less than 9 minutes and provides reasonable results also for 10⁴.

I think this can be approved.

[vstinner]

repeat 10: Mean +- std dev: 5.76 us +- 0.55 us
repeat 10^2: Mean +- std dev: 524 us +- 39 us
repeat 10^4: Mean +- std dev: 5.65 sec +- 0.48 sec

LGTM.

https://bugs.python.org/issue39503 and https://bugs.python.org/issue43075 are about exponential complexity, something like: O(n^2). Here, it looks more like O(n) complexity which is the expected result.

[frenzymadness]

I've added one more patch here for CVE-2021-3737. The fix also contains a test and it passes. It is not possible to build Python 2.7 in rawhide now but I've managed to do that in mock with downgraded openssl to version 1.1 and here is verification of the fix using the official reproducer:

Old version:

[root@e5f1018c94e0 src]# python3 server.py &
[1] 72

[root@e5f1018c94e0 src]# time python2 client2.py 

… no response for minutes - infinite loop …

^CTraceback (most recent call last):
  File "client2.py", line 5, in <module>
    r = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1177, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 457, in begin
    skip = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
KeyboardInterrupt

real	3m2.678s
user	2m7.421s
sys	0m54.106s

New version:

[root@73a17da0838d src]# python3 server.py &
[1] 71

[root@73a17da0838d src]# python2 client2.py 
Traceback (most recent call last):
  File "client2.py", line 5, in <module>
    r = conn.getresponse()
  File "/usr/lib64/python2.7/httplib.py", line 1191, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 475, in begin
    skipped_headers = _read_headers(self.fp)
  File "/usr/lib64/python2.7/httplib.py", line 381, in _read_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
httplib.HTTPException: got more than 100 headers

@vstinner could you please take a look once more? No rush, I guess we have to wait for openssl 1.1 packages in rawhide.

[vstinner]

@vstinner could you please take a look once more? No rush, I guess we have to wait for openssl 1.1 packages in rawhide.

Hum, you may mention that it's a backport of: python@47895e3

I checked the backport: yes, it LGTM ;-)

[hroncok]

For the record why I merged this: I've accidentally merged the Fedora PR, so I went ahead and merged this one as well.