[security][CVE-2020-27619] Python testsuite calls eval() on content received via HTTP

[serhiy-storchaka]

BPO	41944
Nosy	@vstinner, @ned-deily, @zware, @serhiy-storchaka, @The-Compiler, @pablogsal, @miss-islington
PRs	bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests #22566 bpo-41944: No longer call eval() on content received via HTTP in the UnicodeNames tests #22575 [3.9] bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) #22576 [3.8] bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) #22577 [3.7] bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) #22578 [3.6] bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) #22579

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-20.04:47:52.035>
created_at = <Date 2020-10-05.14:40:52.277>
labels = ['type-security', '3.8', '3.9', '3.10', '3.7', 'tests']
title = '[security][CVE-2020-27619] Python testsuite calls eval() on content received via HTTP'
updated_at = <Date 2020-11-04.13:09:52.449>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-11-04.13:09:52.449>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-20.04:47:52.035>
closer = 'ned.deily'
components = ['Tests']
creation = <Date 2020-10-05.14:40:52.277>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 41944
keywords = ['patch', 'security_issue']
message_count = 19.0
messages = ['378036', '378104', '378105', '378106', '378107', '378108', '378110', '378111', '378114', '378117', '378118', '378119', '378120', '378125', '379082', '379085', '379713', '380319', '380320']
nosy_count = 7.0
nosy_names = ['vstinner', 'ned.deily', 'zach.ware', 'serhiy.storchaka', 'The Compiler', 'pablogsal', 'miss-islington']
pr_nums = ['22566', '22575', '22576', '22577', '22578', '22579']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue41944'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[serhiy-storchaka]

As was reported by Florian Bruhin, Python testsuite calls eval() on content received via HTTP (in Lib/test/multibytecodec_support.py).

[The-Compiler]

I wonder if I should request a CVE for this as well? Just to make sure the word gets out to distributions/organizations/etc. running the Python testsuite, given that we can't be sure it which contexts this happens (and as it could be exploited by e.g. spoofing a WiFi network or so).

[vstinner]

I don't think that a CVE is justified.

I don't know anyone running the Python test suite on production. Only developers of Python itself run Python.

[vstinner]

Oops: Only developers of Python itself run the Python test suite.

[The-Compiler]

That assumption is false. For starters, distribution packagers do:

https://github.com/archlinux/svntogit-packages/blob/3fc85177e35d1ff9ab000950c5d1af9567730434/trunk/PKGBUILD#L72-L84

https://src.fedoraproject.org/rpms/python3.9/blob/master/f/python3.9.spec#_1168

When I build a Python from source (via an Arch User Repository package), I do as well, and so does anyone installing those packages by default.

Anyone building with --enable-optimizations (PGO) will likely do so as well, though I'm not sure if that runs this part of the testsuite.

[vstinner]

I'm not saying that this issue is not a vulnerability, just that the scope is limited.

By default, downloaded from the Internet are disabled. You have to opt-in for that using -u network (or -u all which enables the network resource) command line option of "./python -m test".

Impacted:

• "make testall", "make testuniversal" and "make buildbottest" commands are impacted (pass -u all to the test suite).

• Python buildbot workers are impacted: they run the "make buildbottest" command.

• Travis CI is impacted: it runs "./python -m test -uall,-cpu (...)".

• Multiple GitHub Action jobs are impacted (coverage, Windows, macOS, Ubuntu): run "-uall,-cpu".

• Azure Pipelines jobs are impacted: use -uall,-cpu.

https://src.fedoraproject.org/rpms/python3.9/blob/master/f/python3.9.spec#_1168

Fedora packages are not impacted: no -u option is passed to the test suite.

Anyone building with --enable-optimizations (PGO) will likely do so as well, though I'm not sure if that runs this part of the testsuite.

PGO build is not impacted, it uses "./python -m test --pgo" (download is disabled). Moreover, multibyte codec checks are not run by this command (see Lib/test/libregrtest/pgo.py, only test_codecs of codec tests is run).

[The-Compiler]

Thanks for the clarification - I wasn't aware those tests aren't run by default.

FWIW I found another place where a similar thing is done, though by chance it's probably not exploitable - see #66765.

[vstinner]

FWIW I found another place where a similar thing is done, though by chance it's probably not exploitable - see #66765.

I agree that test_ucn is not exploitable, but it would be nice to harden it anyway.

Extract of the code:

        self.assertEqual(unicodedata.lookup(seqname), codepoints)
        with self.assertRaises(SyntaxError):
            self.checkletter(seqname, None)

test_ucn downloads http://www.pythontest.net/unicode/13.0.0/NamedSequences.txt and calls checkletter() on each line, but first it ensures that unicodedata.lookup(seqname) works as expected.

I don't see how it would be possible to inject arbitrary Python code in the 'seqname' variable without making unicodedata.lookup() to fail.

[vstinner]

I'm now tracking this vulnerability at:
https://python-security.readthedocs.io/vuln/cjk-codec-download-eval.html

[serhiy-storchaka]

New changeset 2ef5caa by Serhiy Storchaka in branch 'master':
bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566)
2ef5caa

[miss-islington]

New changeset b664a1d by Miss Skeleton (bot) in branch '3.9':
bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566)
b664a1d

[vstinner]

New changeset 6c6c256 by Miss Skeleton (bot) in branch '3.8':
bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) (GH-22577)
6c6c256

[vstinner]

Since it's a security vulnerability, I created backports to 3.6 and 3.7 as well.

[vstinner]

New changeset a8bf44d by Florian Bruhin in branch 'master':
bpo-41944: No longer call eval() on content received via HTTP in the UnicodeNames tests (GH-22575)
a8bf44d

[ned-deily]

New changeset 43e5231 by Miss Skeleton (bot) in branch '3.7':
bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) (GH-22578)
43e5231

[ned-deily]

New changeset e912e94 by Miss Skeleton (bot) in branch '3.6':
bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566) (GH-22579)
e912e94

[vstinner]

Thanks for the fix Serhiy and thanks Florian Bruhin for the bug report!

[vstinner]

The CVE-2020-27619 has been assigned to this issue.

[vstinner]

Red Hat advisory: https://access.redhat.com/security/cve/CVE-2020-27619

[serhiy-storchaka]

@gvanrossum, is there anything else that needs to be done in this issue?

[gvanrossum]

Whoops, didn’t mean to reopen. This has happened a few times — I look at an issue, maybe subscribe, and it gets reopened?