[security] CVE-2021-3426: Information disclosure via pydoc -p: /getfile?key=path allows to read arbitrary file on the filesystem

[hroncok]

BPO	42988
Nosy	@malemburg, @gpshead, @vstinner, @ned-deily, @ambv, @serhiy-storchaka, @JulienPalard, @hroncok, @frenzymadness, @miss-islington, @Fidget-Spinner
PRs	bpo-42988: Improve pydoc web server security #24285 bpo-42988: Fix security issue in the pydoc server #24337 bpo-42988: Remove the pydoc getfile feature #25015 [3.9] bpo-42988: Remove the pydoc getfile feature (GH-25015) #25064 [3.8] bpo-42988: Remove the pydoc getfile feature (GH-25015) #25065 [3.7] bpo-42988: Remove the pydoc getfile feature (GH-25015) #25066 [3.6] bpo-42988: Remove the pydoc getfile feature (GH-25015) #25067

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-03-29.19:39:54.149>
created_at = <Date 2021-01-21.12:18:37.837>
labels = ['type-security', '3.8', '3.9', '3.10', '3.7', 'library']
title = '[security] CVE-2021-3426: Information disclosure via pydoc -p: /getfile?key=path allows to read arbitrary file on the filesystem'
updated_at = <Date 2021-03-29.19:39:54.148>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2021-03-29.19:39:54.148>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-29.19:39:54.149>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-01-21.12:18:37.837>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 42988
keywords = ['patch']
message_count = 30.0
messages = ['385412', '385413', '385415', '385418', '385420', '385421', '385422', '385435', '385460', '385485', '385488', '385489', '385492', '385503', '385710', '385721', '385866', '386221', '386222', '388399', '388451', '388452', '388455', '388645', '389452', '389695', '389699', '389700', '389710', '389711']
nosy_count = 11.0
nosy_names = ['lemburg', 'gregory.p.smith', 'vstinner', 'ned.deily', 'lukasz.langa', 'serhiy.storchaka', 'mdk', 'hroncok', 'frenzy', 'miss-islington', 'kj']
pr_nums = ['24285', '24337', '25015', '25064', '25065', '25066', '25067']
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue42988'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[hroncok]

Hello Python security,
a Fedora user has reported the following security vulnerability to us (I was able to verify it):

Running pydoc -p allows other local users to extract arbitrary files.

Steps to Reproduce:

1. start pydoc on a port
2. as a different user guess or extract the port
3. call getfile on the server to extract arbitrary files, e.g. http://localhost:8888/getfile?key=/home/dave/.ssh/id_rsa

Actual results:
any local user on the multi-user system can read all my keys and secrets

Expected results:
Access is prevented.

Additional info:
At least a warning should be printed, that this is insecure on multi-user systems.

Python notebook works around this by providing a token that is required to access the notepad. Depending on the system being able to read arbitrary files can allow to impersonate my, by e.g. stealing my ssh-key (if it is non-encrypted)

I've originally reported this to security@python.org but I was asked to open a public issue here.

[JulienPalard]

Nice find! I am able to reproduce it too in many Python releases.

I see differnt ways we can fix it:

# Using a random secret generated at startup time

Used any way, like by providing an hmac on getfile urls to ensure they are signed with the server secret.

Con: getfile URLS won't work from a run to another run (the secret should be random and changed at every start), and can't be shared (do someone share them in the first place?)

# Allowlist according to sys.path

In getfile implementation, we can check if the asked path is under a path from sys.path.

Con: If someone have ~/ in sys.path, we still can access all its home, or if someone start it using python -m pydoc while being in its home directory as Python will prepend the cwd in sys.path.

# Allowlist populated while generating links

Idea is: each time the server generates a getfile link, the target is added to an allowlist.

Each time a getfile link is requested, if the file is not in the allowlist, request is denied.

Con: Refreshing a page won't work after a server restart (thus having an empty allowlist).

# fnmatch allowlist

We could allow only .py files.

Con: Secrets stored in .py files under user project could still be leaked.

-----------------

My personal preference goes for the allowlist populated while generating links.

[vstinner]

Downstream Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1917807

[malemburg]

Looking at the _url_handler() code in pydoc.py, this was clearly not written with web server standards in mind. None of the handlers apply security checks on the user input and there are most likely several other vulnerabilities in there to be found.

It's not just the getfile query allowing reading arbitrary files. The user may well have code in his or her Python installation which is not meant to be published to other users on the same server.

I'd suggest to print a big warning on the console, explaining that the web server will potentially make all content accessible by the user visible to anyone else on the same server.

Perhaps adding some extra check to the html_getfile() handler would be good as well, making sure that the path is on sys.path and maps to a Python file (there could be non-Python file resources in package dirs as well).

Alternatively, perhaps the whole getfile logic could be removed and the web server just provide the path to the source file (as file:// link), so that the user can easily open it, but needs access permissions for this to be successful.

[vstinner]

I searched for "pydoc by Ka-Ping Yee" in Google and only found two online pydoc services:

• https://gae-pydoc.appspot.com/
• http://www.cc.kyoto-su.ac.jp/~atsushi/Programs/VisualWorks/CSV2HTML/CSV2HTML_PyDoc/index_of_modules.html

The first one runs on Python 2.7 which doesn't have the getfile feature (added in Python 3.6), the second one is a static website.

=> there is no public vulnerable website: good!

I don't think that pydoc is commonly used to run a server on the Internet.

[vstinner]

An option is also to remove the whole getfile feature. It was added in bpo-2001 by:

commit 7bb30b7
Author: Nick Coghlan <ncoghlan@gmail.com>
Date: Fri Dec 3 09:29:11 2010 +0000

Improve Pydoc interactive browsing (bpo-2001).  Patch by Ron Adam.

* A -b option to start an enhanced browsing session.
* Allow -b and -p options to be used together.
* Specifying port 0 will pick an arbitrary unused socket port.
* A new browse() function to start the new server and browser.
* Show Python version information in the header.
* A *Get* field which takes the same input as the help() function.
* A *Search* field which replaces the Tkinter search box.
* Links to *Module Index*, *Topics*, and *Keywords*.
* Improved source file viewing.
* An HTMLDoc.filelink() method.
* The -g option and the gui() and serve() functions are deprecated.

[vstinner]

The getfile feature is used to display the source code of a Python module.

For example, on the difflib documentation, there a link to difflib.py. If you click, a webpage displays the content of the file.

I suggest to remove the whole feature. I don't think that it's so useful, compared to the vulnerability.

[Fidget-Spinner]

I created a PR to remove the getfile function - now it just places the hyperlinked file path there but clicking on it won't render the file contents.

Personally I agree with Marc-Andre Lemburg's comments on how _url_handler probably has other vulnerabilities somewhere. But I don't really see an easy solution other than removing the web server altogether. It uses http.server, which has a disclaimer on the docs page saying it isn't recommended for production. Someone looking hard enough can probably find a few more vulnerabilities in http.server itself rather than just pydoc.

I think the "Allowlist populated while generating links" suggested by Julien is pretty clever.

I thought about file: // approach too - it's probably the most secure. But it would require a lot of change (and also generating all the .py files to .html initially).

Maybe I'll make a PR exploring the other approaches if the current one isn't favorable.

Thanks for your time.

[vstinner]

I'd suggest to print a big warning on the console, explaining that the web server will potentially make all content accessible by the user visible to anyone else on the same server.

I dislike this idea. If they are vulnerabilities, they should be fixed. Users usually have no idea what to do when seeing such warning.

[malemburg]

On 22.01.2021 01:28, STINNER Victor wrote:

STINNER Victor <vstinner@python.org> added the comment:

> I'd suggest to print a big warning on the console, explaining that the web server will potentially make all content accessible by the user visible to anyone else on the same server.

I dislike this idea. If they are vulnerabilities, they should be fixed. Users usually have no idea what to do when seeing such warning.

The problem is that neither the docs nor the help text in the command
make it clear what exactly is exposed via the web server pydoc
launches.

While the getfile API endpoint can be used to view non-Python files
as well (which is certainly not intended), the tool also makes available
all Python modules which can be found on sys.path of the user starting
pydoc -p. It shows all doc-strings, functions, the class structure and
literal values of any constants found in those modules.

In a corporate environment this can easily result in data leaks
of e.g. unreleased software, personal information, disclosure of
NDA protected code, designs, algorithms and other secrets.

Fixing just getfile or replacing those links with file:// ones will
only address one part of the problem. The other is educating the
user about possible consequences of running a server on the machine
-- just like you warn users about deleting files before going ahead
with it.

Python's http.server at least warns about this in the docs:
https://docs.python.org/3/library/http.server.html
and limits the serving to the current dir (and subdirs).

My guess is that pydoc -p really is just intended to be useful
for the current user. Rather than having it serve files under
a blanket URL, it could restrict browsing to a random URL
token generated at pydoc startup and open this in the browser
via the "b" command or the -b option, e.g.

"""
Server ready at http://localhost:8080/uLy6t87AD-ScPthd/
Server commands: [b]rowser, [q]uit
server>
"""

That would make it harder to guess the base URL and limit
exposure.

[serhiy-storchaka]

Why not limit the serving to sys.path?

[vstinner]

Fidget-Spinner wrote on the PR:

AFAIK no. However, pydoc currently works by calling inspect on files it sees in path, and this may reveal private code as Marc-Andre Lemburg pointed out on the bpo. I will try the random url token he suggested via secrets.token_urlsafe to see if it helps.

pydoc shows global constant values in the doc. So yes, if you find a settings.py of a Django project, you can discover secrets.

I'm working on bpo-42955 "Add sys.module_names: list of stdlib module names (Python and extension modules)".

One option would be to restrict pydoc to stdlib modules by defaults, and ask to opt-in for discovery of any module installed on the system (sys.path).

[vstinner]

Python's http.server at least warns about this in the docs:
https://docs.python.org/3/library/http.server.html
and limits the serving to the current dir (and subdirs).

I would be fine with a warning in the pydoc documentation, but I dislike warnings display on the command line. When I see such warning, I feel that the machine considers that I'm dumb and I have no idea of what I am doing.

If it's unsafe, can we make it safe by default?

[Fidget-Spinner]

I have updated the PR to do the following:

• removed html_getfile
• implement a unique secret as suggested above

Now it says:
>>> python.exe -m pydoc -b
Server ready at http://localhost:52035/Y1YzOyEbitE9BB_dtH0YXbMgGXbcg3ytXLpvpg8P7GEM3z1hlCkTXgxaojtAordVqs2s6oHZHPMbXqq9mXq_wbJCVW8jnHrgQeYE5hFUQuI/

FWIW, it seems that Jupyter notebook server deals with the same problems in a similar manner: https://jupyter-notebook.readthedocs.io/en/stable/security.html#security-in-the-jupyter-notebook-server

I removed the warning message in the PR because I think this is secure enough.

[serhiy-storchaka]

PR 24337 uses different approach. It keeps compatibility, but checks that the argument is a file path of the source of one of modules (using the same algorithm as /search).

[Fidget-Spinner]

@serhiy,

While this approach solves the getfile problem, I don't think this will solve the other problem of pydoc leaking secrets stored in python files:

Quoting from Marc-Andre Lemburg's message:

the tool also makes available all Python modules which can be found on sys.path of the user starting pydoc -p. It shows all doc-strings, functions, the class structure and literal values of any constants found in those modules.
In a corporate environment this can easily result in data leaks of e.g. unreleased software, personal information, disclosure of NDA protected code, designs, algorithms and other secrets.

Quoting from Victor's messages:

pydoc shows global constant values in the doc. So yes, if you find a settings.py of a Django project, you can discover secrets.

Ultimately, the problem seems to be that .py files (other than those in the stdlib) may contain sensitive info, which pydoc can read.

[ned-deily]

Resolution of this issue is blocking 3.7.x and 3.6.x security releases and threatens to block upcoming maintenance releases.

[vstinner]

While this vulnerability is bad, it only impacts very few users who run pydoc server. I suggest to not hold the incoming Python release (remove the "release blocker" priority) just for this one. If it's fixed before: great! But IMO it can wait for another Python release.

[hroncok]

I agree.

[hroncok]

Todd Cullum from Red Hat Security team:

"I don't have an account on Python's tracker, would you mind forwarding to upstream on my behalf that this is not only locally exploitable, but it can be exploited by actors on the adjacent network as well because 6a396c9 was introduced in Python 3.7.0 alpha 1. I just used the -n option and got to read some of my own files using my cell phone on the WiFi. It does require the port to be unblocked by firewall though."

[hroncok]

This is now CVE-2021-3426.

[vstinner]

I created https://python-security.readthedocs.io/vuln/pydoc-getfile.html to track this vulnerability. The is no CVE section yet since the CVE is currently only *RESERVED*.

[vstinner]

Fedora downstream issue: https://bugzilla.redhat.com/show_bug.cgi?id=1937476

[gpshead]

FWIW, I don't think we should even have a server feature in pydoc...

[vstinner]

The "pydoc -p port" command only listen on the local link ("localhost") by default, even if it's possible to listen on another IPv4 address using -n HOSTNAME command line option.

While the "getfile" feature is convenient when the pydoc server is accessed from a different machine, I don't think that it's worth it, compared to the security risks and the complexity of PR 24285 and PR 24337 fixes.

I propose to simply remove the "getfile" feature with PR 25015, but keep links using file:// scheme. So we delegate the security to the web browser. If the web browser is allowed to read sensitive data of a local Python file, it's not our problem: pydoc doesn't make things worse.

[vstinner]

New changeset 9b99947 by Victor Stinner in branch 'master':
bpo-42988: Remove the pydoc getfile feature (GH-25015)
9b99947

[miss-islington]

New changeset 7e38d33 by Miss Islington (bot) in branch '3.8':
bpo-42988: Remove the pydoc getfile feature (GH-25015)
7e38d33

[miss-islington]

New changeset ed753d9 by Miss Islington (bot) in branch '3.9':
bpo-42988: Remove the pydoc getfile feature (GH-25015)
ed753d9

[ned-deily]

New changeset 7c2284f by Miss Islington (bot) in branch '3.7':
bpo-42988: Remove the pydoc getfile feature (GH-25015) (bpo-25066)
7c2284f

[ned-deily]

New changeset 5b1e502 by Miss Islington (bot) in branch '3.6':
bpo-42988: Remove the pydoc getfile feature (GH-25015) (GH-25067)
5b1e502

[apetro76]

Hello Python security, a Fedora user has reported the following security vulnerability to us (I was able to verify it):

Running pydoc -p allows other local users to extract arbitrary files.

Steps to Reproduce:

1. start pydoc on a port
2. as a different user guess or extract the port
3. call getfile on the server to extract arbitrary files, e.g. http://localhost:8888/getfile?key=/home/dave/.ssh/id_rsa

Actual results: any local user on the multi-user system can read all my keys and secrets

Expected results: Access is prevented.

Additional info: At least a warning should be printed, that this is insecure on multi-user systems.

Python notebook works around this by providing a token that is required to access the notepad. Depending on the system being able to read arbitrary files can allow to impersonate my, by e.g. stealing my ssh-key (if it is non-encrypted)

I've originally reported this to security@python.org but I was asked to open a public issue here.

Curious, what is actually being called out as a vulnerability here? The fact that pydoc doesn't enforce authentication? Seems this works the same as any other service/application works (when accessing the application/service, you are granted the permissions of the account running that application/service when it interacts with the operating system). I can think of dozens of applications and protocols that would be victim to this same finding. For example, running python -m http.server 80 . from a home directory does the same thing (still works even on version 3.11) does that mean that python3.11 is broken?

[gpshead]

It's a matter of meeting expectations for users of a tool so that they know when they're choosing to expose themselves to risks.

Python's standard library http.server module is documented as not suitable for production use. ie: the user chooses to run that despite being warned, that isn't a flaw, that is their choice.

pydoc's former built in web server could reasonably have been expected not to serve other data, as that isn't what the user asked for or would ever assume it would allow. Thus it was reasonable to consider it doing so a vulnerability.

[apetro76]

thanks for the explanation, that does make a bit more sense. Personally I am not familiar with pydoc, so when reading the vulnerability through the first time it sounded no different than if nginx, apache or iis was misconfigured, or running a tftp, ftp, or smb share that disclosed sensitive documents or for that matter any other production application that is used for file sharing. I appreciate the explanation and will chalk it up to my ignorance to pydoc and how its used.