Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ReDoS vulnerability in svnurl.py #287

Open
SCH227 opened this issue Sep 22, 2022 · 34 comments
Open

ReDoS vulnerability in svnurl.py #287

SCH227 opened this issue Sep 22, 2022 · 34 comments

Comments

@SCH227
Copy link

SCH227 commented Sep 22, 2022

Good night!

I found that this regex is vulnerable to Regular Expression Denial of Service.

PoC:

>>> from py._path.svnurl import InfoSvnCommand
>>> payl = "   2256      hpk        165 Nov 24 17:55 __init__.py" + " " * 5000
>>> InfoSvnCommand(payl)

Attack vector:

An user accessing a (possibly remote) subversion repository that provides malicious "info" data.
Or an attacker injecting 'svn ls http://...' output (less realistic).

Fix:

Use a pattern with non-overlapping groups. I can help in finding a better regex and testing if needed.

@The-Compiler
Copy link
Member

Related: #256

@bluetech
Copy link
Member

I doubt there's anyone using this code, so I don't think it would warrant any sort of security notification that users should bother with, but if you prepare a PR for this I'll merge & release it.

@skialpine
Copy link

GHSA-w596-4wvx-j9j6
https://nvd.nist.gov/vuln/detail/CVE-2022-42969

@jenstroeger
Copy link

As @skialpine mentions, the advisory now triggers pip-audit and thus can fail CI runs (like this one).

Is there a fix on the horizon?

@RonnyPfannschmidt
Copy link
Member

The issue is not considered critical, im not aware of anyone working towards a fix

@The-Compiler
Copy link
Member

The-Compiler commented Oct 19, 2022

Well, congratulations to whoever it is that decided that the right path of action here is getting a CVE for...

  • Something that seems very questionable to be titled a "vulnerability" in the first place (if I can control an SVN repo/server, I might as well just Slowloris the initial info request, I suppose... though admittedly I didn't try that.)
  • For some ~18 year old code...
  • ...which is only there for historical reasons and discouraged to use
  • ...which, to the best of our knowledge, is not used anywhere in the wild outside of some old PyPy development scripts nobody probably uses anymore (and certainly not against random SVN servers). Note the search results seem to be copies of those PyPy scripts, as far as I can tell.
  • ...which, given the above, nobody is terribly interested in maintaining
  • ...yet you ended up generating nothing but noise for hundreds of thousands of pytest users, which for historical reasons depends on pylib (since it came from the PyPy project, like pylib does). Yes, not everyone of the half quarter a million projects there will monitor CVEs against dependencies, but at the same time, lots of people that do (in companies and such) are probably not in that list on GitHub.

Given that it will still take some time for pytest to get fully rid of it's py internals, and people seem to like getting CVEs for this project (which just happens to be very popular via pytest, but pretty much unused outside of pytest), but without really making an attempt to understanding the context... can we please consider:

  • Either vendoring the remaining code (py.path I assume) in pytest, and archiving pylib...
  • ...or releasing a pylib 2.0.0 which simply drops all the code that isn't used in pytest?

Sorry if I sound frustrated. But requesting a CVE without understanding the context of the project/issue you're reporting, and then generating false reports for hundreds of thousands of pytest users is doing a major disservice to both pytest/pylib maintainers and all those users.

@RonnyPfannschmidt
Copy link
Member

I'd go as far as to label this cve report a supply chain attack

Pytest should keep dropping pylib as it does

The cve should be added as common false positive

@The-Compiler
Copy link
Member

I'd go as far as to label this cve report a supply chain attack

I would not, but I'd certainly add it to my "examples of behavior causing open source maintainer burnout" list, given that we're now getting the first pytest issue about this, and it almost certainly won't be the last one.

@bluetech
Copy link
Member

bluetech commented Oct 19, 2022 via email

@RonnyPfannschmidt
Copy link
Member

@bluetech @The-Compiler i woudnt mind to cut a release that drops the svn wc stuff alltogether

@The-Compiler
Copy link
Member

@RonnyPfannschmidt agreed, and perhaps a bit more even - I opened #288 with some overview on that.

@The-Compiler
Copy link
Member

FWIW, I've also proposed adding a note to the GitHub advisory (github/advisory-database#761) and tweeted a PSA.

@The-Compiler
Copy link
Member

GitHub have now reacted and amended the advisory:

The particular codepath in question is the regular expression at py._path.svnurl.InfoSvnCommand.lspattern and is only relevant when dealing with subversion (svn) projects. Notablely the codepath is not used in the popular pytest project.

and apparently also added that information for Dependabot alerts:

I've also added the codepath to the advisory so that dependabot can more intelligently target alerts.

@woodruffw
Copy link

pip-audit maintainer here. I have no say in it, but it's a shame (in my personal opinion) that this kind of low-quality finding was assigned a CVE ID without any significant cross-checking.

I've filed a CVE rejection request with MITRE, since they're the CVE CNA in this case. If they successfully reject it, getting it removed from GHSA should also be easy.

@woodruffw
Copy link

Actually, looks like I'll be able to propose a GHSA withdrawal even if the CVE itself hasn't been retracted. I'll open a PR for that in a moment.

@woodruffw
Copy link

I've filed github/advisory-database#762 to mark the GHSA report as withdrawn.

@jenstroeger
Copy link

jenstroeger commented Oct 20, 2022

@The-Compiler agreeable and funny #287 (comment) 😁

@woodruffw thanks for following up on the advisory! That means that pip-audit won’t pick up that CVE any longer once pypa/pip-audit#385 has merged, correct?

@woodruffw
Copy link

@woodruffw thanks for following up on the advisory! That means that pip-audit won’t pick up that CVE any longer once pypa/pip-audit#385 has merged, correct?

Once that's merged and the OSV entry is marked as withdrawn, yeah.

@Denelvo
Copy link

Denelvo commented Oct 21, 2022

@The-Compiler @bluetech @woodruffw & other maintainers: don't sweat it! Even though 18 years ago, someone wrote a regex, not thinking of 2022-levels of security paranoia, your efforts are still very much appreciated and your contributions are incredibly productive.

The CVE did trigger some alarms in builds of valuable systems used by financial institutions. But that's good. It doesn't necessarily mean that pytest is suddenly unusable. It just forces downstream developers in highly secured places to stop and think. Alerts can be ignored, assessed, categorized, accepted,...

Please look at CVEs as mostly just a catalyst. Having the CVE removed is a harsh reaction, but the right one if we're absolutely sure that that particular regex is not exploitable. But try to think of it from the point of view of a security officer, with full paranoia goggles on (and no intimate knowledge of the module). The regex is present in the code. It could be executed. A hacker could penetrate, set up a SVN repo, craft some commands, take advantage of other existing vulnerabilities on the system,... You have to think worst case and then some.

In the end, I think a correctly worded CVE with a low severity score which clearly includes all the IFs, would have been the best option here.

Thanks!

@The-Compiler
Copy link
Member

Taking the freedom to quote what I wrote over at github/advisory-database#762 (comment):

but to my eye the advisory is in fact talking about a real redos vulnerability. I do understand the annoyance with what you call CVE spam, but if the advisory is valid then we want to include it in our data set.

In theory, when viewed in a vacuum: indeed.

However, even a very broad search on GitHub for the affected code yields 92 results. I've looked through them all, and from what I can tell, all the matches fall into one of those categories:

  • Copies of py using it (in svnurl(), which the search captures as well)
  • Copies of pypy (where the py library originates from historically), in it's development tools directory. However, PyPy has moved away from SVN in 2010, so they're probably just bitrotting. In any case, they use the repository the current file is in, or are hardcoded to run on a specific old PyPy team server (codespeak.net). If someone has control over that, they might as well just change those scripts. Even if there are certain scripts which actually run on arbitrary SVN servers, the context they are run in is unlikely to make a DoS a real problem. But given that they won't actually work anymore for their intended original purpose, the chance of anyone running them is pretty much zero.

On the other side of the coin, you have at least half a million projects depending on pylib via pytest, which got noise in their inbox.

I believe the main point of a CVE and security advisories is to make people aware of problems which have a real (even if small) chance of affecting them. An advisory which just adds noise to what I believe will be 100.0% of the receivers is just going to hurt the whole system and community.

In this particular case, and when viewed in context, the sheer amount of noise the advisory generates vs. its real usage is so high that the word "spam" is unfortunately very fitting. I can't help but think that "oh, I get a shiny CVE in a popular project" was the only motivation behind it.

@woodruffw
Copy link

Just to make sure there's no confusion: I'm not a maintainer of this project or of pytest, so my opinions are not those of the maintainers. I'm the maintainer of a separate dependency scanning project, so I have an interest in dependency feeds having a high signal-to-noise ratio. After this comment, I'll step out of this thread now (since I'm not a maintainer and I only stepped in to coordinate on the pip-audit side).

With that being said: I disagree that it's good that a CVE was filed for this behavior. The fact that one was filed and published seemingly without any review represents a series of communication and authority breakdowns; the fact that the project's own maintainers have done more investigation into exploitability potential than the original reporter seems to have is an indicator of this.

As for why that matters: not everybody is a bank, with roles dedicated to reviewing a constant deluge of security reports. In the context of more limited resources, managing security fatigue is far more important than reporting weakness classes like ReDoS, which don't manifest as exploitable vulnerabilities in the overwhelming majority of cases. When users get tired of useless reports, they disable their security tools entirely.

@SCH227
Copy link
Author

SCH227 commented Oct 24, 2022

First of all, I did fill for a CVE, but I didn't publish it. Someone else did it. I find illogical it was you, but the GH advisory says "Credits - @The-Compiler".
The other thing I can think of, is that GH made it automatically because this is described in a public issue. But I find it strange, because I already reported other security issues to MITRE and they never got published before I informed they were made public

@mfilenko
Copy link

mfilenko commented Dec 2, 2022

Hey @hotenov,

Just curious where does 51457 come from?

Adding 2022-42969 (CVE ID) to my pipenv check --ignore didn't help, but 51457 did 😃

Asking for future references 🙈

@hotenov
Copy link

hotenov commented Dec 2, 2022

Hey @hotenov,

Just curious where does 51457 come from?

Adding 2022-42969 (CVE ID) to my pipenv check --ignore didn't help, but 51457 did 😃

Asking for future references 🙈

Hello, @mfilenko I stole it from previous reference :) by KeNaCo
But you can take ID from safety report:

image

huang06 added a commit to huang06/python-project-template that referenced this issue Dec 4, 2022
lordkyzr added a commit to iovation/launchkey-python that referenced this issue Dec 8, 2022
lordkyzr added a commit to iovation/launchkey-python that referenced this issue Dec 9, 2022
malor added a commit to malor/cpython-lldb that referenced this issue Dec 12, 2022
There was a vulnerability in py <= 1.11.0, which does not really
matter for cpython-lldb, but dependabot was unhappy.

See pytest-dev/py#287 (comment)
for details.
malor added a commit to malor/cpython-lldb that referenced this issue Dec 12, 2022
There was a vulnerability in py <= 1.11.0, which does not really
matter for cpython-lldb, but dependabot was unhappy.

See pytest-dev/py#287 (comment)
for details.
malor added a commit to malor/cpython-lldb that referenced this issue Dec 12, 2022
There was a vulnerability in py <= 1.11.0, which does not really
matter for cpython-lldb, but dependabot was unhappy.

See pytest-dev/py#287 (comment)
for details.
malor added a commit to malor/cpython-lldb that referenced this issue Dec 12, 2022
There was a vulnerability in py <= 1.11.0, which does not really
matter for cpython-lldb, but dependabot was unhappy.

See pytest-dev/py#287 (comment)
for details.
malor added a commit to malor/cpython-lldb that referenced this issue Dec 12, 2022
There was a vulnerability in py <= 1.11.0, which does not really
matter for cpython-lldb, but dependabot was unhappy.

See pytest-dev/py#287 (comment)
for details.
waynesun09 added a commit to RedHatQE/notify-service that referenced this issue Jan 4, 2023
Fix security issue:
pytest-dev/py#287

Signed-off-by: Wayne Sun <gsun@redhat.com>
braffes pushed a commit to braffes/reportseff that referenced this issue Jan 9, 2023
pdodds added a commit to kodexa-ai/kodexa that referenced this issue Jan 13, 2023
…onduct a ReDoS (Regular expression Denial of Service) attack via a Subversion repository with crafted info data, because the InfoSvnCommand argument is mishandled.

The particular codepath in question is the regular expression at py._path.svnurl.InfoSvnCommand.lspattern and is only relevant when dealing with subversion (svn) projects. Notably the codepath is not used in the popular pytest project. The developers of the pytest package have released version 7.2.0 which removes their dependency on py. Users of pytest seeing alerts relating to this advisory may update to version 7.2.0 of pytest to resolve this issue. See pytest-dev/py#287 (comment) for additional context.
picciama added a commit to wilhelm-lab/spectrum_fundamentals that referenced this issue Jan 22, 2023
@ret2libc
Copy link

Is the POC in the description actually working for anyone?

>>> from py._path.svnurl import InfoSvnCommand
>>> payl = "   2256      hpk        165 Nov 24 17:55 __init__.py" + " " * 5000
>>> InfoSvnCommand(payl)

For me it executes everything in a fraction of a second.

@RonnyPfannschmidt
Copy link
Member

It's thinkable that recent python versions have complexity enhancements

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests