urllib3>=2.0 does not work with system Python on macOS

[andy-maier]

Subject

The system Python on macOS 12.6.5 is Python 3.9.6 whose ssl module is compiled with LibreSSL 2.8.3.

The removal of LibreSSL support in urllib3 2.0 makes it impossible to be used with the system Python on macOS.

What are you recommending to macOS users that do not want to or are not allowed to use a Homebrew version of Python?

In our Python packages that use requests / urllib3, we will now either have to pin urllib3 to <2.0, or have to have a trouble shooting documentation for how to deal with that.

At the least, I think urllib3 should add some trouble shooting info on how to find out what the Python ssl module is compiled with, and the options to deal with the issue.

Note that issue #2168 misses one important case: The user created a virtualenv based on the system Python. Using a virtualenv makes it possible and reasonable to install packages from Pypi. That is how a macOS user gets into the problem of using a Python that was compiled with LibreSSL and uses urllib3>=2.0.

That is the case that issue describes as "The combination of the above three is very unlikely.", but that is exactly how it happens and I don't think it is unlikely. There is no rule (that I know of) that prevents or discourages from using the system Python for a virtualenv, and then to install packages from Pypi.

[sigmavirus24]

While true, CPython is removing support for compiling against Libressl so it's unlikely to be the case long term.

[andy-maier]

Note that PEP-644 lists macOS in its OpenSSL section as "macOS (python.org installer)" and does not show the system Python provided in macOS at all, which uses LibreSSL up to Ventura. IMO, that PEP should have listed "macOS (system Python)" in its LibreSSL section as well.

Considering the use of the system Python on macOS, the statement in PEP-644 "No macOS and Windows user will be affected by the deprecation." is simply not true. There are environments where you simply don't have the choice to install another Python version.

So if urllib3 refers to PEP-644, it should also be aware of the fact that that PEP seems to ignore the use of the system Python on macOS.

If I look at the situation from an end user perspective, then the PSF apparently ignores what Apple does, and urllib3 follows that and also ignores it. That is not helpful for Python users on macOS.

If you take a look at #2168, then the meanwhile larger part of its comments (at least when measured as vertical screen size) are from the last 4 days where users started to realize the incompatibilities.

[sigmavirus24]

The system python on macOS is actively recommended against by most resources I'm aware. It's there for macOS tooling and development (software developed by Apple).I believe the PSF has no sway over language decisions so you want to blame the Steering Council for accepting the PEP. (Although, really wildly accusing other humans who volunteer to work on a project like Python of malicious intent or incompetence is really shitty behavior regardless of whether you address the right group or not.)

The reality is that Python offers dmg files for macOS users that are complied against openssl and there are other options like asdf(, pyenv, etc.) for local user installation and homebrew.

If you're looking to blame someone for incompatibilities, look to the Libressl project. They forked hard. Between Google's boringssl, Libressl, and openssl the APIs are too different for one person (the only person willing to maintain ssl) who does the maintenance entirely in their free time. (At least that's my guess - that it's too much to be able to be confident in the secure usage of the various APIs).

It's not a great place to be, I agree, but regardless of whether this library checks that or not, Libressl is not going to be well supported and likely not trustable within Python on any platform that compiles against it. There will be patches used to make it compile and likely those will be carried forward until something makes it hard to get those to work or a CVE is found in the fragile integration.

[andy-maier]

Although, really wildly accusing other humans who volunteer to work on a project like Python of malicious intent or incompetence is really shitty behavior regardless of whether you address the right group or not.

I did not intend to accuse anyone (and don't think I did), and particularly not the great folks that lead the development of Python. I just pointed out a particular single-sidedness of that PEP w.r.t. macOS, where it seems to assume that people don't use the system Python, while the PEP does not do that for any of the other OSs listed there.

[andy-maier]

I have the same feelings about LibreSSL as you expressed, but with Apple choosing that library, the situation has become more difficult. That decision has given LibreSSL a certain weight.

[andy-maier]

Do you have a link to where Apple states that the system Python in macOS has the restricted purpose you stated above?

[andy-maier]

I think I'm not really arguing for python or urllib3 supporting LibreSSL.

But the problem this causes for macOS users should be more appreciated, IMO. That's why I suggested adding trouble shooting info.

[sigmavirus24]

Do you have a link to where Apple states that the system Python in macOS has the restricted purpose you stated above?

Apple doesn't state that explicitly but if you file a radar about something with system python and your development against it, they generally close those (is what I've heard over the last decade of being involved in the community). I've never filed a radar so I can't speak to it myself.

Apple choosing that library, the situation has become more difficult. That decision has given LibreSSL a certain weight.

I agree it's definitely complicates a lot of things. That said, Apple also has hard forked bash to 3.x because 4.x adopted GPLv3 iirc and so Apple's built-in software isn't designed for users in these cases but to protect Apple from legal issues. Previously the openssl that was on macOS was ancient and heavily patched by Apple. There used to be guides that recommended brew installing a newer openssl and python to avoid the problems with it (no SNI, old cipher suites, iirc).

I'm on my phone so it is far more difficult to compile a list of citations for you, but I don't believe the community advice should be that difficult to find either. It was in most books, I think even some of the libraries had documentation around this. I think the python-guide recommended (possibly still does) against system python.

[sigmavirus24]

But the problem this causes for macOS users should be more appreciated, IMO

I agree. I think there are discussions happening around this. The difficult bit is that we want to be confident in the security we provide for users and we rely on ssl for that.

I think that the SC probably discussed this issue but I as someone who noticed the PR, didn't think of macOS users (even though I use it at work) I suspect for the same reason that other folks didn't - most of us use non-system python.

[sthen]

While true, CPython is removing support for compiling against Libressl so it's unlikely to be the case long term.

That's not what was proposed in the OpenSSL PEP and it is not what has happened. CPython is no longer adding workarounds for libressl-related things (so a passive "not going out of its way to support it") but the PEP clearly says that it's not taking an active step of rejecting compiling against LibreSSL.

Specifically, what I see that's been changed in 3.10 in this area is removal of what are mostly workarounds for obsolete LibreSSL versions that are no longer useful anyway (and may even get in the way - we've found that to be the case fairly often with other software).

[pquentin]

Thanks for your excellent points. We might revisit our decision, but need to investigate more. That said, I wanted to answer to one specific question.

Do you have a link to where Apple states that the system Python in macOS has the restricted purpose you stated above?

Yes, see the macOS 10.15 release notes: https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes#Scripting-Language-Runtimes

Scripting Language Runtimes

Deprecations

• Scripting language runtimes such as Python, Ruby, and Perl are included in macOS for compatibility with legacy software. Future versions of macOS won’t include scripting language runtimes by default, and might require you to install additional packages. If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. (49764202)
• Use of Python 2.7 isn’t recommended as this version is included in macOS for compatibility with legacy software. Future versions of macOS won’t include Python 2.7. Instead, it’s recommended that you run python3 from within Terminal. (51097165)

Python 2.7 was then officially deprecated in macOS 12.0 and removed in macOS 12.3. python/cpython#95284 acknowledged this.

There's no more "system Python": Python 3 is now installed only if you install the Xcode Command Line Tools, and is meant for LLDB scripting, as described in the LLVM docs and the Xcode 13 release notes which state:

• LLDB’s Python scripting no longer supports Python 2. (73956573)

[pquentin]

Even though the "system Python" is only installed with developer tools and is only meant for scripting a debugger when using Xcode, I realize this is hurting users.

We mentioned that we were following the lead of PEP 644, but:

• I don't think this Xcode Python existed when the PEP was written in October 2020?
• The wording in the PEP is harsh, but even CPython is not actively dropping LibreSSL, see gh-89532: Remove LibreSSL workarounds python/cpython#28728 for example. (Patches are still needed on Gentoo though: https://github.com/gentoo/libressl/tree/master/dev-lang/python/files)
• Unlike Gentoo which already patched urllib3, users hitting the LibreSSL issue on macOS are often new to Python, and this is not welcoming.
• cryptography dropped support for OpenSSL<1.1.1, but they still actively support LibreSSL

Should we open the door to LibreSSL? One issue is that LibreSSL 2.8.3 was released in December 2018 and has been EOL since December 2019. Still, my opinion is that requiring OpenSSL is harsh and a warning should be enough for users that are not using OpenSSL for now.

@sethmlarson @sigmavirus24 What do you think?

[sigmavirus24]

I am on board with relaxing the restriction. I'm only worried about debugging Libressl problems if they come up when it's not something we can easily support or test against and the support provider will likely be per-downstream.

What do folks think about something like what requests has to provide additional debugging info in the bug template?

[pquentin]

What do folks think about something like what requests has to provide additional debugging info in the bug template?

Good idea. requests has https://github.com/psf/requests/blob/main/.github/ISSUE_TEMPLATE/Bug_report.md which asks to run python -m request.help which is quite comprehensive, but we have https://github.com/urllib3/urllib3/blob/main/.github/ISSUE_TEMPLATE/02_bug_report.md and I can extend it to print ssl.OPENSSL_VERSION which gives OpenSSL 3.0.8 7 Feb 2023 on my Linux laptop and LibreSSL 2.8.3 on my Macbook.

[sethmlarson]

I'm also on board with relaxing the restriction to allow any version of LibreSSL as long as we explicitly emit a warning that LibreSSL isn't supported and continue to raise an error for non-OpenSSL/LibreSSL implementations. The unfortunate thing is I'm unsure how we are going to test that whatever we've done is at least correct initially for us to feel comfortable releasing it at all, anyone have ideas how to get an authentic Python build that uses LibreSSL?

[mistermocha]

Hey I came across this, and I do the bulk of my work in virtual environments. My newly created venv picks up the ssl module from whatever-the-system-ssl-module is.

It took lots of digging to figure out that if I install openssl1.1.1 from homebrew, blow away my bin folder in my venv, then recreate my venv entirely, it links the ssl package properly.

I'm happy to document the process for those that are having trouble if someone can point me to the right place for it,

[sigmavirus24]

@mistermocha thanks for the offer! I don't believe that documentation belongs in urllib3. If you're already using homebrew, I would just install Python from that, not try to have openssl work with XCode Python. There are other opinionated guides like https://docs.python-guide.org/starting/install3/osx/#install3-osx that exist for this express purpose. Alternatively, you could write a blog post about this. Documentation isn't just in the official repository for any open source project. Sometimes the best documentation is someone describing how they solved a problem in their own words on their own blog

[mistermocha]

I understand. This was the first big break that happened for me as a result of the upgrade. I'll work on my own post somewhere so that it helps someone, because I'm sure there's another person who's having the same problem.

[timsutton]

I don't think this Xcode Python existed when the PEP was written in October 2020?

It was: Apple had begun shipping a Python 3 linked against LibreSSL with their Command Line Tools and Xcode when Catalina was released in October 2019, first previewed in June 2019.

I re-watched a section of this Python-on-macOS talk I did a couple years ago and it seems like its linked LibreSSL version was at the same 2.8.3 that it is today :/

[sethmlarson]

@hoochanlon I've hidden your comment since it's actively harmful for folks to follow it, people should not pin to an old version of urllib3 (like 1.26.0 which is what you suggested) instead people should pin to urllib3<2 if they are unable to upgrade to urllib3 v2.x in order to always have the latest version of 1.26.x to receive security fixes.

[aleksei-dereviankin]

Macports version 3.9 compiled with openssl and works properly. install macports and use python from it, working on my m2

[pquentin]

This is not related to the architecture. Most likely, you're using the Python provided by the Xcode Command Line Tools on your M1, and another Python (maybe provided by Homebrew?) on your Intel Mac.

[sethmlarson]

Locking this thread as we've already relaxed the restriction from an error to a warning in the latest versions of urllib3 2.x. We won't be removing the warning, it's up to users to either move to a Python compiled with OpenSSL or be okay with running in an untested configuration. If you're seeing an error and accept the risks of running an untested configuration then you can upgrade to the latest urllib3.

Issues that aren't related to the above should be opened separately.