Cryptography 3.3 LTS release without Rust bindings #5799
Comments
|
The primary challenge I see for this is keeping CI functional. Since we don't need to maintain a functioning wheel build system for this the problem is simpler, but, for two examples, we'd probably also want to freeze the docker containers on a specific tag and pin our test dependencies (which we currently leave open deliberately). 3.3.x was also the last release we supported Python 2 on so the test infra would need to maintain the ability to test against that. I agree we need an official end date if we do this, but I'm not sure what the right date is for it. |
|
My default disposition is strongly anti-LTS. I think they tend to add a lot of maintenance burden for the folks who maintain them, as well as other OSS maintainers who get demands to support older versions of things as a result. Indeed, @reaperhulk and I discussed doing an LTS a year ago for Python2 support, and concluded it was simply easier to support Python2 for an additional year rather than to support a separate LTS branch. I also have philosophical concerns with the LTS theory of security: fixing vulnerabilities isn't the only way to improve security. Hardening, detection, privsep, etc. contribute much more to security (https://alexgaynor.net/2018/jul/20/worst-truism-in-infosec/). Just because you have all known bugs fixed doesn't make you equally secure: https://alexgaynor.net/2019/mar/07/chrome-windows-exploit-security-beyond-bugfixes/ There's also a practical concern: If we do source only releases, this will work well for Linux distribution maintainers who do their own builds, but wreak havoc on folks who install from PyPI. An alternate solution might be to simply not post to PyPI, but now we're in seriously strange territory. I'm still open to being persuaded, but this is where I'm at now. |
|
@alex The target audience for an extended security fix release would be users that download sources from PyPI. Linux distribution package maintainers typically cherry-pick and backport security fixes. For example I maintain internal forks of cryptography for RHEL. The RHEL 8.4 rebase to 3.2.1 contain backport of CVE-2020-36242, compatibility with old pytest, NPN APIs for older PyOpenSSL, @reaperhulk |
|
Python 2 isn't getting security fixes. I don't really have any stake in this, but I'd consider it reasonable if the 3.3.x branch didn't have Python 2. |
|
Alpine has deprecated Python 2, so we have no interest in LTS for that reason. Rust has a lot of problems for musl distributions (the default targets in rustc force static linking for example), so we have to define new targets, which makes getting rust everywhere problematic (we still lack rust for 2 release archs due to being unable to build cross compilers). I suspect that there are other scenarios where upgrading or introducing Rust into an already released distribution is problematic. Regarding commitment level, we would be glad to maintain the LTS ourselves without @alex having to spend time on it, other than his notifying us about things that need to be backported. |
|
If your definition of "needs backport" is "has a CVE", then we'll continue to issue CVEs for any vulnerability, and we'll make sure the version metadata is accurate. Do y'all have a capacity to automatically pull in CVEs, or do you need an explicit notification process? |
|
As long as there is CVEs listed in the git history, we can handle backporting anything we need, even if it means fixing the C code based on changes done to the equivalent Rust code. I don't think Alpine is alone in this situation though, I suspect that CentOS/RHEL 8 is unlikely to ever get a new enough Rust, for example, but I don't know their policies. edit: apparently CentOS does have a new enough rust. I guess that's what appstreams are about :) |
|
We don't consistently put CVEs in our git logs at the moment, but there's no real reason we couldn't make that part of our security response policy. |
|
Right, we put them in the changelog (and the CVE database, obviously). I
don't see why we couldn't put them in git though!
…On Tue, Feb 9, 2021 at 5:50 PM Paul Kehrer ***@***.***> wrote:
We don't consistently put CVEs in our git logs at the moment, but there's
no real reason we couldn't make that part of our security response policy.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#5799 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBEOSCNXQRTWJAD23GLS6G3ZVANCNFSM4XK7UDDQ>
.
--
All that is necessary for evil to succeed is for good people to do nothing.
|
Yeah, RHEL 8.3 and CentOS 8.3-2011 have rustc-1.45. It's totally feasible to rebase the RHEL 8 python-cryptography package to a future version with Rust. I have most pieces in place. |
|
I'm really confused. How could the introduction of rust bindings ever be considered a minor version when there is no backwards compatibility. This specifically breaks established spec: https://semver.org/ Just asking because I'm sure this could have been handled a bit cleaner with a bump to 4.0.0. |
|
Take a look at #5801 where we discuss this very question. (Spoiler: it's not at all clear this meets the semver standard for a major release at all.) |
|
Well, thank you for your efforts. I have no intention of tell you what you should be doing. I do agree with your arguments against C and the direction towards rust for future use. It was just a bit disruptive unfortunately. I do think a 3.3.x would be beneficial. |
This change is temporary waiting for Alpine support or even the 3.3 branch LTS [1]. [1]: pyca/cryptography#5799
|
#5819 documents that we will now put CVEs in git commits in addition to our previous policy of the changelog and a GitHub security advisory. Having read through this thread several times now I don't see a particularly good solution for us (as in PyCA) to maintain this. If we release source only to PyPI then the first source only release will cause significant breakage (antithetical to the concept of an LTS), but it's unreasonable for us to maintain our infrastructure for wheel building on this branch. That leaves just keeping a branch updated and not uploading releases from it (but possibly tagging them?) which, as @alex previously noted, is a very strange thing to do. At the moment my inclination would be to be very explicit in the ways #5819 commits to and maintain open lines of communication with any distribution packagers who have additional questions in the event that a backport of a CVE fix becomes necessary, but not to do any new releases on the 3.3.x branch. I'd be interested in hearing from packagers (both distribution level or others who package for more than just themselves) about whether they view this as a functional path forward while they work out ecosystem challenges around Rust. |
This change is temporary waiting for Alpine support or even the 3.3 branch LTS [1]. [1]: pyca/cryptography#5799
|
Neither @kaniini nor I are in need of an LTS branch for our packaging work. #5819 makes cherry-picking and backporting of security fixes even easier. No other packager has raised an interest in a LTS branch either. You don't want to do an LTS branch for good reasons. I propose we wait a couple of days to give other packagers a chance to speak up, then close the ticket. |
|
Hi, Sascha here from Psono. First thanks for all your work. Psono's backend heavily relies on it and you are doing a great job! I am using alpine as a base image for my docker containers and would love to switch to the new Rust'ed version, yet without proper wheels the upgrade is not "easy". I am sure that others previously supported OS / Architectures feel the same :( Maybe you could support the 3.3 Version till you re-established "normal" pip install for others? |
|
The new Rust dependencies have kickstarted an effort to define manylinux binary weels for musl-based distros like Alpine, https://discuss.python.org/t/wheels-for-musl-alpine/7084 The next release of cryptography will work with older Rust on Alpine 3.12. You can find more information in #5823 and PyO3/pyo3#1420. |
|
Hi - I'm one of the Python/cryptography maintainers for OpenWrt. There is ongoing work to bring the Rust toolchain into our build process (openwrt/packages#13916) but there is no time frame for when it may be ready. Asking our users to install cryptography from pip would not be ideal (we package cryptography because it is a dependency of other OpenWrt packages) and may not be entirely possible given the different architectures we support. Having an LTS branch would be helpful during this time 🙏 |
|
Looking at the openwrt discussion, they make the point that if they switch to 3.4 using CRYPTOGRAPHY_DONT_BUILD_RUST today, they might have to downgrade to 3.3 when 3.5 comes out (because 3.4 would be be EOL but 3.3 would still get LTS support). Would it make sense to select 3.4 as the LTS release ? Seems to me (outsider view warning) it would be the same amount of work on the pyca side, it would avoid the "3.4 with workaround today then downgrade to 3.3" conundrum, it would push the EOL date for the rust-free pyca further, and it would expose more users to the fact the Rust is coming. |
|
If you read this comment, you'll see that the current plan is to not provide an LTS release at all, but to clearly publish all CVEs and the associated fixes, so that maintainers of downstream packages can easily backport the fixes. For OpenWRT, this might be the best approach: Using |
To me the current plan appears to be to allow packagers to voice their opinion before making the final decision. If upstream doesn't want to maintain an LTS branch, then downstream projects that need one will have to maintain their own branches. This sounds like a duplication of effort to me, or maybe OpenWrt is the only project in this situation. We are all busy people - I suggest waiting for more than a couple of days for packagers to find this ticket. |
|
I was having issues installing Cryptography while installing Ansible on WSL2 Ubuntu 18.04 using
Upgrading pip through |
|
This rust dependency is blowing my mind. That might bring me to swear off the entire Python ecosystem. Who thought it was a good idea to introduce a hard dependency on an entirely optional language environment? I know dependency management is hard, I've given lectures on that. But not requiring stuff that is unlikely to be present is sort of lesson one. I respect that you guys want to use some kind of Rust backend, but make the build dependent on actually finding the prerequisites, please. This is just ridiculous. And I'm being very polite when use those words. It basically breaks Python. Not because Python needs this package, but because any reasonably complex application does. You can be experimental and shit in a hobby project, but not in something that approaches infrastructure. And an LTS release is entirely the wrong approach, for what it's worth. That implies that sooner or later everyone must pull the Rust ecosystem in for running Python. Boggles the mind. |
|
Yes, considering the Python 2 EOL has completely wiped out the use of that old version, you're probably right. |
a) It's not a hard dep in 3.4, there's an env var to disale it.
Yup, you've got it. As I've said many many many times, we're always eager for feedback on how we can do this better, whether it's communications, tooling, compatibility, etc. What we're not interested in is feedback that we shouldn't try to address language-level memory safety at all. The scourge of C and C++ on computer security is simply too great for me to countenance walking away from this work. |
Daniel's latest blog post on curl's security is a good data point. Half of curl's vulnerabilities are C mistakes. It's only half because Daniel is a very experienced developer and the project utilizes several code analyzers. |
|
Well, I won't try to argue something I obviously won't win, but that kills Python for a vast variety of applications. Sad, really, but that's out of my hands. I had a good 20 years with it, time to move on. |
|
You haven't really explained why this kills Python for you, other than
that you don't want Rust involved. So it's hard for me to provide any
sort of meaningful feedback, other than to wish you the best of luck
in whatever language you move to.
…On Wed, Mar 10, 2021 at 11:29 AM Jens Finkhaeuser ***@***.***> wrote:
Well, I won't try to argue something I obviously won't win, but that kills Python for a vast variety of applications.
Sad, really, but that's out of my hands. I had a good 20 years with it, time to move on.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
@alex, I was about to apologize for the abusive language of my fellow countryman. But then I noticed that he is Bavarian and not German. :) Anyway, Github Support is notified. |
|
I'm not sure what rose to the level of abuse here unless there was a deleted comment. It was rude and entitled but nothing worthy of a report. However, I don't think this issue is being productive any more so I'm going to close and lock it. The final decision is here is that we will not be releasing an LTS branch, but for any security fixes we will be providing more metadata in the commit messages so downstream packagers can cherry-pick them far more easily. |
@kaniini proposed the creation of a PyCA Cryptography LTS release branch. The LTS branch would give users more time to adjust their infrastructure for Rust requirements. Alpine and musl libc maintainers might even get manymusl binary wheels ready in time. I support their proposal and would be interested to assist with and use a LTS release in RHEL 8 and CentOS.
To reduce maintenance overhead and to set user expectations, a long term support branch should be clearly scoped. Therefore I propose
This proposal should keep the workload small and cover majority of users.
10 months until EOL should be sufficient for majority of users to get Rust infrastructure in place. The next stable Debian 11 (Bullseye) should be released roughly half a year before the EOL date. Latest Alpine, CentOS 8, Fedora, and Ubuntu already have recent enough Rust.
The text was updated successfully, but these errors were encountered: