Challenges and future issues

[bugcrazy]

Void Linux is thinking of switching to openssl, as described here void-linux/void-packages#20935 and another one had already switched to openssl, to be able to build on different architectures, described here ClickHouse/ClickHouse#8218, the lack of algorithms and features that other *ssl uses are detrimental to the adoption of libressl.

The project needs to have more features that make it more competitive, so it will be adopted by other projects.

I suggest that it be used and adopted ASM, so that libressl can be built on several architectures, it already has code ready for libressl, here libressl-portable-asm, these are basic things that many projects depend on!

[RoundDuckKira-deprecated]

A Void user here and maintainer-ish of the PikoPixel package in Void, so I guess I'm an above average dumb dumb here... EDIT: unless if the average is a programmer and thus then I'm really big dumb dumb :p

But yeah Void is thinking of moving away, and for so many good reasons. Personally I'm a bit conflicted as it was one of the defining elements of Void and helped make the distro even more unique among GNU/Linux distros, but it does go to show the issues with the LibreSSL project and what they should do to win users back, before even more distros leave the project.

I know it's their project and LibreSSL is for OpenBSD by OpenBSD for the sake of OpenBSD but they need to realize their biggest success is not their core OS but their projects that emanate from the OS and are available on so many other projects. And with behavior like no LTS release, slow response for bugs not a part of the core OpenBSD base system (so not even of their ports), their irrationalism about sticking strictly to the ISC rather than going to Apache 2 and thus being able to be compatible with OpenSSL, and the lack of ASM optimizations (especially considering a big goal of Void is portability), no wonder distros didn't adopt it and Void, one of the biggest cheerleaders, is (edit: almost certainly, if you look at the proposal) leaving. KISS will now be the most prominent distro with LibreSSL instead of Void.

I really hope devs listen about this and work with Void and so on to encourage them not to leave LibreSSL and instead work with them, but then again Theo probably won't listen to the point about licensing, and maybe shouldn't anyways if there's a way to get clout with software developers to support LibreSSL and still fix the other issues. :p

ehh just two cents from some random doofus

[RoundDuckKira-deprecated]

Also, umm, libressl-portable-asm IS developed by a Void dev btw, so even despite them wanting to leave LibreSSL, they may still stick with it if LibreSSL were to help them out, and at least make the ASMs officially supported. I mean, OpenBSD is on ARM 64 right? 🤷‍♀️

[busterb]

Howdy, thanks for the pointer to the patch tree. Was somebody reaching out before this issue was raised? Just curious why nobody submitted any of the new ASM PRs before. I'm pretty sure we'd be happy to integrate them for whatever architectures folks wanted, and this is pretty close to a PR. Are we that scary :) ?

[RoundDuckKira-deprecated]

@busterb
are ye one of the fabled three devs behind portable LibreSSL? :p

[RoundDuckKira-deprecated]

jokes aside this is actually nice to know, and I wonder if Void did reach out or something. Maybe they also felt that with the focus of things and so on that they felt like they were pressuring the project? I'm just putting in my guesses, not too sure overall.

[RoundDuckKira-deprecated]

ah umm, well the problem still is the other issues are still major pain points that I talked about, which is why q66 didn't go over here. It wasn't just a stopgap patch (that I feel would be nicely maintained in LibreSSL but I digress) but also that Void wanted better OpenSSL 1.1+ compatibility, as so many open source programs are still only built for OpenSSL. I know it's possible and recent updates worked to this but the core issue is that LibreSSL can't just easily import stuff from modern OpenSSL for licensey reasons, and has to kinda reimplement them instead from scratch I'd guess 🤷‍♀️

[RoundDuckKira-deprecated]

@busterb just look into the discussion about the topic, it does a better job explaining than I ever could, heh

[RoundDuckKira-deprecated]

there's also mention about ASMs in x86 and other arches that are missing too since LibreSSL either reduced them or didn't import newer OpenSSL ones, that's another thing to mention

[q66]

Well, fine, since this was already raised here, might as well make an effortpost to clarify the whole situation, so that people don't just talk about it randomly without having the whole picture.

Void's concerns

1. We have a relatively significant patch burden right now, especially with e.g. Rust projects using openssl-sys, also Qt5, and e.g. Python 3's hashlib having a reduced API when compiled against pre-1.1 OpenSSL APIs, which has already been problematic in practice (e.g. scrypt is missing with it); this will only get worse over time, as projects drop support for OpenSSL 1.0 API
2. LibreSSL's ABI breakages make things harder for us as well - it's a significant repository rebuild every update where that happens
3. Overall it's becoming an increasingly hard sell for us, considering how much OpenSSL has improved in recent time. While it makes sense for OpenBSD to continue maintaining LibreSSL with OpenSSL moving license (and Apache 2.0 being incompatible with OpenBSD), this is a non-concern for us
4. However, it does mean that LibreSSL will no longer be able to backport any changes, and the development pace already seems fairly slow in comparison to OpenSSL, and that is concerning for us; on the other hand, OpenSSL is getting more attention than ever, has a larger test suite and to my knowledge undergoes more auditing
5. In addition to that, the portable project gets even less attention and has its own specific bugs (e.g. build system and __STRICT_ALIGNMENT #602)
6. The missing assembly files are a problem, yes, especially as we cover a significant number of architectures (some of which aren't even covered by OpenBSD itself; e.g. their powerpc64 port is just starting out and I'm fairly sure it's limited to big endian at this point). My project already linked here addresses some of that, but not all, since a bunch of the optimized code is missing even on x86_64, and I only aimed to bring things to parity with the x86_64 baseline (e.g. there is no assembly for chacha20)
7. Assorted missing features, e.g. full support for TLS 1.3

On the other hand, LibreSSL currently has only two advantages for us that I can think of:

1. libtls; but that mostly affects projects coming from OpenBSD and can be addressed in other ways
2. bootstrapping is easier with LibreSSL since it doesn't require perl; but that is not much of a problem, since at least our glibc flavors already require it to bootstrap other things, and there have been thoughts about moving xbps to BearSSL which would remove either OpenSSL or LibreSSL from the base system set entirely

The extra assembly project

Yes, no PR has been raised, apologies for that. I mostly wrote that as a weekend project to address our immediate issues (i.e. poor performance on POWER and aarch64 platforms), so that it could be integrated into the version of LibreSSL we're shipping, and haven't found time to raise it here.

It was, in fact, written with the goal of making things easy for LibreSSL to adopt; that's why all the assemblies are hand-picked from the last OpenSSL commits that were still under the original license, rather than the latest Apache 2.0 stuff. I just never got to it.

There are some things in it that might be concerning for LibreSSL. For example, the LibreSSL 32-bit ARM assembly stuff for things like ghash are full of __STRICT_ALIGNMENT code paths for whatever reason, with __STRICT_ALIGNMENT getting haphazardly disabled on ARMv6 and newer (otherwise the assembly is pretty much neutered by default, at least for ghash), and I couldn't quite make sense of why all that is being done; so I replaced these with the OpenSSL versions too while at it, which don't have these checks and yield a fairly large extra performance boost on at least on ARMv7, and then enabled __STRICT_ALIGNMENT for all the C code on all 32-bit ARM. You can see the decisions I've taken in the patches. I have no idea how this works out on OpenBSD.

If you want to integrate the patches/files, I'm not opposed to working with you towards doing that, since it may help other projects. However, I've still yet to see a real convincing argument for Void to stay with LibreSSL in long term; feel free to raise counterpoints, of course. I do appreciate LibreSSL's efforts, but find myself questioning if it's all worth it for the distro.

[RoundDuckKira-deprecated]

@q66 thanks for that :)

[busterb]

Thanks @q66, that was very helpful, and I appreciate the work you put into it.

Things like 1.1 APIs have been backported based largely on what APIs apps use in the OpenBSD ports tree, but I can understand that doesn't help other distros as much, since you're dealing with a different release cadence and a wider diversity of packages. There are other things like ABI updates tied to the ~6-month OpenBSD release cycle, which while the major and minor versions of LibreSSL's libraries bump in this project each time, I can appreciate that most distributions are not setup for the idea of hosting multiple library ABIs at once. I can't see that changing in the future either. The licensing compatibility issue does have me slightly confused as well, since I do see new Apache 2.0 code going into the upstream tree, e.g. LLVM 10, so I'm not sure if the ice is thawing there or not.

In the short term, I'd like to focus with @kinichiro on getting some of the outstanding portable-specific and performance issues addressed that you raised. I'll admit, my own personal itches for this project were scratched by the same advantages you mentioned, in addition to Windows build support. Maybe @bob-beck and @4a6f656c have some ideas about the rest, but I feel there will always be issues where this project won't be the best fit.

[q66]

With LLVM I guess it's pretty much a necessity, as there is no other alternative toolchain that would cover the needs and either staying with an old toolchain or maintaining a fork seem like non-options... another thing is integrating Apache 2.0 code into codebases that are already OpenBSD's, which I doubt that will happen. That's why I picked the asm files from pre-AL2.0 tree, in practice it fortunately doesn't mean much, there have been only very minor enhancenents in some of the aarch64 stuff since (and nothing that would be strictly necessary). There is a path going forward for those as well, it just needs Andy Polyakov to import them into cryptogams - which is still provided under the original license (most of them are already sourced from there, but a bunch of stuff particularly for ARM is missing for now)

[bugcrazy]

Look @busterb and @kinichiro, @bob-beck, this link https://voidlinux.org/news/2021/02/OpenSSL.html

Libressl's future will depend a lot on community support and devs!

[dhewg]

For the lack of better spot to ask about it:

Since the v3.4.1 release there've been multiple API additions, which are required for python v3.10.
Because of that dependency a new release with those additions would be much appreciated, are there any plans for it yet?

See also openwrt/openwrt#4749

[botovq]

Since the v3.4.1 release there've been multiple API additions, which are required for python v3.10. Because of that dependency a new release with those additions would be much appreciated, are there any plans for it yet?

The 3.4 branch is stable, so we won't backport API additions. The next stable release will normally be end of April, at the same time as the next OpenBSD release. At some point before that we will release development versions of 3.5 that include this API. These are not currently planned as it's a bit early in the cycle. Would that help you?

[dhewg]

Half a year is pretty far off, so yes, an early release candidate would definitely help, thanks!

[orbea]

@botovq It would be very helpful for users outside of OpenBSD to have developmental releases for 3.5 if that is at all possible, thanks!

[botovq]

On Sun, Dec 26, 2021 at 10:48:08PM -0800, orbea wrote: @botovq It would be very helpful for users outside of OpenBSD to have developmental releases for 3.5 if that is at all possible, thanks!

Understood. Timing isn't great. There is some major churn coming that will be somewhat painful for downstreams to handle. It'll take a few weeks until the dust settles.

[orbea]

No worries, I appreciate the response and whenever the timing is better it would be great!

[bugcrazy]

There is an interesting discussion with the possibility, again LibreSSL return to Alpine.
https://gitlab.alpinelinux.org/alpine/tsc/-/issues/28

[illiliti]

On the other hand, LibreSSL currently has only two advantages for us that I can think of

Another advantage is no CLA requirement, which is IMHO main reason why libressl is superior to openssl. Imagine you found some issues in openssl, but due to abusive CLA you won't be able to contribute a fix. Just to note, openssl CLA requires phone number, address, real name, ... as well as giving up your rights.