Docker images not working as expected with SSL

[mattoz0]

We have just updated our docker images, we were stuck on the soketi/pws images for a while. After taking a look a the newer images located at quay.io/soketi/soketi i have noticed two issues.

1. The images that are post-fixed with -alpine seem to have a segmentation fault and don't work at all. The seg fault happens as soon as you try to connect to the echo server from a frontend application.

echo  |   📡 soketi initialization....
echo  |   ⚡ Initializing the HTTP API & Websockets Server...
echo  |   ⚡ Initializing the Websocket listeners and channels...
echo  |   ⚡ Initializing the HTTP webserver...
echo  |   🕵️‍♂️ Initiating metrics endpoints...
echo  |
echo  |    🎉 Server is up and running!
echo  |    📡 The Websockets server is available at 127.0.0.1:6001
echo  |    🔗 The HTTP API server is available at http://127.0.0.1:6001
echo  |    🎊 The /usage endpoint is available on port 9601.
echo  |
echo  |    [Fri Apr 08 2022 04:30:25 GMT+0000 (Coordinated Universal Time)] Promoted from node to master.
echo  |   [object Object]
echo exited with code 139

2. The images that are post-fixed with -distroless seem to no longer be generating since about 2 months ago (last one was 0.30.2-16-distroless). Its worth noting these images are recommended for production in the documentation. here: https://docs.soketi.app/getting-started/installation/docker

I have confirmed that the distroless images do not have the seg fault and work fine. I have also confirmed the seg fault was not present in the older soketi/pws images.

[rennokki]

Now I'm aware of the distroless issue.

I still don't know what causes the segmentation fault, yet I've tried several times and had no luck replicating the issue.

Do you have a step by step guide for this? I'm trying to get my hands dirty.

[mattoz0]

Ok! i've looked into this a lot more today. This issue only seems to happen when SSL_CERT and SSL_KEY environment variables are defined. I assume its an issue with the uWebsockets package and the closes issue i could find was this one.
uNetworking/uWebSockets.js#71

Strange thing is with the same exact configuration quay.io/soketi/pws:latest-16-alpine works but quay.io/soketi/soketi:latest-16-alpine does not (SSL certs are definitely present).

With a bunch of experimentation i can confirm that this issue still happens even if i use an older version of alpine (such as the one that was on the older images). So the issue is either present in the soketi code or the uWebsockets code (not the image itself).

Ill continue to dig into this tomorrow!

[mattoz0]

Ok i've looked into this more, a couple of releases ago uWebSockets.js released an update that moved their cryptographic package away from OpenSSL and instead used BoringSSL. This seems to have broken SSL or somehow changed what is required.

v20.1.0 release notes here: https://github.com/uNetworking/uWebSockets.js/releases/tag/v20.1.0

For now i've created a PR to revert to v20.0.0 to get SSL working. However i will also create a ticket in the upstream repo and get this fixed there. I have tested this and can confirm this fixes the issue.

#465

[mattoz0]

Apparently Alpine isn't supported at uWebsockets.js which could be causing this issue. I can think of two options off the top of my head:

• Change the base image
  OR
• Rebuild uWebsockets on docker build and compile with OPEN_SSL.

Thoughts?

[rennokki]

I can, at most, set up a parallel image for Debian-based Node.js, just to keep the reverse compatibility with the current Alpine images.

uWebSocket should work fine non-SSL in Alpine, right?

[mattoz0]

Yeah, I've had no issues without ssl and reverse proxy configs seem fine. We might just need to monitor it as features could break at anytime. A Debian image would be good though gives some options.

[ghost]

uWebSocket should work fine non-SSL in Alpine, right?

Absolutely not. Never has been. Alpine is a muslc distro that, in their own FAQ:

Binary compatibility is much more limited [...] all but the simplest glibc-linked applications will fail if musl is dropped-in in place of /lib/ld-linux.so.2.

Besides, muslc will ruin performance of uWS if it even manages to run without blowing up immediately. It is sheer luck that it hasn't blown up before.

I tested SSL support and it worked just fine.

uNetworking/uWebSockets.js#26

[rennokki]

Will be released in next 0.x update.

[ghost]

There are small glibc Docker images. Busybox is one that is 5MB or something. We can only support Node.js from an enterprise perspective; LTS versions and only on official platforms. If Node.js were run by Apple or Microsoft or Oracle or ant such company, this would be enforced and always correct, but Node.js is run by partly companies and partly amateurs who just mix and match with the sources as they like, without caring the slightest about ABI rules. For instance, some distros compile Node.js with entirely wrong OpenSSL versions which breaks compiled modules. And they do this because they want all packages to use the same OpenSSL version, and will gladly break ABI that way. So right now the only dependency we have is glibc and that is a hard dependency.

Rebuild uWebsockets on docker build and compile with OPEN_SSL.

This mentality worries me. Instead of just following set rules, the mentality here is to further break the rules only to use unsupported platforms, which are not even good for uWS - for instance we rely on memchr which is SIMD optimized in glibc and entirely unoptimized for loop in muslc. And we know that BoringSSL is 18% faster than OpenSSL for the loads we see, so nothing of this is for the better.