OpenSSL-related crashes on Docker Alpine image

[jeda]

Subject of the issue

Looking at my vaultwarden logs, I noticed something peculiar in them. Apparently, vaultwarden has had occasional crashes due to OpenSSL errors, which seem pretty random.

[2022-03-19 17:55:36.236][response][INFO] (login) POST /identity/connect/token => 200 OK
[2022-03-19 17:55:36.398][request][INFO] GET /api/sync
[2022-03-19 17:55:36.425][response][INFO] (sync) GET /api/sync?<data..> => 200 OK
crypto/evp/evp_enc.c:510: OpenSSL internal error: assertion failed: b <= sizeof(ctx->final)
/--------------------------------------------------------------------\
|                        Starting Vaultwarden                        |
|                      Version 1.24.0-8d06d9c1                       |
|--------------------------------------------------------------------|

[2022-03-19 18:15:37.503][parity_ws][INFO] Listening for new connections on 0.0.0.0:3012.
[2022-03-19 18:15:37.506][start][INFO] Rocket has launched from http://0.0.0.0:8080

[2022-03-20 16:14:15.683][response][INFO] (login) POST /identity/connect/token => 200 OK
[2022-03-20 16:14:16.022][request][INFO] GET /api/sync
[2022-03-20 16:14:16.048][response][INFO] (sync) GET /api/sync?<data..> => 200 OK
crypto/evp/evp_enc.c:156: OpenSSL internal error: assertion failed: ctx->cipher->block_size == 1 || ctx->cipher->block_size == 8 || ctx->cipher->block_size == 16
/--------------------------------------------------------------------\
|                        Starting Vaultwarden                        |
|                      Version 1.24.0-8d06d9c1                       |
|--------------------------------------------------------------------|

[2022-03-20 16:15:42.751][parity_ws][INFO] Listening for new connections on 0.0.0.0:3012.
[2022-03-20 16:15:42.753][start][INFO] Rocket has launched from http://0.0.0.0:8080

I have not noticed any disruption while using it, but it raises an eye.

Deployment environment

• Vaultwarden version: v1.24.0-8d06d9c1 (latest vaultwarden/server:testing-alpine, amd64)
• Web-vault version: v2.26.1
• Running within Docker: true (Base: Alpine)
• Environment settings overridden: false
• Uses a reverse proxy: true
• IP Header check: true (X-Real-IP)
• Internet access: true
• Internet access via a proxy: false
• DNS Check: true
• Time Check: true
• Domain Configuration Check: true
• HTTPS Check: true
• Database type: PostgreSQL
• Database version: PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
• Clients used: Multiple of them, including Web Vault, Desktop and Android apps.
• Reverse proxy and version: Caddy v2.4.6

Steps to reproduce

Unsure. I have pretty much a "regular" setup, with WEBSOCKET_ENABLED and that's it.

It seems to have happened during a while after regular sync intervals, no entries added or anything.

[BlackDex]

It almost looks like you also have Vaultwarden configured to serve HTTPS to Caddy.

The SSL implementation for the Rocket version used is a bit unstable sometimes. That is why we suggest to use a reverse proxy and let it offload the SSL also.

I'm not sure if you have it configured like that, but if that is the case i would suggest two things to try

1. Disable SSL on the Vaultwarden side and let it be handled by Caddy.
2. Try the testing-alpine version and see if that helps. That version has a newer library.

[jeda]

I don't have TLS enabled on Vaultwarden. It's served via HTTP to Caddy.

I am using testing-alpine already, as pointed out in the original issue. I don't recall seeing this issue in the latest stable version, but I can't give out any confirmations right now.

[BlackDex]

Ah indeed. Sorry late here already hehe.
Is the postgresql connection secured with ssl?
That would be the only item left which i can think of.

[jeda]

Actually, good point. Isn't PostgreSQL secured with TLS by default?
That could possibly be it. Would also explain why it's not exactly when a HTTP request is received.

[jeda]

Yep.

postgres=# SELECT * FROM pg_stat_ssl;
  pid   | ssl | version |         cipher         | bits | client_dn | client_serial | issuer_dn 
--------+-----+---------+------------------------+------+-----------+---------------+-----------
 186528 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186526 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186527 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186529 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186530 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186531 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186532 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186533 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186534 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 
 186535 | t   | TLSv1.3 | TLS_AES_256_GCM_SHA384 |  256 |           |               | 

postgres  186526  0.0  0.3 357448 16032 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39474) idle
postgres  186527  0.0  0.3 357448 16032 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39476) idle
postgres  186528  0.0  0.3 357448 16036 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39478) idle
postgres  186529  0.0  0.3 357448 16032 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39480) idle
postgres  186530  0.0  0.3 357448 16036 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39482) idle
postgres  186531  0.0  0.3 357448 16032 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39484) idle
postgres  186532  0.0  0.3 357448 16036 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39486) idle
postgres  186533  0.0  0.3 357448 16032 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39488) idle
postgres  186534  0.0  0.5 359044 21256 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39490) idle
postgres  186535  0.0  0.5 359376 22384 ?        Ss   19:15   0:00 postgres: 14/main: vaultwarden vaultwarden 172.18.0.2(39492) idle

[BlackDex]

Its either that, or there is a weird issue with the client which is still using a PostgreSQL v11 library.

The library does indeed first tries ssl, if that fails it tries without.

[jeda]

I'm gonna try adding ?sslmode=disable to the PostgreSQL URI and see if it still crashes. Shouldn't matter that much since they're in the same server.

Will report back with results.

[BlackDex]

When a newer version of the testing will be released it will be using a newer version of OpenSSL. Maybe that helps, but i doubt it actually. But who knows.

But please keep us posted.

Edit:
If needed, i could build a version which uses PostgreSQL v14 btw instead of PQ11

[jeda]

If needed, i could build a version which uses PostgreSQL v14 btw instead of PQ11

Let's see if it crashes again, and if so, I'll ask for it. You can build it anyway if it's no issue.

Looking good so far, though.

[BlackDex]

I tried this my self using a postgre docker container with a self-signed ssl-cert.
I'm not seeing any issues so far.

I created a personal vault with 9000+ items in it and a org vault with 2200+ items.
It seems to keep working.

Looking the the version string of your PostgreSQL Instance, you are probably using the exact same container.
I'm not sure which parameters you used to start it, but i used the following:

docker run --rm --name vw-postgres \
  -e POSTGRES_PASSWORD=MySecretPassword \
  -e POSTGRES_USER=vaultwarden_user \
  -e POSTGRES_DB=vaultwarden_db \
  -v "$(pwd)/server.crt:/server.crt:ro" \
  -v "$(pwd)/server.key:/server.key:ro" \
  -p5432:5432 \
  postgres:14.2-bullseye \
  -c ssl=on -c ssl_cert_file=/server.crt -c ssl_key_file=/server.key

And i started the Vaultwarden container like this:

docker run --rm --name vaultwarden \
  -e RUST_BACKTRACE=full \
  -e LOG_LEVEL=debug \
  -e DISABLE_ADMIN_TOKEN=true \
  -e DATABASE_URL="postgresql://vaultwarden_user:MySecretPassword@x.x.x.x:5432/vaultwarden_db?sslmode=require" \
  -p8181:80 \
  vaultwarden/server:testing-alpine

I then used the Bitwarden Desktop, CLI and Browser extension to login to the instance and add/sync items.
I left it running for a long time and did some more testing and no issues as of yet.

Not sure what could be the issue on your side then though.
Have you tried to restart the PostgreSQL instance/server?

[jeda]

I restarted it before creating this issue.

So far, it has still been working fine. No crashes or any odd errors. I'll try switching back to TLS.

[jeda]

...And nothing.
It's just.. working.

Software.

[BlackDex]

Still strange.
I can understand that you would prefer a secure db connection.

I dived a bit into the specific errors, the only thing i can see is that it could be some weird OpenSSL version difference somewhere. The Alpine image has a statically linked OpenSSL which could differ a bit from the OpenSSL installed by Alpine, but i'm not seeing how those two would even interact, since the vaultwarden application shouldn't be using the OS libraries at all.

Maybe a newer build of the Alpine image would solve this, since they then probably both use OpenSSL v1.1.1n on all levels.

[jeda]

I have no idea what's happening in here either, but yeah, it's functional..

Probably a newer build would be fine. The crashes are indeed strange.

[spantaleev]

I've just randomly experienced this on docker.io/vaultwarden/server:1.26.0-alpine on amd64, so it seems like it's still a problem.

crypto/evp/evp_enc.c:156: OpenSSL internal error: assertion failed: ctx->cipher->block_size == 1 || ctx->cipher->block_size == 8 || ctx->cipher->block_size == 16

Not sure how it could be reproduced or about more details (logging level was set to info and without extended logging).

[BlackDex]

on what kind of host are you running this, and during which calls? Also, did you configured Vaultwarden to use SSL certs it self? Or is this handled by a reverse proxy?

[spantaleev]

This is on a Rocky Linux 8.7 (amd64) Proxmox VM. The Docker CE version is 20.10.21.

I'm reverse-proxying with Traefik. There's no SSL configuration for Vaultwarden itself. I've since added ?sslmode=disable to my DATABASE_URL.

Another thing that does SSL is SMTP email sending - I'm using SMTP port 587 (starttls). But I'm not sure what could have sent an email when I noticed Vaultwarden going down.. It happened as I was modifying some local (non-Organization-stored) password entries via the web UI.

[BlackDex]

That could be a database connection indeed.
Which version of Postgres are you running as server?