MS Exchange On-Prem Self-Signed SMTP cert prevents 2FA Mails

[senpro-ingwersenk]

Subject of the issue

We have Vaultwarden deployed in a k3s cluster that is configured via the SMTP_* env variables to send emails through our on-premise Exchange server. However, this morning, a collegue noticed that, after I updated the instance this morning, that he could no longer receive emails for his MFA - and I found the related log message soon after.

[2024-04-08 08:26:35.442][panic][ERROR] thread 'tokio-runtime-worker' panicked at 'Error sending incomplete 2FA email: SMTP error: Connection error: Connection error: error:0A000086:SSL routines:tls_post_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1889: (self-signed certificate)': src/api/core/two_factor/mod.rs:273
   0: vaultwarden::init_logging::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::result::unwrap_failed
   7: vaultwarden::api::core::two_factor::send_incomplete_2fa_notifications::{{closure}}
   8: tokio::runtime::task::raw::poll
   9: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  10: tokio::runtime::scheduler::multi_thread::worker::run
  11: tokio::runtime::task::raw::poll
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: std::sys::unix::thread::Thread::new::thread_start

Deployment environment

• vaultwarden version: image: ghcr.io/dani-garcia/vaultwarden:1.30.5-alpine

• Install method: Kubernetes via k3s, deployment

• Clients used: Web Vault (Extension and Desktop Client work)

• Reverse proxy and version: Traefik as dictated by k3s

• MySQL/MariaDB or PostgreSQL version: image: docker.io/library/postgres:15-alpine

• Other relevant details: None I could really think of, sorry.

Steps to reproduce

According to my collegue, all he did was attempt to log in via the Web Vault to obtain login credentials. Upon entering his creds, he received an error page with the error above (minus the log wrap itself). The extension works fine, so I assume it just falls back to offline storage.

Expected behaviour

An email to be sent

Actual behaviour

It appears that Vaultwarden does not like Exchange's shenanigans related to certificates. Although we use starttls and the appropriate port (587), the above error is thrown.

I tried to look for a way to, at least temporarily, disable TLS verification to see if that solved the issue, but found none. Exchange isn't the most obvious about it's certs, according to another collegue with admin previleges (which I do not have), so I have to assume that Microsoft is doing Microsoft things...fun.

Troubleshooting data

See above.

[BlackDex]

You seem to be using a self signed certificate.
Either add the CA to the containers ca-certificates.crt or set Vaultwarden to ignore it.

https://github.com/dani-garcia/vaultwarden/blob/e1a8df96dbadfbf5ad36ce9aa2f31f34396166c2/.env.template#L517..L526

[senpro-ingwersenk]

Definitively not the "best practice" solution, but the one that works.
Thanks, exactly what I needed!

[mirolm]

Also you can make a local postfix relay to make things easier to customize.

[senpro-ingwersenk]

Good idea! Haven't done a postfix deploy in kubernetes before though.
The tldr is that I inherited this cluster from a coworker that was let go and was kinda squished into the whole kubernetes thing o.o I come from "just a crapton of docker-compose deployments"... xD
Will make a note to look into that; disabling the validity check kinda irks me. But for now, my collegue can work again...