SMTP Error due to SSL

[dl-lim]

Deployment environment

• docker image vaultwarden/server:1.21.0-alpine

• Reverse proxy and version: docker image jc21/nginx-proxy-manager:2.9.7

Reverse proxy works and SSL is up-to-date. This feature used to work well under the same conditions.

Steps to reproduce

Didn't have this issue previously on this version, even. But recently, when I did the following, it did not succeed.

I added a new user to the organisation, and Smtp error message showed in red. Logs shown above.

No emails were sent, and no emails were received. User was not able to be confirmed in vaultwarden

Expected behaviour

Email should be sent and received.

Troubleshooting data

[2021-10-14 10:56:29.636][request][INFO] POST /api/organizations/35666776-afda-4a1b-9bf9-8912a6333b55/users/1312eca0-1eb6-4631-b26c-c246a111a8c0/reinvite
[2021-10-14 10:56:30.119][error][ERROR] SmtpError.
[CAUSE] lettre::transport::smtp::Error {
    kind: Connection,
    source: Failure(�
        Ssl(
            Error {
                code: ErrorCode(
                    1,
                ),
                cause: Some(
                    Ssl(
                        ErrorStack(
                            [
                                Error {
                                    code: 336134278,
                                    library: "SSL routines",
                                    function: "ssl3_get_server_certificate",
                                    reason: "certificate verify failed",
                                    file: "s3_clnt.c",
                                    line: 1269,
                                },
                            ],
                        ),
                    ),
                ),
            },
            X509VerifyResult {
                code: 10,
                error: "certificate has expired",
            },
        ),
    ),
}
[2021-10-14 10:56:30.119][response][INFO] POST /api/organizations/<org_id>/users/<user_org>/reinvite (reinvite_user) => 400 Bad Request

[BlackDex]

The mail server you are using is using an invalid certificate. Looking at the message it is expired.

[BlackDex]

Ah, wait, you are using an older alpine image i see.
It could be that this has something to do with the Lets Encrypt Certificate DST Root which is expired.
This will probably be fixed if you are going to use the testing image instead of the latest.

The testing image is using the latest OpenSSL version which ignores that certificate and looks at the next one.

[ninjamonkey198206]

I just tried the testing image and it does not solve the problem.

[ninjamonkey198206]

To be clear, I used the testing image, not the testing-alpine image.

[BlackDex]

Well then the certificate is expired i think.
You could verify it via https://www.sslshopper.com/ssl-checker.html
Which supports port numbers

[BlackDex]

The testing Alpine is using the very latest OpenSSL available. The Debian based are using the versions provide by Debian.

[ninjamonkey198206]

@alderson59 are you using Google Workspace or whatever the hell Google is calling their business G Suite these days?

I tested this via my private server and my business server, same settings aside from user/password that would obviously need to be different.

My private server sends fine. The business server using the Gmail business services fails.

All settings correct in both instances.

[ninjamonkey198206]

I've tested Latest, Testing, and Testing-Alpine. All fail when attempting to use Google Workspace either via smtp.gmail.com or smtp-relay.gmail.com (properly configured at both ends), with or without forced TLS, using less secure apps setting or app passwords. I even tried setting the relay to allow from any source, not just from sources in the domain.

Private gmail account works fine.

[dl-lim]

Apologies for the late reply, been busy as hell. I'm selfhosting mailcow, actually. SSL/TLS works fine on emails being sent and received. None of that Google "less secure apps" nonsense.

[ninjamonkey198206]

Just tested the same settings configuration in another hosted service we run on the same VM, using the Workspaces smtp-relay and direct login, and the emails went through just fine, so the issue is definitely on the Vaultwarden end.

[ninjamonkey198206]

Again, the issue seems to be specific to Google Workspaces addresses. The private email account I tested worked fine using all the images mentioned previously, and the Workspaces account worked using the same settings in another app.

[BlackDex]

But, if it returns that the certificate is expired on the exact same smtp host for the workspace account, but not for the personal one, on port 465, that is kinda impossible. Since the certificate is checked before any account checking is done. I can't test it right now my self, but that is strictly not able to happen.

[ninjamonkey198206]

I've learned to not say that something can't happen, only that it shouldn't.

[ninjamonkey198206]

The issue is on the side of the vaultwarden image.

Personal email account works fine. Workspaces account fails with different errors depending on what config you try. Both private and Workspaces work in a different application on the same domain and hosted VM.

Can't tell you why, it's weird and doesn't make sense given the info I have and my understanding of how it is supposed to work.

But here we are.

[BlackDex]

I would suggest to enable SMTP_DEBUG=true and try again.
That works give a bit more details, unless it really is an expired cert, then it won't reach that part at all

[BlackDex]

Well, i don't know what is wrong with the attempts you do, but for me it works without any issue. I have tried both private and google workspace accounts with an less-secure-app-password and there are no issues. All tests went fine.

The only thing left for you to test, since it is reporting an expired certificate is to test the connection it self from within the container.
So, you need to login into the container, docker exec -it vaultwarden sh after that you can run openssl s_client -showcerts -connect smtp.gmail.com:465 scroll back to the top where you typed the command, and then check if you see something like this.

/ # openssl s_client -showcerts -connect smtp.gmail.com:465
CONNECTED(00000003)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = smtp.gmail.com
verify return:1
---
Certificate chain

All chains should return verify return:1, if not, then there is something wrong. And maybe there is something in between your server and the server from google?

The thing is, an expired certificate just can't happen if the cert really isn't expired.
Also, check the date/time on the host/server/container.

Also, check the /admin/diagnostics page, which will show some generic checks if they are ok or not.

[ninjamonkey198206]

Tried both direct and relay, same type of error. Diagnostics page shows all ok.

[2021-10-16 09:53:50.399][request][INFO] POST /admin/test/smtp/

[2021-10-16 09:53:50.670][lettre::transport::smtp::client::connection][DEBUG] << 220 smtp-relay.gmail.com ESMTP ob13sm865008pjb.4 - gsmtp

[2021-10-16 09:53:50.670][lettre::transport::smtp::client::connection][DEBUG] Wrote: EHLO bitwarden

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400250-8BITMIME

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400250-8BITMIME250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400250-8BITMIME250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH250-ENHANCEDSTATUSCODES

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400250-8BITMIME250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH250-ENHANCEDSTATUSCODES250-PIPELINING

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400250-8BITMIME250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH250-ENHANCEDSTATUSCODES250-PIPELINING250-CHUNKING

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] << 250-smtp-relay.gmail.com at your service, [34.68.199.125]250-SIZE 157286400250-8BITMIME250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH250-ENHANCEDSTATUSCODES250-PIPELINING250-CHUNKING250 SMTPUTF8

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] server smtp-relay.gmail.com with {SmtpUtfEight, Authentication(Xoauth2), EightBitMime, Authentication(Plain), Authentication(Login)}

[2021-10-16 09:53:50.711][lettre::transport::smtp::client::connection][DEBUG] Wrote: AUTH PLAIN gibberish==

[2021-10-16 09:53:50.988][lettre::transport::smtp::client::connection][DEBUG] << 235 2.7.0 Accepted

[2021-10-16 09:53:50.988][lettre::transport::smtp::client::connection][DEBUG] Wrote: MAIL FROM:mail@emaple.com

[2021-10-16 09:53:51.031][lettre::transport::smtp::client::connection][DEBUG] << 250 2.1.0 OK ob13sm865008pjb.4 - gsmtp

[2021-10-16 09:53:51.031][lettre::transport::smtp::client::connection][DEBUG] Wrote: RCPT TO:me@example.com

[2021-10-16 09:53:51.070][lettre::transport::smtp::client::connection][DEBUG] << 250 2.1.5 OK ob13sm865008pjb.4 - gsmtp

[2021-10-16 09:53:51.070][lettre::transport::smtp::client::connection][DEBUG] Wrote: DATA

[2021-10-16 09:53:51.313][lettre::transport::smtp::client::connection][DEBUG] << 354 Go ahead ob13sm865008pjb.4 - gsmtp

[2021-10-16 09:53:51.314][lettre::transport::smtp::client::connection][DEBUG] Wrote: Message-ID: ca61f523-00f1-4112-9922-cd9bff141e69@example.comTo: me@example.comFrom: Bitwarden mail@example.comSubject: Vaultwarden SMTP TestMIME-Version: 1.0Date: Sat, 16 Oct 2021 14:53:50 -0000Content-Type: multipart/alternative; boundary="lh9qN3fwjRkr8ahy08JeEB3paVTFH3owJ3zjuVVU"--lh9qN3fwjRkr8ahy08JeEB3paVTFH3owJ3zjuVVUContent-Transfer-Encoding: base64Content-Type: text/plain; charset=utf-8VGhpcyBpcyBhIHRlc3QgZW1haWwgdG8gdmVyaWZ5IHRoZSBTTVRQIGNvbmZpZ3VyYXRpb24gZm9yIGh0dHBzOi8vYml0d2FyZGVuLnJlbnRhbmVyZGNvbnN1bHRpbmcuY29tLg0KDQpXaGVuIHlvdSBjYW4gcmVhZCB0aGlzIGVtYWlsIGl0IGlzIHByb2JhYmx5IGNvbmZpZ3VyZWQgY29ycmVjdGx5Lg0KDQo9PT0NCkdpdGh1YjogaHR0cHM6Ly9naXRodWIuY29tL2RhbmktZ2FyY2lhL3ZhdWx0d2FyZGVu--lh9qN3fwjRkr8ahy08JeEB3paVTFH3owJ3zjuVVUContent-Transfer-Encoding: base64Content-Type: text/html; charset=utf-8 Gibberish

[2021-10-16 09:53:51.314][lettre::transport::smtp::client::connection][DEBUG] Wrote: .

[2021-10-16 09:54:06.334][lettre::transport::smtp::client::connection][DEBUG] Wrote: QUIT

[2021-10-16 09:54:21.439][error][ERROR] Smtp.

[CAUSE] lettre::transport::smtp::Error {

kind: Network,

source: Os {

    code: 11,

    kind: WouldBlock,

    message: "Resource temporarily unavailable",

},

}

[2021-10-16 09:54:21.439][response][INFO] POST /admin/test/smtp (test_smtp) => 400 Bad Request

[ninjamonkey198206]

That's using the config listed on the page you posted, with app password.

[BlackDex]

Strange, it does look it accepts the cert. And it even sends out an mail which google seems to except. But after/during the quit it seems to loose it's connection which results in an error. Is the mail also not delivered? Since it did send the . which means end of mail.

[ninjamonkey198206]

The mail never gets delivered and doesn't show up anywhere I can find in the Google admin or the account.

The one thing I don't have access to for some reason is the quarantine section of the admin page.

[ninjamonkey198206]

I'm gonna need to install some padding on the walls so I don't crack my skull from beating my head against them.

[ninjamonkey198206]

@BlackDex For troubleshooting:

Our VM that I'm using is hosted by Google, all traffic allowed out and standard http, https, and their other standard ports open (essentially Googles default firewall config for their hosted VMs, aside from one additional port for another service).

Container mapped ports are exposed on localhost only (mapped 127.0.1.1:3012:3012, for example).
All other containers work flawlessly.

IPTables firewall on the VM, no management app. I've omitted unrelated rules:

# Generated by iptables-save v1.8.4
*filter
:INPUT DROP [45:12527]
:FORWARD DROP [0:0]
:OUTPUT DROP [63:6851]

# Allow outbound SMTP
-A OUTPUT -o ens4 -p tcp --dport 25 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -i ens4 -p tcp --sport 25 -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Allow outbound SMTP SSL/TLS
-A OUTPUT -o ens4 -p tcp -m multiport --dports 465,587 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -i ens4 -p tcp -m multiport --sports 465,587 -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Allow outbound HTTP and HTTPS
-A OUTPUT -o ens4 -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A INPUT -i ens4 -p tcp -m multiport --sports 80,443 -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Allow incoming HTTP and HTTPS
-A INPUT -i ens4 -p tcp -m multiport --dports 80,443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o ens4 -p tcp -m multiport --sports 80,443 -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Allow Vaultwarden proxy access (necessary if using full drop by default to enable ingress to containers via localhost, use container internal ports)
-A INPUT -i br-81888841de4b -p tcp -m multiport --sports 80,3012 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o br-81888841de4b -p tcp -m multiport --dports 80,3012 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT

# Allow loopback
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

# Allow established and related traffic
-A INPUT -i ens4 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A OUTPUT -o ens4 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Enable logging
-A INPUT -j LOG

# Drop invalid packets
-A INPUT -m conntrack --ctstate INVALID -j DROP

COMMIT
# Completed

HAProxy is installed and provides HTTPS redirect, SSL offloading, and reverse proxy services. Listening on 80 and 443, routing to 127.0.1.1 on appropriate ports.

[BlackDex]

I'm not sure why you have separate rules for output of 465,587 and 25 while you say all outgoing is allowed by default. I think you can remove those rules and try to see what happens.
I have added these to my firewall, and it didn't have any effect, but who knows what it does in your environment.

[BlackDex]

@amousset would increasing the timeout be something to test then? Currently the default is set to 15 seconds in Vaultwarden.
@ninjamonkey198206 could you try to increase the timeout? You need to change the SMTP_TIMEOUT environment variable for this. You could try to set it to 60 or something, that would seem to be more then enough.

[ninjamonkey198206]

@BlackDex @amousset no joy. Tried setting it to 60 and then 120. Same result.

[ninjamonkey198206]

Well this is interesting. I've got it working over here currently. Either something changed on Google's end, or I found out something interesting.

Apparently something changes in the last communication stages when it sends the . and quit, as both essentially time out.

I've mentioned before that I have a drop by default IPTables firewall on the VM itself, and bind the exposed ports to localhost, using HAProxy to direct traffic appropriately. Using drop by default breaks accessing the docker containers via localhost regardless of where you bind the docker ports, but adding rules to allow traffic to the docker network solves that without exposing the listening ports to the world.

That has never stopped any normal necessary communication with the outside world for any container on any of my systems before (DNS, email, etc) but even with a blanket accept all rule to the docker bridge the closing part of the email was failing.

On a whim I added new rules explicitly allowing all related traffic to the docker bridge network, and it worked. I've never needed to do that before, as I already have rules accepting established and related traffic, and have no idea why it allowed all other traffic but the last two steps during the email process, nor why it worked until recently if that was an issue.

[ninjamonkey198206]

It shouldn't make any difference at all, as everything else worked, it had worked previously, and other containers with the same firewall rules in place worked perfectly fine sending emails via the same server/relay, and yet...

¯\_( ' _ ' )_/¯

[paolobarbolini]

Regarding the -crlf, if i'm correct, but @paolobarbolini should correct me if i'm wrong, that is exactly what lettre is sending. So, with -crlf is the correct way, and should represent the same as is done with lettre.

Correct

[paolobarbolini]

Hello, pretty late here but I'll try giving you something to try. I think we can rule out the wrong port ipothesis as failing to connect would have never gotten lettre to the point of sending commands to the SMTP server (#2040 (reply in thread))

What you could try as a quick test is to use openssl to open a TLS connection to the relay server and manually issue SMTP commands. From a quick glance on Google most SMTP guides seem incomplete, for example most of them don't tell you to login. You can follow those but also check against commands issued by lettre (which you can see in the logs) to see if they missed a step.

I'll try looking deeper into this and sending more specific instructions in the following days.

[dl-lim]

Hi all, just to reiterate on my opening thread, I'm not using Google or any of their strange "less secure apps" protocols. I'm selfhosting mailcow (https://github.com/mailcow/mailcow-dockerized), and this has been running well for my system for a long time. SSL/TLS works on incoming and outgoing emails.

Is there currently a way to bypass this invitation process so that I can get a new user set up?

[BlackDex]

Great, then i think you were using a LetsEncrypt cert which needed some OpenSSL updates or since ca-certificate updates from the base image

[Blackclaws]

The issue still persists with the 1.24.0 and latest images. Testing works. When can we expect an official release with the updated certificates? Anyone using current letsencrypt certificates is affected by this I think.

[BlackDex]

v1.24.0 and latest are identical so that is not strange.
testing is based upon the main branch and build every time there is a new commit/merge.

We are still working on some optimizations and there has been a new release of some libraries which we probably want to update to.
Which needs some testing.

[Blackclaws]

Gotcha. Might I suggest an updated v1.24.0 image then that just rebuilds with the latest base image? That should solve the issue.

[BlackDex]

Not sure if that would solve the issue, since those images are based upon an older version of debian, and i'm not sure if that has the correct updates to solve your issue.