nginx HTTPS reverse proxy causes the client to fail to connect to server in unintuitive way

[Kexkey]

Hi,

Electrum Wallet 3.3.4 on OSX, same problem on Windows.

I'm testing an instance of electrum server. Everything works well when using it in clear (http port 50001) as well as through Tor, but when I use it with TLS (I installed a letsencrypt TLS certificate), Electrum wallet always gives me this error:

[i][electrum.kexkey.com] disconnecting due to: SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)')

Used -s electrum.kexkey.com:50002:s when starting up.

Are you aware of any recent TLS problems with python or electrum wallet? When I hit https://electrum.kexkey.com on port 443 or 50002, the browser tells me the certificate is valid.

Thanks!

[FrancisPouliot]

Confirm i am affected by this issue on Windows with 3.3.4

[cluelessperson]

You don't need a letsencrypt cert.

Electrum Client saves the first certificate it downloads from the server, and just requires that it match in the future. If you changed certs, it'll be different and invalid.

You can fix it by going into your electrum directory and deleting your previous server certificate.

[SomberNight]

Is that server online now? I would like to try to connect.
There are no known issues about this atm.

[SomberNight]

Electrum Client saves the first certificate it downloads from the server, and just requires that it match in the future. If you changed certs, it'll be different and invalid.

To be clear, trust on first use only applies to self-signed certificates. For more details, see #1782 (comment)

[Kexkey]

Hi, thanks for the tips.

So I deleted ~/.electrum/certs/electrum.kexkey.com and it doesn't connect ([n] couldn't launch iface electrum.kexkey.com:50002:s -- TimeoutError()). I created an empty file (touch) and it still doesn't connect ([i][electrum.kexkey.com] disconnecting due to: CancelledError()). I decided to copy the certificate in the file (I exported it from the browser) and it gives me the same SSLError message (CERTIFICATE_VERIFY_FAILED).

@SomberNight did you try the server? Did it work? Out of curiosity, do you know why Electrum Wallet doesn't just simply use the CA signed cert if available, behaving like a browser dealing with a cert?

I noticed some more CERTIFICATE_VERIFY_FAILED messages with other servers today, too.

Thanks for your support.

[SomberNight]

did you try the server? Did it work?

I've tried connecting to electrum.kexkey.com:50001 using TCP, there I get:
[i][electrum.kexkey.com] disconnecting due to: ConnectionRefusedError(111, "Connect call failed ('158.69.33.222', 50001)")

I've also tried electrum.kexkey.com:50002 using SSL, there I get:
[n] couldn't launch iface electrum.kexkey.com:50002:s -- TimeoutError()

I've assumed you turned off the server.
Reaching a timeout does not suggest SSL related errors. Are you sure your server is on, listening on that port, and that port is open?

Out of curiosity, do you know why Electrum Wallet doesn't just simply use the CA signed cert if available, behaving like a browser dealing with a cert?

It does that! Electrum is very flexible here, as explained in my link above, it accepts both CA signed certs and self-signed certs.

I created an empty file (touch) and it still doesn't connect ([i][electrum.kexkey.com] disconnecting due to: CancelledError()). I decided to copy the certificate in the file (I exported it from the browser) and it gives me the same SSLError message (CERTIFICATE_VERIFY_FAILED).

You shouldn't fiddle too much with the stuff in the certs dir (for the client); apart from maybe deleting files already there, for debugging purposes.

[Kexkey]

I exposed port 50001 just for the time I needed to test it. It worked. Now I am exposing port 50002 and 443, both with SSL (with the above errors), and also as a Tor service (it works, I'm currently using it).

I'll try with a self-signed cert (instead of letsencrypt) to see if I get the same results.

In the meantime, if you have more ideas for me to try, I'm all ears. :)

Oh by the way, I think there was a "SSL" checkbox in the network dialog before. Why did it get removed? It would be nice to be able to switch to an onion server via the GUI. When I tried, it was trying to connect to the onion address with :s SSL. I had to exit and launch the wallet with the args on the command-line. (-s awielufh.onion:50001:t). Same thing when I was trying on port 50001, it was trying to connect to :50001:s when I was entering the url in the gui.

I'll let you know if there's something new. Thanks again!

[SomberNight]

Oh by the way, I think there was a "SSL" checkbox in the network dialog before. Why did it get removed? It would be nice to be able to switch to an onion server via the GUI. When I tried, it was trying to connect to the onion address with :s SSL. I had to exit and launch the wallet with the args on the command-line. (-s awielufh.onion:50001:t). Same thing when I was trying on port 50001, it was trying to connect to :50001:s when I was entering the url in the gui.

It was removed to simplify the GUI a bit, and because there is no reason for ordinary end users to disable SSL. It's true that in the case of .onion servers, it might make sense to use TCP as they are already encrypted/authenticated. You can still use .onion servers through SSL though, and the majority of operators of .onion servers expose SSL ports. Also, if you have your own server, or you want TCP for debugging reasons (which is why it was kept at all), then you can do via the CLI option you wrote.

[SomberNight]

What server software are you running? Is it recent ElectrumX?

It seems as soon as the client sends server.version, the connection gets closed (connection_lost gets called with value None).
Note that this is after the SSL handshake and connection establishment, so your issue does not seem to be directly related to SSL. The client gets the cert and considers it valid.

Btw, how come that you originally said you are getting SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)') but this is no longer the case, the symptom changed?

[Kexkey]

What server software are you running? Is it recent ElectrumX?

Electrs: https://github.com/romanz/electrs

Btw, how come that you originally said you are getting SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)') but this is no longer the case, the symptom changed?

When the file ~/.electrum/certs/electrum.kexkey.com contains the cert, I still get that error. It's when I tried deleting that file as well as creating an empty file that I got other errors (see earlier comment).

Note: it works well without SSL as well as through Tor (tcp).

[SomberNight]

Your port 50002 points to an nginx HTTP webserver. Of course Electrum cannot connect....
It should point to the JSON RPC endpoint of an electrum server.

Compare

electrum.qtornado.com:50002:s

$ openssl s_client -connect electrum.qtornado.com:50002
CONNECTED(00000003)
depth=0 C = US, ST = California, O = Internet Widgits Pty Ltd, CN = electrum.qtornado.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = California, O = Internet Widgits Pty Ltd, CN = electrum.qtornado.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/O=Internet Widgits Pty Ltd/CN=electrum.qtornado.com
   i:/C=US/ST=California/O=Internet Widgits Pty Ltd/CN=electrum.qtornado.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFRjCCAy4CCQDGYJrD0tQhazANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV
UzETMBEGA1UECAwKQ2FsaWZvcm5pYTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0
cyBQdHkgTHRkMR4wHAYDVQQDDBVlbGVjdHJ1bS5xdG9ybmFkby5jb20wHhcNMTcw
NTI1MTIxMzIyWhcNMjcwNTI2MTIxMzIyWjBlMQswCQYDVQQGEwJVUzETMBEGA1UE
CAwKQ2FsaWZvcm5pYTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRk
MR4wHAYDVQQDDBVlbGVjdHJ1bS5xdG9ybmFkby5jb20wggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCbvxjWkt2zcHOwemj0HPbaXO721L/uWgxx0voBUnfN
WAbJwjnCE5a5pCe0bRphKIdqaZk2AbOO/jihT8+W7Nc+EzHM+U2BIQedWLYr2bMb
se4YxCHm0Ql++3XOgyadXipNplIuAopy+sb3XXYsZAVRkGFXYneA3bVzyuqP9vo1
8Sq0i0pVTXveephyFi7UYEascrWcNZbBNWu0fNjhCix63II8+eP8laQ7ojyRbYIh
6+lteHk9E5v5i91siyq9E4MWCjWgsebB/T1jtnET+dItnzIjrtTU7f0RxFCeihvq
ltqptqg2vQplJHfFdwlPPZGSGUSsxU0fznRAbGVcw6vA0qBWseFQUQGSVuyTXUzH
6Zhah4eu9mfnnHClNgZpmZmYBXDBS+F6vP9ODj3RcPbb7ZPvxm7eaZ+Ycp8P26EL
57DKf5EQeLP2+u+Yx6H5gwE4TB/yWnUmq17v/zGB1+qQOC46Q3prMbrj4vLOBasH
v0N+qXznxHCf57OkFkpxLIgvujWvRP9Z4K5Hq8uJDyphvACbrWlQz1pWyF1rZYZF
FSSGnLtSPk4ekRCydeq2h4TNO6IKvXLX1UmDBptxRAcxU3UcYeIGXl5TjEGENQbS
X/brxSWTaVxFIjS51G27uONDOFKcpAvodSEKbs8J1eBKo5LBrXI+4e2D+s0Fdyor
xwIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQA899Uc5BoBuLpsLvIvj+KKM3JtUipC
bPh5TubnqaWzDdVhSHMQEy9ZTMbzK0DQ9H1DmaNJhHgUIVX3pmqcPEJVFLSnTdWB
s0DmsfRuOsjRh9U5TUUcjGoHRdzed0ujcMBJXOgPwuj8UWmSheZzwMlaf1fbu4af
4VivNdO49YXxqHn09rZcJeS1p5oOAFeJnuC+aQcFhZ4HwJz6Jhg1HHinHUaoc+NG
wzUAMl5l4fEWMnlBJ6GfU2CiBdtOKRTq7q1o13goaSk4D/ONuQPMP6jeDS5WOiLo
p2nWG6jQxCbqUMHL3y40wj+4c6EfubCO2LOfayzw8fqo0VFTtcrB49whqC8B7wjU
+6dAirhduhYwMs6V/3cNFBhO+78thfa4tOlomCuU2OYYf4pOc3hTYaBGL1C/YIVe
dk1J2Wzf85GYORiwsbD/8ilhY7weBPLQM109CTEiQcUt8/uz/zRfW/rnomjVLnrX
qOq3lVRBOsWacPOpMcCS5OtcmvqGEvw4nwd/9GLNWQy8m/+rOXzeJVaSME8cB7NB
uoeql1OoNwwTPT8MlcnKIdCmPo84QTZgl74Vso9Ljrsc4EK2VjcMjwK4OyFH4YNc
Vl84MGkcat2ANrJmq0RXkTT+p34djAYCw9tyGg/eYQK0jIpvFiaqyhACfcxnCAyy
TS9pO7QEW/zZEQ==
-----END CERTIFICATE-----
subject=/C=US/ST=California/O=Internet Widgits Pty Ltd/CN=electrum.qtornado.com
issuer=/C=US/ST=California/O=Internet Widgits Pty Ltd/CN=electrum.qtornado.com
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2235 bytes and written 269 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 6FE2ACAACFF2EA5D8225C71CD9343B8C9E4085F19019471CDCA915298421EC75
    Session-ID-ctx: 
    Master-Key: 068A285D15C93ACD5814DA0CDD0738E3A3EF748E9E726EC3B086C99EB7C437C9B87FEDCE4C8CB748BFA0CFB795F1328F
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - c1 35 e3 bd e1 1f 68 e4-10 be e1 4a a6 56 63 11   .5....h....J.Vc.
    0010 - 92 ac 7c 32 34 d9 56 38-3a 7e 3f 8f 85 76 f3 81   ..|24.V8:~?..v..
    0020 - 61 8d f2 5e 13 c4 4c 48-4c 23 9a 18 b8 23 5c 5e   a..^..LHL#...#\^
    0030 - b4 ed 61 60 3b 6b 1d 93-7d 0d 23 8e 28 d4 42 6a   ..a`;k..}.#.(.Bj
    0040 - e7 24 0e ad 3d 71 8f d1-24 e4 5f 6f ca a4 07 63   .$..=q..$._o...c
    0050 - 4a 18 84 03 b7 5b d8 34-e9 4b 73 1f 49 14 ad 9d   J....[.4.Ks.I...
    0060 - de f6 ce bc d1 77 24 20-2d a4 1f bb 16 38 8b fa   .....w$ -....8..
    0070 - 1a 70 1f 9f 16 7b 95 a1-10 c9 34 05 c5 f5 62 79   .p...{....4...by
    0080 - 08 80 fb b8 52 ec c0 ec-46 85 6b 0f 09 77 c3 2e   ....R...F.k..w..
    0090 - 26 ca f4 9c 0a 7a dd 9f-ad e0 ac d9 d2 07 75 44   &....z........uD

    Start Time: 1552942274
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: yes
---
{"id": 1, "method": "server.version", "params": ["testing", "1.4"]}
{"jsonrpc": "2.0", "result": ["ElectrumX 1.9.5", "1.4"], "id": 1}

with

electrum.kexkey.com:50002:s

$ openssl s_client -connect electrum.kexkey.com:50002
CONNECTED(00000003)
depth=0 CN = letsencrypt-nginx-proxy-companion
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = letsencrypt-nginx-proxy-companion
verify return:1
---
Certificate chain
 0 s:/CN=letsencrypt-nginx-proxy-companion
   i:/CN=letsencrypt-nginx-proxy-companion
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE1DCCArwCCQCXHXAzOIgYBzANBgkqhkiG9w0BAQsFADAsMSowKAYDVQQDDCFs
ZXRzZW5jcnlwdC1uZ2lueC1wcm94eS1jb21wYW5pb24wHhcNMTkwMzE0MTYxODE4
WhcNMjAwMzEzMTYxODE4WjAsMSowKAYDVQQDDCFsZXRzZW5jcnlwdC1uZ2lueC1w
cm94eS1jb21wYW5pb24wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCz
pDe8jYJsrjBW9BlpPOK5kdRsC5kT5/Nysc/07U2VN4IJhSTTexGX4m90OA7AOSrR
UTRI+Ov1yQwCKI/RA2tLHNokfhd46pzoG/dMMoHAN4hpXsamXHyusEU9eK054qrg
6Qwea2Teyc6/fvomj/YcaFDYX27JhrbemWskj+3va4Olh/oqMRRi/gFdadOQAxrP
IJe5wh3nrUiefwjsPnSm6Kp9wlKcx5zras7q0QRgr7HS8mYAQ/x59kSf0pBh9ZKr
5AwXvcVGVEPthj2ZDCkAn2rMFw6yHea/miLUs84kvVAnGLSTxcqNXQrBt0q2S7Aw
SX27HMhecVmrbpjXwIWZfpSSmQzpi3Iq9qAcQejPOU421dOHa8YthgIM3KOzegcr
sOEyylNH+NZzJAHM6vxbKtiyNvYZKxVe84UeCZg28GJLj9AGQyl88CCKL7GjSdsT
hgdJWjhGFqpGKI3lAvl9ILyhrfkGGzTUDrC42molJVNSZsxxd7i9CMuaEkmKLmbe
qlaUpm1gGf8BqhIC+w57OekNQrZPBzQx/LPDcNp2Rh9vhxerQzf4aiZnyuLoLnGL
DZWD0NbG8fuxAi4RLzrTSKCw0P2rc2BoQyUTztE2k30Bp6+SwKTNflqsX+45IqYc
3U00NZ9Z82YIGPZC4QIlqUpTZqqXUQ+KH9jN9kEjWwIDAQABMA0GCSqGSIb3DQEB
CwUAA4ICAQBUoQnjjHP96zRbbFyZF1s/3zbtz/+K69JM+nfCCzUy+pcCucj9fRvP
iJkKZCmjVsD2HUUgK79IMt+ZaOIUXYEFUlivwPzZ9Gsw9WX3C7d54mCAuek+8FTl
4mSRWV8yS3uQQ7aeghc4TulIRi3+rjy26kgkLOxP5Ylibn6mE1RnSgUOme5QSpUT
zNkmz/EJPlAlR/bs3nM7OPHHIPpL8wGry1Sl+/i4D8KrQ8qvsIz0tuctFCcgWDjv
ro0OUdQ5Nu1atSCa3tawqSPJr7Ck24LuOVAKOGxpUY2H0o1iYTrI610TQzLV6iRP
xrN3AV9vZQ8E4inIYxMs3J/JdoQeAG676lNvI/CMZAkiLfV16QL/wrtAqu7pscMR
+nTrFwPcFHXACU5pXMXvda1K7p4hjgQ5tJvVVqVvlCWFyL4ysea6CPNXb9EU+XEO
IV9jPy1C5vyaeHClpyG/f3uQExXpnHb1un1UKXdTytQzzzvfUQtpfhX4m8Q4dBSe
cmFS+om1Ohe8kjTYxf9jknT43E7C1dsLBq5vLLVRUQC8Ctk1thnaTjpp22moU+Kv
kjZIy1hjUEUOPRfS3fMevKK/sRTpoGbvKFnCb4sbkUa+O1/jSNZ1zZqY0GeCls61
UgLcEY4UxBDNGV6gH5YJSfz3YMUMHheYBszqA45gZ0pT4fV0Y3rvtA==
-----END CERTIFICATE-----
subject=/CN=letsencrypt-nginx-proxy-companion
issuer=/CN=letsencrypt-nginx-proxy-companion
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1974 bytes and written 269 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 4C54DA293B694200A8E8928735D2EF676763FD2D1E525EE294BD0F459189A7D5
    Session-ID-ctx: 
    Master-Key: F2231FF397683E5873DD050284864933C5828CEF60D740FEC60B85E531418734D59E4F8A66039D14AC579E4C39C4A497
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1552942358
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: yes
---
{"id": 1, "method": "server.version", "params": ["testing", "1.4"]}
HTTP/1.1 400 Bad Request
Server: nginx/1.14.1
Date: Mon, 18 Mar 2019 20:52:44 GMT
Content-Type: text/html
Content-Length: 173
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.14.1</center>
</body>
</html>
read:errno=0

[Kexkey]

Yes, I am using an nginx instance in front of the server to deal with the certificate creation/renewal. So you are telling me that I can't use an nginx server to proxy the requests to the electrum server? It only acts as a reverse proxy.

[SomberNight]

The electrum protocol is not HTTP based, so I don't understand what an HTTP proxy in the middle of a connection can do.

"tcp" in electrum means: JSON-RPC over TCP

"ssl" in electrum means: JSON-RPC over TLS over TCP
note that it is NOT JSON-RPC over HTTP over TLS over TCP

[Kexkey]

Hello @SomberNight! Good catch, the problem was my nginx proxy config locked to HTTP. I reconfigured it so it serves TCP requests (as well as takes care of the letsencrypt certs) and now it works.

The error message about the SSL certificate not being verified (or anything related to SSL) was very misleading. The error was actually receiving a 400 Bad Request is not an expected or valid response for Electrum Wallet.

Thank you very much for your help and patience. Hopefully this issue helps other people in the same situation.

[adamgall]

@Kexkey mind sharing that nginx config?

[Kexkey]

I'm sure there are better ways to do it, but you can have a look at my repo here: https://github.com/SatoshiPortal/electrs-deploy

• I run electrs instead of electrum server
• I run the stack in Docker with these containers:
  • Bitcoin Core
  • nginx-proxy
  • letsencrypt-nginx-proxy-companion
  • electrs
  • Tor
• I didn't automate everything: I have a manual step at the end, once the LE certs are in, I modify the nginx config so that TLS/TCP is correctly configured, and then restart nginx.

Follow the README file and adapt the whole thing to your needs. See build.sh and nginx/bootstrap.sh for the interesting stuff.

Good luck!

[adamgall]

@Kexkey thank you 👍

[rafaelagp]

I'm having the exact same problem with a very similar setup.
Electrum Wallet log from running on Windows 10:

10T030550.625154Z | INFO | interface.[thoth:50002] | disconnecting due to: ConnectError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)
20200410T030550.625154Z | INFO | network | couldn't launch iface thoth:50002:s -- CancelledError()

I'm also running Electrum Rust Server and NGINX as reverse proxy on a Linux server, except my nginx.conf is already the same as @Kexkey 's and yet it does not work.

stream {
        upstream electrs {
                server thoth:50001;
        }

        server {
                listen 50002 ssl;
                proxy_pass electrs;

                ssl_certificate [certificate].crt;
                ssl_certificate_key [private].key;
                ssl_session_cache shared:SSL:1m;
                ssl_session_timeout 4h;
                ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
                ssl_prefer_server_ciphers on;
        }
}

Over plain HTTP it works fine but over SSL the wallet drops the connection. Any ideas?
The certificate I'm using is from SSL For Free.