self-hosted ngrokd - client fails with 'Certificate signed by unknown authority' on Mac OS X

[lyrixderaven]

I'm seeing an issue connecting an ngrok client to a self-hosted ngrokd. The ngrokd is being supplied with a valid key/crt pair, not self-signed (CACert signed, in fact), but the client running on OS X still fails to connect with:

[EROR] control recovering from failure x509: certificate signed by unknown authority

After some research I figured out that the crypto package for OS X does use the OS X keychain to look for fitting Root-CA's (and doesn't rely on a different certificate store, like, say, the OpenSSL store), so I added the Root-CA's to the KeyChain and trusted them. The effect remains the same.

To try a different approach, I also copied the Root-CA into the assets/client/tls directory, both leaving the same name and replacing the existing assets/client/tls/ngrokroot.crt file and then running both make client and make release-client, but that does not change anything either (although I can see that the release-client does recognize the additional .crt files as assets and adds them to the file assets-release.go). The certificate the server is supplying is part of a two-step chain (there is an intermediary certificate before the 'real' root-cert, I tried both there as well).

What am I doing wrong? Where else would ngrok go looking for that certificate? Can I supply the certificate to the client directly?

I'm really hoping someone can help me here - ngrok is really great and would really help with our development efforts, but not unless I can get it to run....

Thanks for any help!

[lyrixderaven]

From what I've deduced, it seems like the go crypto library doesn't find the Root CA I'm sending from the server. I've tried adding it to the OS X Keychain, compiling it into the client in various ways (as mentioned above), but I still can't get the client to accept the server's root ca. I'm stumped, please help me out - does anybody have any experience with GO and the OS X keychain?

[kk86bioinfo]

Refer to #84

[laike9m]

Same problem. Everyone is talking about self-signed CA, but according to doc, if I use a valid CA, I don't need to recompile but set path to cert in config file.

[longle255]

I got same problem with signed CA by letsencrypt, Mac's client. The problem seemed to be from the rootCA file. I've managed to solve it by copying letsencrypt's signed CA to assets/client/tls/ngrokroot.crt (overwrite the ngrokroot.crt file). Then recompiling server and client would finish it.

[laike9m]

@longle255 Check if you're using ngrok server 1.x, turned out my problem was due to using ngrok server 2.x and client 1.x.

[noodlebreak]

@longle255 Can you detail out the steps you took to use letsencrypt's certs for ngrokd?
Or if you could point out the fault in my steps, that would be highly helpful:

Here are my letsencrypt domain key/cert files:

[ec2-user@ip-172-XX-XX-XX bin]$ sudo ls -l /etc/letsencrypt/live/noodlebreak.org/total 12
lrwxrwxrwx 1 root root   45 Mar  1 05:07 cert.pem -> ../../archive/noodlebreak.org/cert1.pem
lrwxrwxrwx 1 root root   46 Mar  1 05:07 chain.pem -> ../../archive/noodlebreak.org/chain1.pem
lrwxrwxrwx 1 root root   50 Mar  1 05:07 fullchain.pem -> ../../archive/noodlebreak.org/fullchain1.pem
lrwxrwxrwx 1 root root   48 Mar  1 05:07 privkey.pem -> ../../archive/noodlebreak.org/privkey1.pem

After getting the certificates from letsencrypt, I did this (overwriting default ngrokroot.crt):
sudo cp /etc/letsencrypt/live/noodlebreak.org/cert.pem ../assets/client/tls/ngrokroot.crt

I'm running the daemon as:

sudo ./ngrokd -tlsKey=/etc/letsencrypt/live/noodlebreak.org/privkey.pem -tlsCrt=/etc/letsencrypt/live/noodlebreak.org/cert.pem -domain="noodlebreak.org" -httpAddr=":8080" -httpsAddr=":8443"

Also, my client is version 1.7, the server is 1.0, since it says in the README here that v2.0 is not open source.

[longle255]

@noodlebreak you should replace the cert file of ngrok by the the rootCA file of letsencrypt from their website (https://letsencrypt.org/certificates/), particularly this one https://letsencrypt.org/certs/letsencryptauthorityx1.pem instead of using the generated file, then recompiling ngrok server and client would help.

[noodlebreak]

@longle255 But then would I have to change the tlsKey and tlsCrt arguments when running the server daemon?

[longle255]

no, just keep the tlsKey and tlsCrt point to the ones that you generated

[noodlebreak]

@longle255 Actually, that's one thing I'm confused about. letsencrypt generates these:

lrwxrwxrwx 1 root root   45 Mar  1 05:07 cert.pem -> ../../archive/noodlebreak.org/cert1.pem
lrwxrwxrwx 1 root root   46 Mar  1 05:07 chain.pem -> ../../archive/noodlebreak.org/chain1.pem
lrwxrwxrwx 1 root root   50 Mar  1 05:07 fullchain.pem -> ../../archive/noodlebreak.org/fullchain1.pem
lrwxrwxrwx 1 root root   48 Mar  1 05:07 privkey.pem -> ../../archive/noodlebreak.org/privkey1.pem

So is it correct if I map tlsKey to privkey.pem and tlsCrt to cert.pem? Because that's what I'm doing right now.

[longle255]

you're doing correctly, in term of mapping tlskey and tlscrt. The step need to be altered is replacing the part /etc/letsencrypt/live/noodlebreak.org/cert.pem in sudo cp /etc/letsencrypt/live/noodlebreak.org/cert.pem ../assets/client/tls/ngrokroot.crt with the one taken from letsencrypt site

[noodlebreak]

@longle255 Thanks a tonne, that worked perfectly.
@inconshreveable I want to know more about the trust_host_root_certs: false that we have to put the in a configuration file.

• Q1: Why does is it have to be false?
• Q2: If I set it to true, I see in the ngrok server logs this:
  Failed to read message: remote error: bad certificate

And on the client side, I keep seeing this:
Tunnel Status reconnecting
With no error provided. And obviously, the client cannot connect. If I use the configuration with trust_host_root_certs: false, then I can't access my tunneled servers on https. I get in Chromium:

SSL connection error
ERR_SSL_PROTOCOL_ERROR`

EDIT: I forgot it is explained here:
https://github.com/inconshreveable/ngrok/blob/master/docs/SELFHOSTING.md

[unstatusthequo]

@noodlebreak I'm in the same boat as you. I'm a big unclear on the server setup prior to compiling. Am I correct with this, @longle255 ?

SERVER:
Is it correct to touch nothing in /ngrok/assets/server/tls/ ?

CLIENT

1. sudo cp letsencryptauthorityx1.pem /ngrok/assets/client/tls/ngrokroot.crt (per @longle255's suggestion)
2. sudo cp fullchain.pem /ngrok/assets/client/tls/snakeoilca.crt (not sure if fullchain.pem is actually correct here since I'm having it replace snakeoilca.crt)

Then, I compiled the server side with sudo make release-server for Ubuntu 14.04.
Then, I compiled the client side with sudo GOOS=windows GOARCH=amd64 make release-client
Then, I copy the resulting ngrok.exe to my Windows client machine.

I have made sure to setup my config file to accompany the .exe:
server_addr: mydomain.com:8443
trust_host_root_certs: false

I use this to start the server on Ubuntu 14.04:
sudo ./bin/ngrokd -tlsKey=/etc/letsencrypt/live/mydomain.com/privkey.pem -tlsCrt=/etc/letsencrypt/live/mydomain.com/cert.pem -domain="mydomain.com" -httpAddr=":8080" -httpsAddr=":8443"

I kick off the Windows client with:
ngrok.exe -proto="https" -config="ngrok.cfg" 65163

When I do this, I get the following on the server side:

[03/04/16 12:26:07] [WARN] [pub:2a9393e1] Failed to read valid https request: malformed HTTP request "\u007f\x00\x00\x00\x00\x00\x00\x00{\"Type\":\"Auth\",\"Payload\":{\"Version\":\"2\",\"MmVersion\":\"1.7\",\"User\":\"\",\"Password\":\"\",\"OS\":\"windows\",\"Arch\":\"amd64\",\"ClientId\":\"\"}}" [03/04/16 12:26:07] [DEBG] [pub:2a9393e1] Closing

Any ideas how I'm screwing this up? In Firefox, addressing https://mydomain.com:8443 results in this response Tunnel richardlutk.us:8443 not found with the server running. I understand this to be validity of the server working.

[noodlebreak]

@unstatusthequo what is the number at the end of your command for ?

Also, please open up logging on your client side to see what goes on in that end.
Use -log-level=DEBUG -log=stdout arguments.

[unstatusthequo]

That's the port I want opened on localhost that the server routes to.

I think I may have some idea of what's going on. So the same server I have this on is running vhosts on Apache. My sense is I really need the version 2 feature of rewriting headers... my sense is that's what's screwing up the headers.

@inconshreveable , any word on adding some of the creature comfort features of 2.0 to OSS? I'm dead in the water without it, apparently.