Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2: if custom TLS certificate is set, Let's Encrypt sites break #3004

Closed
theel0ja opened this issue Jan 26, 2020 · 14 comments
Closed

v2: if custom TLS certificate is set, Let's Encrypt sites break #3004

theel0ja opened this issue Jan 26, 2020 · 14 comments
Labels
bug

Comments

@theel0ja
Copy link

@theel0ja theel0ja commented Jan 26, 2020

example.com {
        root * /var/www/example.com/public
        file_server

       tls /etc/ssl/certs/example.com.pem /etc/ssl/private/example.com.key
}

cdn.example.com {
       root * /var/www/cdn.example.com/public
       file_server

       tls /etc/ssl/certs/example.com.pem /etc/ssl/private/example.com.key
}

example.org {
	respond "example.org"
}

In this case, example.com and cdn.example.com work, but example.org returns error message in browsers (in Firefox, SSL_ERROR_INTERNAL_ERROR_ALERT) and into Caddy's logs.

If I remove the example.com blocks, example.org works.

caddy[1228]: 2020/01/26 02:22:18 http: TLS handshake error from CLIENTIP:CLIENTPORT: no server TLS configuration available for ClientHello: &{CipherSuites:[19018 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53 10] ServerName:example.org SupportedCurves:[56026 29 23 24] SupportedPoints:[0] SignatureSchemes:[1027 2052 1025 1283 2053 1281 2054 1537 513] SupportedProtos:[h2 http/1.1] SupportedVersions:[43690 772 771 770 769] Conn:0xc0000f2028}
$ caddy version
v2.0.0-beta.13 h1:QL0JAepFvLVtOatABqniuDRQ4HmtvWuuSWZW24qVVtk=
@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Jan 26, 2020

Yes, this is exactly the same that happend to me: https://caddy.community/t/v2-automatic-https-certificate-errors/6847/21

@francislavoie francislavoie added the v2 label Jan 26, 2020
@mholt mholt added the bug label Feb 3, 2020
@mholt mholt added this to the v2.0.0-beta14 milestone Feb 6, 2020
mholt referenced this issue Feb 6, 2020
If user provides their own certs or makes any hostname-specific TLS
connection policy, it means that no TLS connection would be served for
any other hostnames, even though you'd expect that TLS is enabled for
them, too. So now we append a catch-all conn policy if none exist, which
allows all ClientHellos to be matched and served.

We also fix the consolidation of automation policies, which previously
gobbled up automation policies without hosts in favor of automation
policies with hosts. Instead of a host-specific policy eating up an
identical catch-all policy, the catch-all policy eats up the identical
host-specific policy, ensuring that the policy is applied to all hosts
which need it.

See also:
https://caddy.community/t/v2-automatic-https-certificate-errors/6847/9?u=matt
@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 6, 2020

I believe I've fixed this in 4a07a5d -- please pull the latest from the v2 branch and try again, if you would!

@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Feb 18, 2020

I've just rebuild Caddy v2 with latest master (including 57c6f22) and I get panics again, when 1. using LE sites only in Caddyfile, and 2. in combination with sites with specified certificate paths in Caddyfile.

Go version 1.13.8 and System is: Debian 10 x86_64.

I tried only specifying one hostname with specified certificate path, and latest master of Caddy v2 works fine with it:
Caddyfile:

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}

Logs:

./caddy run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/18 10:09:54.745 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/18 10:09:54.747 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/18 10:09:54.747 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/18 11:09:54 [INFO][cache:0xc00049a000] Started certificate maintenance routine
2020/02/18 10:09:54.766 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "hnrk.io", "server_name": "srv0"}
2020/02/18 10:09:54.766 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "www.hnrk.io", "server_name": "srv0"}
2020/02/18 10:09:54.766 INFO    http    enabling automatic TLS certificate management   {"domains": []}
2020/02/18 10:09:54.779 INFO    tls     cleaned up storage units
2020/02/18 10:09:54.779 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2020/02/18 10:09:54.779 INFO    serving initial configuration
^C2020/02/18 10:10:14.846       INFO    shutting down   {"signal": "SIGINT"}

When however specifying a single LE site, Caddy breaks:
Caddyfile:

harmoniks.de, www.harmoniks.de {
        tls henrik@hnrk.io
        root * /etc/caddy/html/harmoniks.de
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Logs:

./caddy run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/18 10:09:28.182 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/18 10:09:28.186 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/18 10:09:28.186 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2020/02/18 10:09:28.186 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/18 11:09:28 [INFO][cache:0xc0001fa6e0] Started certificate maintenance routine
panic: reflect: call of reflect.Value.FieldByName on ptr Value

goroutine 1 [running]:
reflect.flag.mustBe(...)
        /root/go1.13.8/go/src/reflect/value.go:208
reflect.Value.FieldByName(0xf7b7c0, 0xc0006a4b78, 0x196, 0x10d708d, 0xd, 0x196, 0x7d445c, 0xc000480380)
        /root/go1.13.8/go/src/reflect/value.go:890 +0x1ed
github.com/caddyserver/caddy/v2.Context.LoadModule(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf02720, 0xc0006a4b78, 0x10d708d, ...)
        /etc/buildv2caddy/v2caddy/context.go:131 +0x102
github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).Provision(0xc0002a6fa0, 0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0x4, 0x4)
        /etc/buildv2caddy/v2caddy/modules/caddytls/tls.go:92 +0xb49
github.com/caddyserver/caddy/v2.Context.LoadModuleByID(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xc00035daf4, 0x3, 0xc0002a43f0, ...)
        /etc/buildv2caddy/v2caddy/context.go:322 +0x66a
github.com/caddyserver/caddy/v2.Context.loadModuleMap(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf00899, 0x0, 0xf8d9c0, ...)
        /etc/buildv2caddy/v2caddy/context.go:261 +0x260
github.com/caddyserver/caddy/v2.Context.loadModulesFromSomeMap(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf00899, 0x0, 0x0, ...)
        /etc/buildv2caddy/v2caddy/context.go:223 +0x107
github.com/caddyserver/caddy/v2.Context.LoadModule(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xf0e520, 0xc0006c5b80, 0x10b2a05, ...)
        /etc/buildv2caddy/v2caddy/context.go:198 +0xcce
github.com/caddyserver/caddy/v2.run.func3(0x12c76a0, 0xc0003bc780, 0xc000155500, 0xc0006c5b80, 0x0, 0x0, 0x0, 0xc0006c5b80, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/caddy.go:387 +0x98
github.com/caddyserver/caddy/v2.run(0xc0006c5b80, 0x501, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/caddy.go:395 +0x28e
github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun(0xc0002eb800, 0x59a, 0x600, 0x7, 0xc000154630)
        /etc/buildv2caddy/v2caddy/caddy.go:249 +0xf7
github.com/caddyserver/caddy/v2.changeConfig(0x10a9066, 0x4, 0x10b27ff, 0x7, 0xc0002ea000, 0x59a, 0x600, 0x423001, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/caddy.go:156 +0x4ce
github.com/caddyserver/caddy/v2.Load(...)
        /etc/buildv2caddy/v2caddy/caddy.go:102
github.com/caddyserver/caddy/v2/cmd.cmdRun(0xc0000ca7e0, 0x0, 0x0, 0x0)
        /etc/buildv2caddy/v2caddy/cmd/commandfuncs.go:186 +0x352
github.com/caddyserver/caddy/v2/cmd.Main()
        /etc/buildv2caddy/v2caddy/cmd/main.go:72 +0x23e
main.main()
        /etc/buildv2caddy/v2caddy/cmd/caddy/main.go:37 +0x20
mholt added a commit that referenced this issue Feb 18, 2020
When AutomationPolicy was turned into a pointer, we continued passing
a double pointer to LoadModule, oops.
@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 18, 2020

Oops, thanks for the full stack trace @Henrocker that made it quick to find the problem. Fixed, I think, please try again!

@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Feb 19, 2020

@mholt Thanks, that's fixed the panic!

I've recompiled new Caddyv2 including latest commit 0b09b07, if I now put a single domain in Caddyfile with specified certificate path, I get the following logs:

./caddy run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/19 07:19:29.539 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/19 07:19:29.543 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/19 07:19:29.543 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/19 08:19:29 [INFO][cache:0xc000564000] Started certificate maintenance routine
2020/02/19 08:19:29 [INFO][cache:0xc000564000] Stopped certificate maintenance routine
run: loading initial config: loading new config: loading app modules: module name 'tls': provision tls: loading TLS automation management module: module name 'load_files': decoding module config: tls.certificates.load_files: json: cannot unmarshal array into Go value of type caddytls.CertKeyFilePair

Caddyfile:

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}

Thank you for your help!

mholt added a commit that referenced this issue Feb 20, 2020
Also only append 1 catch-all TLS connection policy to a server, even if
multiple site blocks contribute to that server.
@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 20, 2020

@Henrocker Tricky, tricky... fixed in 0005e3a, I believe. Sigh. Kind of hacky but that's how it's going to be I guess. I still maintain that the Caddyfile adapter is the hardest part of building Caddy 2.

@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Feb 20, 2020

@mholt Thanks, with including 0005e3a, I can now run Caddy v2 using my Caddyfile, however, I think, we're not quite there, yet. 😅

I've tried several config combinations with my Caddyfile and see the following behaviour:

  • LE sites work perfectly fine, even in combination with sites with specified certificate in Caddyfile.
  • Having One single vhost-block with dedicated specified certificate (and possibly several mutliple LE sites together in Caddyfile) also works fine, BUT:

With two or more vhost blocks that point to dedicated certificate files, Caddy v2 always throws an SSL_Protocol error on the first vhost (so on (www.)hnrk.io, I receive SSL_Protocol errors, but draw.hnrk.io loads just fine). Config looks like this when it happens:

hnrk.io, www.hnrk.io {
        root * /etc/caddy/html
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server /md* browse
        file_server
}
draw.hnrk.io {
        root * /etc/caddy/html/draw
        tls /etc/caddy/hnrk.io.crt /etc/caddy/hnrk.io.key
        encode brotli zstd gzip
        php_fastcgi unix//run/php/php7.3-fpm.sock
        try_files {path} /index.php?{query}
        file_server
}

Logs:

./caddyOLD1 run --config /etc/v2caddy/CaddyfileOLD1 --adapter caddyfile
2020/02/20 09:03:48.436 INFO    using provided configuration    {"config_file": "/etc/v2caddy/CaddyfileOLD1", "config_adapter": "caddyfile"}
2020/02/20 09:03:48.440 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/20 09:03:48.440 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/02/20 10:03:48 [INFO][cache:0xc0003f8050] Started certificate maintenance routine
2020/02/20 09:03:48.471 INFO    tls     cleaned up storage units
2020/02/20 09:03:48.472 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "draw.hnrk.io", "server_name": "srv0"}
2020/02/20 09:03:48.472 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "hnrk.io", "server_name": "srv0"}
2020/02/20 09:03:48.472 INFO    http    skipping automatic certificate management because one or more matching certificates are already loaded  {"domain": "www.hnrk.io", "server_name": "srv0"}
2020/02/20 09:03:48.472 INFO    http    enabling automatic TLS certificate management   {"domains": []}
2020/02/20 09:03:48.472 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2020/02/20 09:03:48.472 INFO    serving initial configuration
2020/02/20 10:04:15 http: TLS handshake error from 62.157.168.126:16859: no certificate available for 'hnrk.io'		<------------ That's the error!
^C2020/02/20 09:04:24.633       INFO    shutting down   {"signal": "SIGINT"}
2020/02/20 10:04:24 [INFO][cache:0xc0003f8050] Stopped certificate maintenance routine
2020/02/20 09:04:24.633 INFO    shutdown done   {"signal": "SIGINT"}

Do you know a way to debug (generate stack traces) out of static go binaries directly on *nix?

Thanks!

@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 20, 2020

@Henrocker What are all the names (exactly) on the certificate at /etc/caddy/hnrk.io.crt?

@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Feb 20, 2020

@mholt This is a wildcard certificate.

DNS-Name=*.hnrk.io
DNS-Name=hnrk.io

image

@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 20, 2020

@Henrocker Hmm, I can't reproduce it.

I generated a certificate just like yours (two SAN names, test-1.lightcodelabs.com and *.lightcodelabs.com) and then used your Caddyfile except loading my certificates and replacing your domains with mine. curl -v --insecure "https://test-1.lightcodelabs.com/"works just fine for all my domains and subdomains that I'm testing with. When I try with some other hostname, like curl -v --insecure "https://localhost/", then I see the no certificate available for 'localhost' error.

In other words, things are working as expected.

I suppose I'll need your help to further debug.

Can you put log.Printf() statements in certmagic/handshake.go (do a search for "no certificate available for") and just kind of see why it can't find your certificate?

Edit: Oooh, I found a way to reproduce it. Still, your help would be greatly appreciated if you can spare a few minutes! :)

@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Feb 20, 2020

@mholt I am happy to help, hang on!

@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 20, 2020

Thanks! I just found the problem... might be a little tricky to fix. I'll see about it today if I can.

Somehow I need to prevent cert files that are used repeatedly from being loaded twice:

"certificates": {
				"load_files": [
					{
						"certificate": "/Users/matt/Desktop/cert.pem",
						"key": "/Users/matt/Desktop/key.pem",
						"tags": [
							"cert0"
						]
					},
					{
						"certificate": "/Users/matt/Desktop/cert.pem",
						"key": "/Users/matt/Desktop/key.pem",
						"tags": [
							"cert1"
						]
					}
				]
			}

It's looking for cert0, even though cert1 overwrote cert0 in the cache.

mholt added a commit that referenced this issue Feb 20, 2020
See end of issue #3004. Loading the same certificate file multiple times
with different tags will result in it being de-duplicated in the in-
memory cache, because of course they all have the same bytes. This
meant that any certs of the same filename loaded with different tags
would be overwritten by the next certificate of the same filename, and
any conn policies looking for the tags of the previous ones would never
find them, causing connections to fail.

So, now we remember cert filenames and their tags, instead of loading
them multiple times and overwriting previous ones.

A user crafting their own JSON might make this error too... maybe we
won't see it happen. But if it does, one possibility is, when loading
a duplicate cert, instead of discarding it completely, merge the tag
list into the one that's already stored in the cache, then discard.
@mholt

This comment has been minimized.

Copy link
Member

@mholt mholt commented Feb 20, 2020

@Henrocker Fixed in 99f91c4 -- please test, thanks!

PS. Your use of actual domain names and not redacting any information really made this quick and easy. I hope more people will report bugs the way you do... too many people like to change and hide their configs before posting it, and we can't help them. Seriously it happens every day.

@Henrocker

This comment has been minimized.

Copy link

@Henrocker Henrocker commented Feb 20, 2020

@mholt Wow, that was really quick! Thanks, my complete setup works like a charm now.

I really feel the post scriptum part, and thanks to bcrypt for auth, one doesn't even have to look into Caddyfile again before posting it publicly, yay! :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.