Is Certmagic suitable for my use case of adding subdomains pointed to a reverse proxy?

[austincollinpena]

Apologies if this is a duplicate of "Does certmagic have any limitations?"

Here's my use case:

I have a "distributed" reverse proxy/ specialized cache hosted in a few locations across the world that points to an origin server.

1, I'll ask my customers to point a subdomain (or main domain) to my SaaS (both A and AAAA records). This should satisfy the bolded requirement in the docs.

2. For my implementation of certmagic, I'll use on Demand TLS with:

• Use a custom decisionFunc that grabs the domain from my backend and verifies it's a customer of mine
• Write some sort of fancy certmagic.DefaultStorage to either grab the certificate from storage, or fetch it from a REST endpoint.

I'm 99% sure this will work for me, I'm just looking for verification.

Also, does this enable HTTP2?

[francislavoie]

Yup, that's a pretty typical usecase for On-Demand TLS.

Any reason for not using Caddy directly? Seems like it could do this for you out of the box (i.e. no need to write code).

If you do have custom logic, you could implement your proxy handling as a HTTP middleware module in Caddy and get all the extra benefits that Caddy brings at the same time.

[austincollinpena]

Thanks so much for the response!

@francislavoie I had some issues when it came to a few things, is the following possible? If so, do you mind telling me where to look?

1. Use a persistent cache of "rules" for different incoming domains. IE one URL might need to be redirected for a given time.
2. Add custom headers
3. Rewrite the request URL and query parameters based on URL (stateless, just redirecting media files elsewhere)

I'm sure I could figure out 2&3 quickly enough, but what about #1? Is that something I should use?

On top of that, is there any other benefit to using Caddy besides ease of use? I'm currently using Go's built in reverse proxy.

[francislavoie]

2 & 3 are solved by the header directive and the rewrite directive respectively, but 1 isn't something Caddy does out of the box.

Extending Caddy is very easy though: https://caddyserver.com/docs/extending-caddy

Caddy's reverse_proxy is very powerful, has a ton of great features you'd want for load balancing. Caddy can allow you to enable gzip encoding, serve static files easily, etc. It can do pretty much anything you need. And if it's missing something, you can write a plugin.

[austincollinpena]

@francislavoie Where should I start for #1? How should I persist the data? Just with a custom function to either read or get from the caddy cache?

Also, when it comes to multiple client connections and http2, does Caddy enable that?

[francislavoie]

https://github.com/nicolasazrak/caddy-cache is a Caddy v1 plugin, so it won't work here (v1 plugins aren't compatible with v2).

You can use whatever caching strategy you want, I don't know your business requirements here. You say "for a given time", but what enables this rule for example? How do you configure the amount of time? Etc.

Caddy's request matching is very powerful, you can even use CEL expressions which might allow you do to what you need in terms of time-based conditions if that's important. I'm not sure how much would be gained by caching there though, for simple redirects there's hardly any benefit to caching that type of rule.

Caddy does give you HTTP2 out of the box, but I think Go's stdlib server does as well if you have TLS configured (don't quote me on this, I don't know the specifics there, @mholt could confirm) Edit: Yeah I think that's the case, if you start up a net/http server in Go with TLS configured, you'll just get http2 for free.

[austincollinpena]

I will definitely look into Caddy again then.

"if you have TLS configured." That's my understanding as well. My last remaining question, and thank you so much for your thoroughness, is if running certmagic.HTTPS uses TLS under the hood, enabling HTTP2

[francislavoie]

Feel free to ask on the community forums if you need more specific help with Caddy

https://caddy.community

[mholt]

I agree with everything Francis said here.

Go's defaults for HTTPS requests is to negotiate HTTP/2 but if you mess with the transport or TLS config structs, all bets are off (the Go docs are specific about what disables HTTP/2, so you can know for sure).

So yeah, I'd recommend just using Caddy for this, you won't have to rewrite a bunch of the same logic yourself.

[austincollinpena]

Thank you @mholt

[austincollinpena]

One last question for you guys:

I'm trying to set up the OnDemandConfig and am having issues.

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", test)
	certmagic.Default.OnDemand = new(certmagic.OnDemandConfig)
	certmagic.DefaultACME.Agreed = true
	certmagic.DefaultACME.Email = "myemail"
	certmagic.DefaultACME.CA = certmagic.LetsEncryptStagingCA
	s := &http.Server{
		ReadHeaderTimeout: 20 * time.Second,
		ReadTimeout:       10 * time.Second,
		WriteTimeout:      30 * time.Second,
		Handler:           mux,
	}
	ln, err := tls.Listen("tcp", ":443", certmagic.Default.TLSConfig())
	if err != nil {
		panic(err)
	}
	err = s.Serve(ln)
	if err != nil {
		panic(err)
	}
}

When running this in a docker container and trying to make a request from https://localhost I get a null pointer error:

Error here

net/http.(*conn).serve.func1(0xc0000a1680) /usr/local/go/src/net/http/server.go:1800 +0x139 panic(0x7cb5a0, 0xb5dc30) /usr/local/go/src/runtime/panic.go:975 +0x3e3 github.com/caddyserver/certmagic.(*Cache).getAllMatchingCerts(0x0, 0xc0000a69f0, 0x9, 0x0, 0x0, 0x0) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/cache.go:267 +0x4a github.com/caddyserver/certmagic.(*Config).selectCert(0xb64d80, 0xc0000f6780, 0xc0000a69f0, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:173 +0x8e github.com/caddyserver/certmagic.(*Config).getCertificate(0xb64d80, 0xc0000f6780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:120 +0x3be github.com/caddyserver/certmagic.(*Config).getCertDuringHandshake(0xb64d80, 0xc0000f6780, 0x6d5adb0e80101, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:225 +0x7a github.com/caddyserver/certmagic.(*Config).GetCertificate(0xb64d80, 0xc0000f6780, 0xc000093958, 0x40bfd6, 0xc0000f6780) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:70 +0x101 crypto/tls.(*Config).getCertificate(0xc0000ac300, 0xc0000f6780, 0xc0000f6780, 0x0, 0x5) /usr/local/go/src/crypto/tls/common.go:870 +0x41a crypto/tls.(*serverHandshakeStateTLS13).pickCertificate(0xc000125ac8, 0x0, 0x0) /usr/local/go/src/crypto/tls/handshake_server_tls13.go:364 +0x74 crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000125ac8, 0xc00011f200, 0x0) /usr/local/go/src/crypto/tls/handshake_server_tls13.go:52 +0x79 crypto/tls.(*Conn).serverHandshake(0xc00017ca80, 0x453309, 0x55bd2f3e19d3) /usr/local/go/src/crypto/tls/handshake_server.go:53 +0x1b0 crypto/tls.(*Conn).Handshake(0xc00017ca80, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1342 +0x1e1 net/http.(*conn).serve(0xc0000a1680, 0x8d7780, 0xc0000874d0) /usr/local/go/src/net/http/server.go:1816 +0x19d created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2962 +0x35c 4:43:25 app | 2020/09/04 04:43:25 http: panic serving 172.17.0.1:59856: runtime error: invalid memory address or nil pointer dereference goroutine 31 [running]: net/http.(*conn).serve.func1(0xc0000a1720) /usr/local/go/src/net/http/server.go:1800 +0x139 panic(0x7cb5a0, 0xb5dc30) /usr/local/go/src/runtime/panic.go:975 +0x3e3 github.com/caddyserver/certmagic.(*Cache).getAllMatchingCerts(0x0, 0xc0000a6a90, 0x9, 0x0, 0x0, 0x0) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/cache.go:267 +0x4a github.com/caddyserver/certmagic.(*Config).selectCert(0xb64d80, 0xc0000f6840, 0xc0000a6a90, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:173 +0x8e github.com/caddyserver/certmagic.(*Config).getCertificate(0xb64d80, 0xc0000f6840, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:120 +0x3be github.com/caddyserver/certmagic.(*Config).getCertDuringHandshake(0xb64d80, 0xc0000f6840, 0x2352fc5b40101, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:225 +0x7a github.com/caddyserver/certmagic.(*Config).GetCertificate(0xb64d80, 0xc0000f6840, 0xc000093958, 0x40bfd6, 0xc0000f6840) /go/pkg/mod/github.com/caddyserver/certmagic@v0.11.2/handshake.go:70 +0x101 crypto/tls.(*Config).getCertificate(0xc0000ac300, 0xc0000f6840, 0xc0000f6840, 0x0, 0x5) /usr/local/go/src/crypto/tls/common.go:870 +0x41a crypto/tls.(*serverHandshakeStateTLS13).pickCertificate(0xc000125ac8, 0x0, 0x0) /usr/local/go/src/crypto/tls/handshake_server_tls13.go:364 +0x74 crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc000125ac8, 0xc00011f400, 0x0) /usr/local/go/src/crypto/tls/handshake_server_tls13.go:52 +0x79 crypto/tls.(*Conn).serverHandshake(0xc00017ce00, 0x453309, 0x55bd2f778ae7) /usr/local/go/src/crypto/tls/handshake_server.go:53 +0x1b0 crypto/tls.(*Conn).Handshake(0xc00017ce00, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1342 +0x1e1 net/http.(*conn).serve(0xc0000a1720, 0x8d7780, 0xc0000874d0) /usr/local/go/src/net/http/server.go:1816 +0x19d created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2962 +0x35c crypto/tls.(*Conn).Handshake(0xc00017ce00, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1342 +0x1e1 net/http.(*conn).serve(0xc0000a1720, 0x8d7780, 0xc0000874d0) /usr/local/go/src/net/http/server.go:1816 +0x19d created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2962 +0x35c

Any advice would be greatly appreciated.

[mholt]

@austincollinpena Check the godoc for certmagic.Default:

Default contains the package defaults for the various Config fields. This is used as a template when creating your own Configs with New(), and it is also used as the Config by all the high-level functions in this package.

The fields of this value will be used for Config fields which are unset. Feel free to modify these defaults, but do not use this Config by itself: it is only a template. Valid configurations can be obtained by calling New() (if you have your own certificate cache) or NewDefault() (if you only need a single config and want to use the default cache). This is the only Config which can access the default certificate cache.

Calling certmagic.Default.TLSConfig() uses certmagic.Default directly (without construction), which is against the documented proper use. Always use New() or NewDefault() as the documentation describes.

[austincollinpena]

Missed something. Will update with another comment

[austincollinpena]

For potential future visitors, here is my config that seemed to work for me. I will update it if it's a bad example:

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/test", test)
	certmagic.Default.OnDemand = new(certmagic.OnDemandConfig)
	certmagic.DefaultACME.CA = certmagic.LetsEncryptStagingCA
	certmagic.DefaultACME.Email = ""
	certmagic.DefaultACME.Agreed = true
	magic := certmagic.NewDefault()
	s := &http.Server{
		ReadHeaderTimeout: 20 * time.Second,
		ReadTimeout:       10 * time.Second,
		WriteTimeout:      30 * time.Second,
		Handler:           mux,
	}
	ln, err := tls.Listen("tcp", ":443", magic.TLSConfig())
	if err != nil {
		panic(err)
	}
	err = s.Serve(ln)
	if err != nil {
		panic(err)
	}
}

[mholt]

That looks better. The key is certmagic.NewDefault(), which is one of the ways (the easiest) to get a valid, initialized config.

I hate that that's required, but I really struggled to find any other way to do it.

[imedadel]

This looks exactly like the config I need, except that I also need to support one wildcard subdomain. My setup is pretty much similar to Hyperping's. How do I go about this?

[mholt]

@imedadel Did you try magic.Manage([]string{"example.com", "*.example.com"})? (or ManageAsync)

[imedadel]

Did you mean ManageSync instead of Manage? I added magic.ManageSync([]string{"example.com", "*.example.com"}) but I have to add a DNS provider now, is that right?

According to the readme, I should add (or any other provider)

certmagic.DefaultACME.DNS01Solver = &certmagic.DNS01Solver{
	DNSProvider: cloudflare.Provider{
		APIToken: "topsecret",
	},
}

However, DNS01Solver doesn't seem to exist on neither DefaultACME nor certmagic 😕 So I tried

certmagic.DefaultACME.DNSProvider = cloudflare.Provider{
		APIToken: "hoho",
	}

But I'm getting cloudflare.Provider does not implement challenge.Provider (missing CleanUp method).

Am I doing something wrong?

[mholt]

Did you mean ManageSync instead of Manage?

Yes, my bad.

but I have to add a DNS provider now, is that right?

If you're using Let's Encrypt. They require DNS challenge for wildcard certificates.

Make sure to use the latest versions committed (not the latest tagged version), we're going through a transition right now.

In the future please open a new issue for your questions, rather than commenting on closed issues. Thanks!

[imedadel]

Oooh. My bad. Can you please mention that in the readme, please 😄
Sorry for not opening a new issue!
Thank you!