TLS termination with SNI

[vk496]

Hello and thank you for this amazing project!

I'm trying to create a configuration where I use only one port for http, https, and custom TCP protocol of external apps (encapsulated with TLS and forwarded based on SNI).

{
	debug
	local_certs

	servers {
		listener_wrappers {
			http_redirect
			tls
		}
	}

	layer4 {
		:18000 {
			@app1 tls sni app1.internal
			route @app1 {
				tls
				proxy 192.168.1.159:3333
			}

			@secure tls
			route @secure {
				proxy localhost:443
			}

			@insecure http
			route @insecure {
				proxy localhost:443
			}
		}
	}
}

localhost:443 {
	tls internal {
		on_demand
	}

	# reverse_proxy google.com:80
	respond "Hello from the website!"
}

However, I'm not able to forward the traffic based on SNI. Reducing my config to:

{
	layer4 {
		:18000 {
			@app1 tls sni app1.internal
			route @app1 {
				proxy 192.168.1.159:3333
			}
		}
	}
}

If I open a ncat -l 3333 and run echo hello | openssl s_client -connect 127.0.0.1:18000 -servername app1.internal, I get garbage on the ncat terminal (which is expected). However, if I use:

{
	layer4 {
		:18000 {
			@app1 tls sni app1.internal
			route @app1 {
				tls
				proxy 192.168.1.159:3333
			}
		}
	}
}

And run the same command, I get:

$ echo hello | openssl s_client -connect 127.0.0.1:18000 -servername app1.internal
Connecting to 127.0.0.1
CONNECTED(00000003)
4027BB94637F0000:error:0A000438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error:ssl/record/rec_layer_s3.c:865:SSL alert number 80
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 321 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

and the debug messages are:

caddy-1  | {"level":"debug","ts":1724427725.8702772,"logger":"layer4","msg":"matching","remote":"172.19.0.1:59716","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
caddy-1  | {"level":"debug","ts":1724427725.8702905,"logger":"layer4","msg":"matching","remote":"172.19.0.1:59716","error":"consumed all prefetched bytes","matcher":"layer4.matchers.tls","matched":false}
caddy-1  | {"level":"debug","ts":1724427725.8702946,"logger":"layer4","msg":"matching","remote":"172.19.0.1:59716","error":"consumed all prefetched bytes","matcher":"layer4.matchers.http","matched":false}
caddy-1  | {"level":"debug","ts":1724427725.8703141,"logger":"layer4","msg":"prefetched","remote":"172.19.0.1:59716","bytes":321}
caddy-1  | {"level":"debug","ts":1724427725.870344,"logger":"layer4.matchers.tls","msg":"matched","remote":"172.19.0.1:59716","server_name":"app1.internal"}
caddy-1  | {"level":"debug","ts":1724427725.8703463,"logger":"layer4","msg":"matching","remote":"172.19.0.1:59716","matcher":"layer4.matchers.tls","matched":true}
caddy-1  | {"level":"debug","ts":1724427725.8705084,"logger":"events","msg":"event","name":"tls_get_certificate","id":"3d7c68ae-a4fc-407e-ab17-9bc40ead6fce","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"app1.internal","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2074,2075,2076,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"RemoteAddr":{"IP":"172.19.0.1","Port":59716,"Zone":""},"LocalAddr":{"IP":"172.19.0.2","Port":18000,"Zone":""}}}}
caddy-1  | {"level":"debug","ts":1724427725.8705153,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"app1.internal"}
caddy-1  | {"level":"debug","ts":1724427725.8705628,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.internal"}
caddy-1  | {"level":"debug","ts":1724427725.870565,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
caddy-1  | {"level":"debug","ts":1724427725.8705704,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"172.19.0.1","remote_port":"59716","server_name":"app1.internal","remote":"172.19.0.1:59716","identifier":"app1.internal","cipher_suites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
caddy-1  | {"level":"error","ts":1724427725.8706036,"logger":"layer4","msg":"handling connection","remote":"172.19.0.1:59716","error":"no certificate available for 'app1.internal'"}
caddy-1  | {"level":"debug","ts":1724427725.8706071,"logger":"layer4","msg":"connection stats","remote":"172.19.0.1:59716","read":321,"written":7,"duration":0.000339871}

Is this a bug? is intentional? There is any way to handle this correctly?

BR,
Valentin

[mohammed90]

Which commit are you running? I believe there was a recent fix for an SNI related issue

[mholt]

I think that's correct. You're terminating TLS which requires a certificate, and the client is connecting to app1.internal, but nothing in the Caddy config has told it to manage a cert for app1.internal AFAICT.

[vk496]

@mohammed90 Im running this as image:

FROM caddy:2-builder AS builder

RUN xcaddy build \
    --with github.com/mholt/caddy-l4

FROM caddy:2

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

@mholt Thnks for the suggestion. If I use this:

{
	debug
	local_certs

	layer4 {
		:18000 {
			@app1 tls sni app1.internal
			route @app1 {
				proxy localhost:11443
			}
		}
	}
}

:11443 {
	tls internal {
		on_demand
	}

	reverse_proxy 192.168.1.159:3333
}

Im able to terminate the TLS:

$ echo hello | openssl s_client -connect 127.0.0.1:18000 -servername app1.internal
Connecting to 127.0.0.1
CONNECTED(00000003)
depth=1 CN=Caddy Local Authority - ECC Intermediate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 
verify return:1
---
Certificate chain
 0 s:
   i:CN=Caddy Local Authority - ECC Intermediate
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Aug 23 16:44:27 2024 GMT; NotAfter: Aug 24 04:44:27 2024 GMT
 1 s:CN=Caddy Local Authority - ECC Intermediate
   i:CN=Caddy Local Authority - 2024 ECC Root
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Aug 23 16:44:24 2024 GMT; NotAfter: Aug 30 16:44:24 2024 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBwjCCAWigAwIBAgIRANkvABjLjIiSKTeo0Ys8im8wCgYIKoZIzj0EAwIwMzEx
MC8GA1UEAxMoQ2FkZHkgTG9jYWwgQXV0aG9yaXR5IC0gRUNDIEludGVybWVkaWF0
ZTAeFw0yNDA4MjMxNjQ0MjdaFw0yNDA4MjQwNDQ0MjdaMAAwWTATBgcqhkjOPQIB
BggqhkjOPQMBBwNCAAS57GDXE99pgMdygJ2ZWbQid4F8vxkGCudyU+XlJrldPnHP
Eh7ZIaTfK9Ljp63HZDJQ2Jc1l9KjCFVn8fUdC9vgo4GPMIGMMA4GA1UdDwEB/wQE
AwIHgDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFI2R
PNhP3qPvRqjMxiNHUveEarC5MB8GA1UdIwQYMBaAFNGYnaBUWSzBD7FIV4xcfn+m
OgAXMBsGA1UdEQEB/wQRMA+CDWFwcDEuaW50ZXJuYWwwCgYIKoZIzj0EAwIDSAAw
RQIhAIpiakhiVg8bf5qeoQpqASWJPyhPzDjn2QBJetKvW8vIAiBB9OC6HKBleWMB
4TJyd9nrRcTTZy/C5FEJQFwmmAPGkg==
-----END CERTIFICATE-----
subject=
issuer=CN=Caddy Local Authority - ECC Intermediate
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1272 bytes and written 385 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 921416C0230755A1DB1B8A4B6706FF1CBC2190618832A4BBE7A3D18BC580B209
    Session-ID-ctx: 
    Resumption PSK: FFFC786459EE8A77D0BAF01D985E87C2037E8FEE32071DE3C32BA24AECA7B47C
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - de 69 a7 0f 26 2f 59 5c-f0 69 e5 ef db 61 ff a5   .i..&/Y\.i...a..
    0010 - d5 f3 52 78 cf 23 4c 06-57 c4 d2 16 a3 13 f2 2a   ..Rx.#L.W......*
    0020 - 7b f0 2c ad 95 ea 3d ed-f3 bc 2f ea 33 00 7a 81   {.,...=.../.3.z.
    0030 - 37 f1 36 46 1b 32 c4 7c-70 de 4c 30 44 19 99 0c   7.6F.2.|p.L0D...
    0040 - 81 88 5d 5e 9a a7 d5 e7-41 80 cd c9 7e 0d 9b f0   ..]^....A...~...
    0050 - a1 7c e2 67 c3 f2 b9 95-4f 21 77 2d 2e b3 1b 39   .|.g....O!w-...9
    0060 - c6 42 79 49 70 36 d0 e1-6d                        .ByIp6..m

    Start Time: 1724431639
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
DONE

However, in the netcat window I don't receive any message, which make me think that it's actually terminating the connection there. Or Im missing something?

[mholt]

Glad it's working!

Is netcat sending an HTTP request? The HTTP proxy can't really write a response until it has received a request.

[vk496]

netcat is listening on the port 3333.

$ ncat -l 3333

And I'm sending the message hello wrapped with TLS:

$ echo hello | openssl s_client -connect 127.0.0.1:18000 -servername app1.internal

The idea behind is to encapsulate any TCP stream with TLS, since SNI tag would allow to route it to the appropiate host/app. So I'm not expecting to use HTTP messages there, only raw bytes wrapped by TLS

AFAIK, there are no TCP matchers based on custom flags. TLS is the lowest one I can think about

[mholt]

It's just that you're proxying the TLS connection (in the layer4 app) to an HTTP server which then proxies HTTP to 192.168.1.159:3333.

[vnxme]

@vk496, what if you try the following config:

{
	debug
	local_certs

	layer4 {
		:18000 {
			@app1 tls sni app1.internal
			route @app1 {
				# you have to terminate tls before proxy according to your use case
				tls
				proxy 192.168.1.159:3333
			}
		}
	}
}

# you need this block to have a valid certificate
app1.internal:11443 {
	tls internal {
		on_demand
	}

	respond "OK" 200
}

[vk496]

Amazing @vnxme ! That works as expected!

$ echo hello | openssl s_client -connect 127.0.0.1:18000 -servername app1.internal

[mholt]

@francislavoie Didn't you mention to me something like tls automate ... for the Caddyfile so users can specify a hostname to get a cert for without needing a site block?

Maybe we should do that.

[vk496]

I read somewhere in Caddy forums that it was local_certs. Thats why I was getting crazy to understand what was actually failing on my usecase

[francislavoie]

@mholt not quite, I was saying as a way to force the domains of that site to get automated and not skipped if covered by a wildcard.

Though we could have a global option to list some names to automate regardless (merged with list from sites) 🤷‍♂️