Failed to serve quic connection error🐛

[connors2015]

Describe the bug
Tunnel updated to 2024.1.3 and similar to 2024.1.1 broke the quic protocol

To Reproduce
Steps to reproduce the behavior:

1. Run 2024.1.3 tunnel in a kubernetes cluster

If it's an issue with Cloudflare Tunnel:
5. cloudflared config:

https://github.com/cloudflare/helm-charts/blob/main/charts/cloudflare-tunnel-remote/templates/deployment.yaml

Expected behavior
A clear and concise description of what you expected to happen.

Tunnel should function without error.

Environment and versions

• k8s version: 1.27.7
• node image: AKSUbuntu-2204gen2containerd-202401.03.0
• kernel version: 5.15.0-1053-azure

Logs and errors

2024-01-17T18:58:20Z INF Starting tunnel tunnelID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-17T18:58:20Z INF Version 2024.1.2
2024-01-17T18:58:20Z INF GOOS: linux, GOVersion: go1.21.5-devel-cf, GoArch: amd64
2024-01-17T18:58:20Z INF Settings: map[metrics:0.0.0.0:2000]
2024-01-17T18:58:20Z INF Environmental variables map[TUNNEL_TOKEN:*****]
2024-01-17T18:58:20Z INF Autoupdate frequency is set autoupdateFreq=86400000
2024-01-17T18:58:20Z INF Generated Connector ID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-17T18:58:21Z INF Initial protocol quic
2024-01-17T18:58:21Z INF ICMP proxy will use 10.30.0.210 as source for IPv4
2024-01-17T18:58:21Z INF ICMP proxy will use xxx::xxxx::xxxx::xxxx::xxxxx in zone eth0 as source for IPv6
2024-01-17T18:58:21Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 65532 is not between ping group 1 to 0"
2024-01-17T18:58:21Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2024-01-17T18:58:22Z INF Starting metrics server on [::]:2000/metrics
2024/01/17 18:58:22 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024-01-17T18:58:22Z INF cloudflared has been updated version=2024.1.3
2024-01-17T18:58:22Z INF Restarting service managed by SysV...
2024-01-17T18:58:22Z INF PID of the new process is 16
2024-01-17T18:58:22Z ERR Initiating shutdown error="cloudflared has been updated to version 2024.1.3"
2024-01-17T18:58:22Z ERR writing release: Application error 0x0 (local)
2024-01-17T18:58:22Z ERR Failed to serve quic connection error="context canceled" connIndex=0 event=0 ip=xxx.xxx.xxx.xxx
2024-01-17T18:58:22Z INF Tunnel server stopped
2024-01-17T18:58:22Z INF Starting tunnel tunnelID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-17T18:58:22Z INF Version 2024.1.3
2024-01-17T18:58:22Z INF GOOS: linux, GOVersion: go1.21.5, GoArch: amd64
2024-01-17T18:58:22Z INF Settings: map[metrics:0.0.0.0:2000]
2024-01-17T18:58:22Z INF Environmental variables map[TUNNEL_TOKEN:*****]
2024-01-17T18:58:22Z INF Autoupdate frequency is set autoupdateFreq=86400000
2024-01-17T18:58:22Z INF Generated Connector ID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-17T18:58:22Z INF Initial protocol quic
2024-01-17T18:58:22Z INF ICMP proxy will use 10.30.0.210 as source for IPv4
2024-01-17T18:58:22Z INF ICMP proxy will use xxxx::xxxx::xxxx::xxxx::xxxx in zone eth0 as source for IPv6
2024-01-17T18:58:22Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 65532 is not between ping group 1 to 0"
2024-01-17T18:58:22Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2024-01-17T18:58:22Z INF Starting metrics server on [::]:2000/metrics
2024/01/17 18:58:22 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024-01-17T18:58:22Z INF Registered tunnel connection connIndex=0 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=xxx.xxx.xxx.xxx location=iad05 protocol=quic
2024-01-17T18:58:22Z INF Metrics server stopped
cloudflared has been updated to version 2024.1.3

Additional context
The --no-autoupdate flag was also ignored in the deployment.yaml but that's another issue.

[tdack]

Fails here for v2024.1.2 and.3 ...

➜ cloudflared tunnel --no-autoupdate run --token xxxxxxxxxxxxx
2024-01-18T00:48:30Z INF Starting tunnel tunnelID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-18T00:48:30Z INF Version 2024.1.3
2024-01-18T00:48:30Z INF GOOS: linux, GOVersion: go1.21.5, GoArch: amd64
2024-01-18T00:48:30Z INF Settings: map[no-autoupdate:true token:*****]
2024-01-18T00:48:30Z INF Generated Connector ID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-18T00:48:30Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/run-tunnel/as-a-service/
2024-01-18T00:48:30Z INF Initial protocol quic
2024-01-18T00:48:30Z INF ICMP proxy will use xxx.xxx.xxx.xxx as source for IPv4
2024-01-18T00:48:30Z INF ICMP proxy will use :: as source for IPv6
2024-01-18T00:48:30Z INF Starting metrics server on 127.0.0.1:41443/metrics
2024-01-18T00:48:30Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.113
2024-01-18T00:48:30Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.200.113
2024-01-18T00:48:31Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.192.27
2024-01-18T00:48:31Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.27
2024-01-18T00:48:34Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.63
2024-01-18T00:48:34Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.200.63
2024-01-18T00:48:36Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.73
2024-01-18T00:48:36Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.200.73
2024-01-18T00:48:41Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.23
2024-01-18T00:48:41Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.200.23
2024-01-18T00:48:53Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.73
2024-01-18T00:48:53Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.73
2024-01-18T00:49:00Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.192.57
2024-01-18T00:49:00Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.57
2024-01-18T00:49:32Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.113
2024-01-18T00:49:32Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.113
2024-01-18T00:49:41Z ERR Failed to create new quic connection error="failed to dial to edge with quic: INTERNAL_ERROR (local): tls: CurvePreferences includes unsupported curve" connIndex=0 event=0 ip=198.41.200.13
2024-01-18T00:49:41Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13
2024-01-18T00:50:06Z INF Switching to fallback protocol http2 connIndex=0 event=0 ip=198.41.200.13
2024-01-18T00:50:06Z INF Registered tunnel connection connIndex=0 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.200.233 location=bru01 protocol=http2
2024-01-18T00:50:06Z INF Registered tunnel connection connIndex=1 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.192.227 location=ams08 protocol=http2
2024-01-18T00:50:07Z INF Registered tunnel connection connIndex=2 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.200.43 location=bru01 protocol=http2
2024-01-18T00:50:07Z INF Updated to new configuration config="{\"ingress\":.....}" version=1
2024-01-18T00:50:08Z INF Registered tunnel connection connIndex=3 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.192.107 location=ams06 protocol=http2

Reverting to 2023.10.0 and everything works ....

➜ ./cloudflared tunnel --no-autoupdate run --token xxxxxxxxx
2024-01-18T00:47:00Z INF Starting tunnel tunnelID=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-18T00:47:00Z INF Version 2023.10.0
2024-01-18T00:47:00Z INF GOOS: linux, GOVersion: go1.20.13, GoArch: amd64
2024-01-18T00:47:00Z INF Settings: map[no-autoupdate:true token:*****]
2024-01-18T00:47:00Z INF Generated Connector ID: xxxxxx-xxxx-xxxx-xxxx-xxxxxxx
2024-01-18T00:47:00Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/run-tunnel/as-a-service/
2024-01-18T00:47:00Z INF Initial protocol quic
2024-01-18T00:47:00Z INF ICMP proxy will use xxx.xxx.xxx.xxx as source for IPv4
2024-01-18T00:47:00Z INF ICMP proxy will use :: as source for IPv6
2024-01-18T00:47:00Z WRN Your version 2023.10.0 is outdated. We recommend upgrading it to 2024.1.3
2024-01-18T00:47:00Z INF Starting metrics server on 127.0.0.1:44809/metrics
2024-01-18T00:47:01Z INF Registered tunnel connection connIndex=0 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.200.23 location=bru01 protocol=quic
2024-01-18T00:47:02Z INF Registered tunnel connection connIndex=1 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.192.47 location=ams08 protocol=quic
2024-01-18T00:47:02Z INF Registered tunnel connection connIndex=2 connection=xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.200.73 location=bru01 protocol=quic
2024-01-18T00:47:03Z INF Updated to new configuration config="{\"ingress\": ... }" version=1
2024-01-18T00:47:04Z INF Registered tunnel connection connIndex=3 connection=6xxxxxx-xxxx-xxxx-xxxx-xxxxxxx event=0 ip=198.41.192.37 location=ams06 protocol=quic

Built from source on Debian Stretch 9.13.

[chungthuang]

Hi @connors2015, the issue is autoupdate is not supported in kubernetes. The reason is during autoupdate, cloudflared will spawn a new process and kill the old one. But from the perspective of kubernetes, the main process exit, so it thinks the pod needs to be restarted. I saw that you already discussed the no-autoupdate flag in another issue. Can you share the command used to run cloudflared in your kubernetes deployment? When you run cloudflared tunnel --no-autoupdate, it should show use this in the log:

INF Settings: map[loglevel:debug no-autoupdate:true token:*****]

[chungthuang]

Hi @tdack, your issue is different. The error is cloudflared was built with a go version that doesn't support post quantum curves. Where did you download cloudflared? Or did you build it from source?

[tdack]

Hi @tdack, your issue is different. The error is cloudflared was built with a go version that doesn't support post quantum curves. Where did you download cloudflared? Or did you build it from source?

My apologies, it looked similar. Cloudflared was built from source. Version 2023.10.0 using go-1.20.13. Version 2024.1.2 & .3 using go-1.21.5. Trying to build 2024.1.2 or .3 with go-1.20.13 gives an error about quic-go internal modules requiring go-1.21, eg:

# github.com/quic-go/quic-go/internal/qerr
vendor/github.com/quic-go/quic-go/internal/qerr/error_codes.go:42:13: undefined: tls.AlertError
note: module requires Go 1.21

I won't add any more comments here so as not to conflate two different issues in one.

[connors2015]

Hi @tdack, your issue is different. The error is cloudflared was built with a go version that doesn't support post quantum curves. Where did you download cloudflared? Or did you build it from source?

My apologies, it looked similar. Cloudflared was built from source. Version 2023.10.0 using go-1.20.13. Version 2024.1.2 & .3 using go-1.21.5. Trying to build 2024.1.2 or .3 with go-1.20.13 gives an error about quic-go internal modules requiring go-1.21, eg:

# github.com/quic-go/quic-go/internal/qerr
vendor/github.com/quic-go/quic-go/internal/qerr/error_codes.go:42:13: undefined: tls.AlertError
note: module requires Go 1.21

I won't add any more comments here so as not to conflate two different issues in one.

@tdack Your issue is the one I opened the issue for, the other was just a comment I tacked on. Please continue providing updates. @chungthuang if you can open the other ticket back up I can provide more information there regarding kubernetes.

[connors2015]

Hi @tdack, your issue is different. The error is cloudflared was built with a go version that doesn't support post quantum curves. Where did you download cloudflared? Or did you build it from source?

This is the error from 2024.1.3 which is from docker hub.

2024-01-17T20:15:52Z ERR Failed to create new quic connection error="failed to dial to edge with quic: context canceled" connIndex=2 event=0 ip=198.41.192.57
2024-01-17T20:15:52Z INF Registered tunnel connection connIndex=2 connection=xxxxxx-xxxxxx-xxxxxx-xxxxxxx event=0 ip=xxx.xxx.xxx.xxx location=iad02 protocol=quic
2024-01-17T20:15:53Z ERR Failed to create new quic connection error="failed to dial to edge with quic: context canceled" connIndex=3 event=0 ip=xxx.xxx.xxx.xxx
2024-01-17T20:15:53Z INF Registered tunnel connection connIndex=3 connection=xxxxxx-xxxxxx-xxxxxx-xxxxxxx event=0 ip=198.41.200.193 location=iad12 protocol=quic
2024-01-17T20:15:54Z INF Tunnel server stopped

[chungthuang]

@tdack your issue is because cloudflared needs to be built with a go tool chain that supports post quantum curves. Please checkout https://github.com/cloudflare/cloudflared?tab=readme-ov-file#installing-cloudflared for how to install the go tool chain and build from source.

[chungthuang]

Hi @tdack, your issue is different. The error is cloudflared was built with a go version that doesn't support post quantum curves. Where did you download cloudflared? Or did you build it from source?

This is the error from 2024.1.3 which is from docker hub.

2024-01-17T20:15:52Z ERR Failed to create new quic connection error="failed to dial to edge with quic: context canceled" connIndex=2 event=0 ip=198.41.192.57
2024-01-17T20:15:52Z INF Registered tunnel connection connIndex=2 connection=xxxxxx-xxxxxx-xxxxxx-xxxxxxx event=0 ip=xxx.xxx.xxx.xxx location=iad02 protocol=quic
2024-01-17T20:15:53Z ERR Failed to create new quic connection error="failed to dial to edge with quic: context canceled" connIndex=3 event=0 ip=xxx.xxx.xxx.xxx
2024-01-17T20:15:53Z INF Registered tunnel connection connIndex=3 connection=xxxxxx-xxxxxx-xxxxxx-xxxxxxx event=0 ip=198.41.200.193 location=iad12 protocol=quic
2024-01-17T20:15:54Z INF Tunnel server stopped

Were you running the 2024.1.3 docker image, or were you running an image of an older version and it got upgraded to 2024.1.3? If your deployment uses the same command as https://github.com/cloudflare/helm-charts/blob/main/charts/cloudflare-tunnel-remote/templates/deployment.yaml#L29-L38, then autoupdate should be disabled.

[connors2015]

Hi @tdack, your issue is different. The error is cloudflared was built with a go version that doesn't support post quantum curves. Where did you download cloudflared? Or did you build it from source?

This is the error from 2024.1.3 which is from docker hub.

2024-01-17T20:15:52Z ERR Failed to create new quic connection error="failed to dial to edge with quic: context canceled" connIndex=2 event=0 ip=198.41.192.57
2024-01-17T20:15:52Z INF Registered tunnel connection connIndex=2 connection=xxxxxx-xxxxxx-xxxxxx-xxxxxxx event=0 ip=xxx.xxx.xxx.xxx location=iad02 protocol=quic
2024-01-17T20:15:53Z ERR Failed to create new quic connection error="failed to dial to edge with quic: context canceled" connIndex=3 event=0 ip=xxx.xxx.xxx.xxx
2024-01-17T20:15:53Z INF Registered tunnel connection connIndex=3 connection=xxxxxx-xxxxxx-xxxxxx-xxxxxxx event=0 ip=198.41.200.193 location=iad12 protocol=quic
2024-01-17T20:15:54Z INF Tunnel server stopped

Were you running the 2024.1.3 docker image, or were you running an image of an older version and it got upgraded to 2024.1.3? If your deployment uses the same command as https://github.com/cloudflare/helm-charts/blob/main/charts/cloudflare-tunnel-remote/templates/deployment.yaml#L29-L38, then autoupdate should be disabled.

It was updated. Running 2024.1.3 from the start seems to work fine. ty.

However running that helm chart does not disable autoupdate, it should, but it does not as you can see from my original post the autoupdate is still running.

If I use kubectl and apply the same chart manually it does disable the autoupdate. So I'm not sure if the issue is with helm or something else.

[tdack]

@tdack your issue is because cloudflared needs to be built with a go tool chain that supports post quantum curves. Please checkout https://github.com/cloudflare/cloudflared?tab=readme-ov-file#installing-cloudflared for how to install the go tool chain and build from source.

That fixed it. Thankyou! I missed that as I'd been building from source for quite a while without issue.