🐛Cloudflared Service Doesn’t Starting after Update (Ubuntu Jammy)

[marvindanig]

Describe the bug
I updated the Cloudflare daemon installed on my Ubuntu machine (Jammy) today. Immediately after the update, the cloudflared tunnel went inactive and isn’t able to establish QUIC connection anymore.

Logs are shared below:

To Reproduce
Steps to reproduce the behavior:

1. I deleted the inactive tunnel and $ sudo cloudflared service uninstall to start afresh.
2. Then I created a new tunnel instance from my https://dashboard.cloudflare.com
3. Then I copy the sudo cloudflared service install <token> on my machine to establish the connection:
4. The service gets stuck at: 2024-03-02T02:24:08Z INF Using Systemd message and never completes.
5. Tunnel ID : b32d4357-413c-49c1-902a-7bf97ed70073 bubblin-1 2024-03-02T02:02:48Z
6. cloudflared config:

"~/.cloudflared/config.yml" 4L, 159B                                                                                                                                                                                                                                                           2,44          All
url: http://localhost:3000
tunnel: fd4be4fe-a121-412c-a163-015247c7f05a
credentials-file: /home/marvin/.cloudflared/fd4be4fe-a121-412c-a163-015247c7f05a.json

Expected behavior
The QUIC connection isn't established and the service doesn't activate. See the logs below.

Environment and versions

• OS: Ubuntu 22.04.4 LTS (JAMMY)
• Architecture: x86_64
• Version: [e.g. 2022.02.0] cloudflared version 2024.2.1 (built 2024-02-20-1726 UTC)

Logs and errors
If applicable, add logs or errors to help explain your problem.

Mar 01 21:10:11 alexandrium cloudflared[61095]: 2024-03-02T02:10:11Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:10:43 alexandrium cloudflared[61095]: 2024-03-02T02:10:43Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.192.47:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.47
Mar 01 21:10:43 alexandrium cloudflared[61095]: 2024-03-02T02:10:43Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.47:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.47
Mar 01 21:10:43 alexandrium cloudflared[61095]: 2024-03-02T02:10:43Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.47
Mar 01 21:11:00 alexandrium cloudflared[61095]: 2024-03-02T02:11:00Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.193:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:11:00 alexandrium cloudflared[61095]: 2024-03-02T02:11:00Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.193:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:11:00 alexandrium cloudflared[61095]: 2024-03-02T02:11:00Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:11:22 alexandrium cloudflared[61095]: 2024-03-02T02:11:22Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.233:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.233
Mar 01 21:11:22 alexandrium cloudflared[61095]: 2024-03-02T02:11:22Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.233:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.233
Mar 01 21:11:22 alexandrium cloudflared[61095]: 2024-03-02T02:11:22Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.233
Mar 01 21:11:48 alexandrium cloudflared[61095]: 2024-03-02T02:11:48Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.193:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:11:48 alexandrium cloudflared[61095]: 2024-03-02T02:11:48Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.193:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:11:48 alexandrium cloudflared[61095]: 2024-03-02T02:11:48Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.193
Mar 01 21:12:34 alexandrium cloudflared[61095]: 2024-03-02T02:12:34Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.192.47:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.47
Mar 01 21:12:34 alexandrium cloudflared[61095]: 2024-03-02T02:12:34Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.47:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.47
Mar 01 21:12:34 alexandrium cloudflared[61095]: 2024-03-02T02:12:34Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.47

Additional context
The tunnels were working normally before the update.

[marvindanig]

When I try to start a test tunnel from the terminal it shows the following error in the logs:

INF Starting metrics server on 127.0.0.1:38923/metrics
2024/03/01 21:38:31 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024-03-02T02:38:36Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.193
2024-03-02T02:38:36Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.200.193
2024-03-02T02:38:41Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.57
2024-03-02T02:38:41Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.57
2024-03-02T02:38:47Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.227
2024-03-02T02:38:47Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.227
2024-03-02T02:38:56Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.77
2024-03-02T02:38:56Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.192.77
2024-03-02T02:39:15Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.33
2024-03-02T02:39:15Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.200.33
2024-03-02T02:39:41Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73
2024-03-02T02:39:41Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.73
2024-03-02T02:39:49Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.67

[marvindanig]

I increased the buffer size with:

$ sudo sysctl -w net.core.rmem_max=2500000
$ sudo sysctl -w net.core.wmem_max=2500000

…and yet the QUIC connection fails no matter what:

[marvindanig]

I restarted the machine. And it worked for no rhyme or reason.