The Feature Nobody Notices #14
casablanque-code
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Ironically, one of the features I’m most proud of is something users will hopefully never see.
Rollback.
Infrastructure tools have an unfortunate habit of failing halfway through an operation. Maybe the API rate-limits you. Maybe authentication expires. Maybe one request succeeds while the next one doesn’t.
The result is usually the same: a half-configured environment.
A tunnel that exists but isn’t reachable.
A DNS record pointing nowhere.
An Access application protecting a service that no longer exists.
Cleaning up that mess is rarely difficult.
It’s just annoying.
I wanted cfzt to behave differently.
Every deployment is treated as a single operation. If every step succeeds, the service is online. If any step fails, everything created up to that point is removed automatically. No orphaned resources. No guessing what needs to be deleted. No opening the Cloudflare dashboard to clean up after a failed command. It sounds obvious.
In practice, it required keeping track of every API call, every created resource, and exactly how to undo each one.
The result isn’t flashy.
There’s no screenshot that demonstrates it.
But it’s probably the feature that gives me the most confidence when I use the tool myself.
Because the best rollback is the one you never have to think about.
Then I Started Reading the Logs
At some point, the project reached a stage where I stopped adding features and started using it every day.
That’s when the interesting problems appeared.
One evening, after deploying yet another service, I noticed something unusual in the logs.
cloudflared had switched from QUIC to HTTP/2.
That wasn’t surprising.
QUIC relies on UDP, and if UDP becomes temporarily unavailable — a router reboot, a flaky Wi-Fi connection, a restrictive firewall — cloudflared automatically falls back to HTTP/2 over TCP.
That’s exactly what it’s supposed to do.
The surprising part came later.
It never switched back.
Wait… Is This Really Permanent?
At first, I assumed I had misunderstood how the tunnel worked.
Maybe it periodically retried QUIC. Maybe there was a hidden configuration option. Maybe I just hadn’t waited long enough. So I kept watching. Minutes passed. Then hours. The tunnel continued serving requests perfectly. But it stayed on HTTP/2 the entire time.
I searched the documentation. Nothing.
Then I searched GitHub. Eventually, I found the issue. Other people had noticed exactly the same behavior. Once a tunnel falls back to HTTP/2, it never attempts to return to QUIC unless the process is restarted manually. The report wasn’t new. It had already been sitting there for quite a while.
At that moment, I had two choices. Wait until the bug was fixed.
Or assume it wouldn’t be fixed anytime soon and design around it.
I chose the second option.
Building a Watchdog Instead of a Patch
I had no intention of forking cloudflared.
Maintaining a networking daemon just to fix one edge case would have been a terrible trade-off.
Instead, I asked myself a different question.
What information do I actually have?
The answer turned out to be surprisingly simple. Every time cloudflared falls back from QUIC to HTTP/2, it writes a log entry. That log message became the entire API. cfzt doesn't inspect packets. It doesn’t perform health checks. It doesn’t try to infer network conditions. It simply watches the logs.
When it detects a fallback, it starts a timer.
Not because the tunnel is broken — it isn’t — but because the network might recover. If enough time passes and the tunnel is still running over HTTP/2, cfzt quietly restarts the service.
The restart gives cloudflared another opportunity to negotiate QUIC.
If UDP is still unavailable, the tunnel falls back again.
No problem.
The watchdog waits longer before trying again.
Ten minutes.
Twenty.
Forty.
Eventually, it settles into a one-hour retry interval. The goal isn’t to fight the network. It’s to recover automatically when recovery is possible.
In the end, the solution turned out to be surprisingly small.
Not a networking stack.
Not a protocol implementation.
Just a process watching another process.
Sometimes that’s all you need.
It’s not academic. It’s not elegant in a textbook way. But it closed the loop.
Now, when a router reboots in the middle of the night, cfzt catches the drop, backs off, tries again when the storm passes, and quietly moves the tunnel back to QUIC.
All reactions