Cloudflared hangs after internet connection severed #306
Labels
Area: DNS-Proxy
Issues related to DNS-Proxy functionality in cloudflared
Priority: Deferred
Issue has been placed in backlog
Comments
|
First of all, make sure you do not have version 2020.12.0 (which has issues with reconnecting tunnel connections specifically) |
|
Thanks @vidarak. I updated to this version (https://github.com/cloudflare/cloudflared/releases/tag/2021.2.1) but the issue still persists. Here is what Here is the cloudflared log says after i intentionally interrupt the connection (by restarting my modem router): This is And this is the pihole log when this happens (as pihole uses cloudflared as an upstream dns): Pretty sure it is not the pihole's fault since i restarted cloudflared and everything works as normal again. |
|
Yeah, can confirm i've seen this issue for quite some time now. A reboot of cloudflared, which I run in docker solves it every time. |
|
It wouldn't surprise me if this is due to the default retries value being way too low. It is supposed to be only 5 according to Cloudflare support - with exponential backoff. You seem to have 10 entries in your log file within a couple of seconds (assuming 2 entries for every failure). It doesn't seem to do exponential backoff in your case but still max out on 5 retries. Probably just another bug that it doesn't do the backoff, but it doesn't remove the need to increase retries anyway. I set retries to at least 99 in every config because of this wrong default retry value decision by Cloudflare. |
|
can you show me the sample config of that @vidarak ? |
|
I suspect this is the same problem as #302 |
I've added these 2 lines to configs lately (but due to a now resolved bug, the retries option only work on the most recent releases... 2020.11.11 or later): |
|
can this be translated into command line config? i dont use any config file for cloudflared @vidarak |
|
it's the --retries option |
|
thanks for the suggestion @vidarak but it does not seem to fix the problem for me. i tried: |
|
I have the same issue. Raspberry Pi 3B+ Here is what the issue looks like when it happens: Restarting the service works every time to get it going again The issue seems to happen approximately once per month, give or take. Also, note that I have two redundant Raspberry Pis running this configuration, and they seem to fail at about the same time when they fail, and they fail the same exact way. |
|
I believe this is fixed in #290. Try upgrading to the latest version. Feel free to reopen if that doesn't work! |
|
@adamchalmers unfortunately it does not fix the error. I used this now: the (temporary) fix is still the same, needs to restart cloudflared manually. |
|
and please reopen this issue. thank you. |
|
Issue also exist on latest version UPDATE: also exist on latest version Systemd output after Router reconnect to WAN |
|
Also exists on 2021.4.0. Any loss of, or failover of, the WAN and Cloudflared just hangs until manually restarted.... This makes Cloudflared completely unusable for me tbh. |
|
Having the same problem, which for me makes cloudflared unusable for me as well. Switching to standard DNS until this can be resolved. |
I've went with installing Unbound and recursive resolving using the root servers. |
|
Having same problem with Unifi and backup LTE connection, if it switches to backup connection and a different external IP then cloudflared hangs. Running latest editions of Pi-Hole, Cloudflared and Raspberry PI OS. |
|
Same issue here. On version 2021.6.0 (built 2021-06-22-1034 UTC). |
|
|
|
I have this issue too. Except mine happens within 5mins of starting cloudflared service . EDIT: Changing to upstream--conns 50 has worked for me as well. |
I am experiencing the same problem with |
Better idea - use dnscrypt. Problem solved. No one at CF cares about this issue that's been ongoing for years now. It won't be fixed anytime soon. https://github.com/DNSCrypt/dnscrypt-proxy Now I can rest assured that momentary internet drops won't end up in much longer outages. |
|
I too ran into this issue with I have the following launch agent set up as a workaround and it seems to work okay. It basically exits the process if grep finds:
|
|
Up! Same here. |
|
I am having the same issue and it's disrupting the service I provide. It would be great if this could be fixed soon. I'll try and find a workaround, but it won't be possible until I can get physical access to the server. |
|
The tunnel side of the issue is #724 There’s a commit in there that fixes this problem. You can build this from source if the need is urgent. note that this is only for tunnel but not dns-proxy. |
|
FWIW, My work-around is to force cloudflared to restart every 10 minutes or so. |
|
Also having this issue, My instance only ever seems to last for about a day before erroring out and then I need to remote onto the server an close the current instance and start a new one. I have a backup tunnel luckily so its not causing any downtime for me but its a huge pain in the ass because this is my main server and I would like my traffic to go via this server ideally. If anyones got any solution for windows, please let me know. I am trying to find a way to terminate the script automatically at a certain time or maybe upon the error message and start a new one. |
DevinCarr
added
the
Area: DNS-Proxy
|
My solution was to have a PM2 instance assigned to the root user (so that it would have permission to restart pm2) that kept this code running: "use strict";
const { exec } = require('node:child_process');
let restartNeeded = false;
setInterval(async () => {
try {
await fetch("https://oirnoir.dev/"); // Replace with literally any website with reasonable uptime and speed that can be fetched every 30 seconds
console.log("Success!");
if (restartNeeded) {
console.log("Restarting cloudflared...");
exec("systemctl restart cloudflared");
restartNeeded = false;
}
} catch (err) {
restartNeeded = true;
console.log("Restart needed!");
}
}, 30000);The code sets restartNeeded when it detects a break in the internet connection. When the connection is restored, cloudflared is then restarted. |
|
Why are people writing solutions to this broken software?! I switched to it since my earlier comment on July 18, 2022 and I've had no problems. |
|
I use cloudflared in docker and have fixed this by building a custom cloudflared image Dockerfile: docker-comose.yml |
|
Just want to add the underlying issue still persists even with 2023.4.1 I have a OpenSense Firewall and two PiHoles running on Ubuntu 22.04.2 LTS. When I perform firmware updates on the firewall that require a reboot, even with only one packet being dropped, cloudflared on both PiHoles will fail to reconnect to 1.1.1.1 an 1.0.0.1 and provide DoH services for my PiHoles. A quick restart of cloudfared solves the issue. |
|
up! only if the |
|
The problem still exists in 2023.8.2. Tonight we have a internet disconnect of 10 minutes. After reconnecting all DNS queries not working. We are using PiHole with cloudflared. The message in the logs are Only a restart helps. What can we do? |
Give up on cloudflared. Just use dnscrypt. I set it up over a year and a half ago and haven't touched it since. This issue has been open since 2021, even if it ever does get resolved, it's not worth the wait when dnscrypt works perfectly and is rock solid stable. |
|
Issues is still present. |
Issue is still present as of 2024.5.0. |
|
I have same problem if connection lost cloudflare dns proxy not recovery. Jun 10 15:09:20 dns02 cloudflared[276]: 2024-06-10T13:09:20Z ERR failed to connect to an HTTPS backend "https://1.0.0.1/dns-query" error="failed to perform an HTTPS request: Post "https://1.0.0.1/dns-query\": net/http: request canceled (Client.Timeout exceeded while awaiting head> |
|
As workaround, I'm using webhookd via Uptime Kuma alert trigger to restart cloudflared process each time internet connectivity is lost/recovered. |
|
Experiencing same on Debian (LXC), version |
|
I encountered a situation where when the network was interrupted and restored, there was no response in the cloudflared log. ↑ This remains the case 20 minutes after the network connection is disconnected and restored. So I had to write a simple python script to manually perform health checks and restart cloudflared. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
I am running cloudflared on ubuntu arm (raspberry pi) as a proxy dns. Whenever i restarted my router, cloudflared hangs and stops working and cant connect until i restarted the cloudflared process. Here is my cloudflared command:
cloudflared proxy-dns --port 5053 --upstream https://94.140.14.14/dns-queryAny idea of how to make cloudflared process restarts automatically if connection is lost and restored? Sorry if this has been discussed before. Thanks.
The text was updated successfully, but these errors were encountered: