-
Notifications
You must be signed in to change notification settings - Fork 535
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cloudflare issues: multiple attempts/black screen #5302
Comments
+1 |
it could be webrtc issue? have you tried with |
also nodejs 12 is now EOL, please update node to the latest LTS (18) |
I have tested with WebRTC disabled and also enabled, having the same behaviour. One thing to note is that this issue started a few weeks ago. |
Updated nodejs to 18.17.1 and same issue with or without webrtc enabled. I rebooted the server and checked nodejs version with the --version parameter. The logs show connected then immediately disconnected. When it works I noticed that "Relay holding: * (::1) Authenticated" shows as well. WEBREQUEST: (-) /meshrelay.ashx/.websocket?p=1&nodeid=node//iWMDZtF%24QteFec1amvHRz7nA8c4SqcXNrz7d7HN5JYxOTNwrOvK%24WOqkZ9XJjHuq&id=ma8286xee29&rauth=VEcPv1E75FdSda%24JPWUN7XeUM2plL7mDexVJ37kpZ%24v4selPtY3Kb%24Voz%40maz2MGG1E3IP4Pm9VcASvz3YSGyGo9CNWntAzVx3A%24BLwfSdTIPN4by%40o7 |
+1 |
+1 |
Same here, just started a few days ago though. Running through Cloudflare as well. |
+1 |
+1 same, although it didn't start immediately after 1.1.8=>1.1.10 transition, it started giving issues gradually and now it affects all agents no matter the OS. Real-time monitoring using graphs works fine, however it fails to establish any other type of connection, let it be webrtc, stream, rdp, terminal or file transfer. MeshCentral Router also fails to RDP in "Configuring remote session" phase. Also, I'm running mesh through cloudflare too. |
this sounds to me like a cloudflare issue not a meshcentral issue, as nothing has changed recently to do with proxies or tunneling |
I took a look and filtered websocket related issues in google search from last 7 days. These are results. https://community.cloudflare.com/t/websocket-not-stable-some-time-connect-some-time-no/547638 https://community.cloudflare.com/t/websocket-problem/547094 Unfortunately these haven't been answered. Also it seems that CloudFlare bottlenecks websocket connections after reaching certain level of connections (or spikes). Does anybody know if CloudFlare started limiting websocket connections just recently or changed anything related to these limits or to how these are handled? https://developers.cloudflare.com/support/network/using-cloudflare-with-websockets/ |
Great find @supra36, I have the feeling it is Cloudflare related more than the actual update of MC to .10... I am not 100% though. I did a tcpdump and I saw Cloudflare giving [F] and [R] flags while reproducing the issue. Would be nice if someone from the community that experiences the same issue and has a Cloudflare business or higher tier subscription opens a ticket with their support to see if this gets an answer from their side... as using the free tier there is no chance of support afaik. |
There's a Cloudflare Developers on Discord. Might be useful to ask around. A post 3 days ago mentioned websocket disconnects when using IPv6 but not when using IPv4. Someone mentioned disabling IPv6 in Network -> IPv6 Compatibility. |
Hi @NiceGuyIT, Thank you for the suggestion. I have disabled IPv6 through the API and the issue persists. For anyone that wants to give it a try... I used PowerShell: $headers=@{} |
Just to let you know, I ditched cloudflare and used let's encrypt. It's now working fine. |
I decided to provision another Ubuntu server and installed MC 1.1.10 with Let's Encrypt certificate rather than CloudFlare and it also works fine. I wonder what changes have been made on CloudFlare side to be dropping the websocket connections, it must be something recent. I'm sure their support would be able to comment but unfortunately we are on a free tier without any technical support. |
As per the questioning raised by @frogweh on #5309 he seems to be using Nginx and not CloudFlare and having the same issue... would be good to test that to see if it's reverse proxy in general or if we can completely isolated to CF. |
@iribarrenjg if you read the post #5309 he says he uses a cloudflare certificate and if you read the logs shows heads for cf connecting from, so he is indeed using cloudflare! |
+1 on 1.1.10 (Raspberry Pi OS 64 Bit via Cloudflare and Nginx Proxy Manager) |
Like others here, I changed my DNS records away from Cloudflare's proxy and things started working as expected. Also, this was happening before 1.1.10. I upgraded to 1.1.10 in hopes of it being a MeshCentral issue. Now I am trying to figure out how to lock down my environment without Cloudflare's rules. I know it is possible, but Cloudflare made it easy :). |
@jwiener3 you can do an IP allow list for agents and clients which might help? MeshCentral/meshcentral-config-schema.json Lines 592 to 628 in b1d2d1a
|
Thanks, I will take a look at that. |
So I can just disable the proxy checkmark on cloudflare? I have my domain registered with them. |
Yes that is what I did, and then I had to open up my ACL on the server that was hosting Meshcentral to allow connections from anywhere, as I only had it allowing connections from cloudflare IP space. |
+1 docker Raspberry pi with Cloudflare tunnels. Today I update the Cloudflare tunnel app and now is not connecting at all |
@DivMode sadly using cloudflare tunnels is also effected, so even if u used a tunnel to route ur meshcentral server behind a firewall or proxy, you will still experience a black screen issue |
Experiencing the same issue... @si458 you mentioned in another thread that the PAID cloudflare subscription does not experience this issue? Do you know what tier subscription this is, am considering it just so I can use meshcentral again |
From my previous testing, I had a higher success rate just using their paid teir, the first level... But I haven't tried it recently to verify if that's still the case or not. Has anybody else got paid cloudflare and clarity if it still works or not? |
@blaine07 I don't think so? |
@si458 Currently we have Pro plan, but we've been experiencing the same issue for weeks. We disabled the proxy as a temporary solution. We do not use MeshCentral but in the case of a different application we experience exactly the same websocket connection issue. Unpredictable, some browsers just cannot initialise the socket connection, but others can. With DNS only CNAME record we have zero issue. I found this github issue in the Cloudflare Developers discord server, on that thread I have explained more: https://discord.com/channels/595317990191398933/1150828630340022343/1150828630340022343 |
@stoiet so it's a cloudflare issue then! |
At this moment you should consider that cloudfare tunnels are just not compatible with meshcentral. |
Just posted an update on this matter on the cloudfare community page. Hopefully we can get a response there... https://community.cloudflare.com/t/meshcentral-agents-can-no-longer-connect-using-tunnel/551937/13 |
I have setup a reverse proxy on a VPS. If I proxied this through cloudflare while my meshcentral is on DNS only, would my IP address still be hidden from the public? Excuse me for being a noob on the topic |
@Dasno7 you are still proxing through cloudflare so your meshcentral won't work You would have to set the dns to dns only and point the ip of ur meshcentral dns to the VPS of ur reverse proxy, then forward the reverse proxy to ur meshcentral server, this would hide ur meshcentral ip as others have explained |
Not sure when this got fixed, but a friend just casually mentioned to me mesh was working for him over his cloudflare proxy again. I just checked, it's working for us as well. No idea what changed... but definitely didn't change anything. Crossing fingers this isn't temporary this time. |
YES!! It does work AGAIN. I love you all !! |
Oh wow will check when bk off hols. |
Yes they did change something. I can now see the ip addresses of my clients again, Before the change i got only the ip from the CloudFlare proxy back. |
Cloudflare changes "something" again....7+ months later. What crap |
Can confirm that on free tier, the issue has resolved itself. No need for any work from mesh central. I would advise the devs to put a warning in case a similar issue like this happens again. |
Could you please share your (sanitised) config/json file? I cant get mine to work with Cloudflare again, I dont know what I am doing wrong. |
@si458 yes, didn't realise that was a page. Only thing that's incorrect is that if one is running cloudflared, then disable proxy will result in argo tunnel error. @dooley74 my configuration is quite plain and is the default. Make a copy of your current configuration by renaming it to .bk then let mesh central generate a new config I set up mesh central in unraid https://youtu.be/yHuG5qFsYtk |
@Goldmaster, yes using cloudflare tunnel is slightly different from using the dns Proxy. But the issue still stands that cloudflare broke something, and it took them months to fix whatever they changed! So let's hope they keep it that way and it doesn't break again! |
This issue appears resolved while using cloudlfare tunnels. |
How does the config.json file look like through cloudflare tunnels? |
@piranhap like a standard config |
Should this be closed as I am now able to get it to work? If more people say its working then close, or may be best to leave open? |
For now I feel like this issue should be closed. It has been working properly for months now. If this issue should re-occur then a new issue will be better as we can keep track of the time the issue started better. |
Closing this issue as it seems to be resolved now |
Describe the bug
Clicking on the "Connect" button under "Desktop" or "Terminal" results in "Disconnected" approximately 9/10 times. Other times it will connect successfully. The disconnection is immediately shown after clicking "Connect".
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Connects successfully on every attempt.
Screenshots
If applicable, add screenshots to help explain your problem.
Server Software (please complete the following information):
Client Device (please complete the following information):
Additional context
The problem seems to only occur using CloudFlare, so I think that the proxy is causing websocket disconnection. The MeshCentral VM is hosted on the Hetzner platform. Others face the same issue (e.g https://www.reddit.com/r/MeshCentral/comments/15y28x3/random_disconnects_behind_cloudflare/)
Your config.json file
The text was updated successfully, but these errors were encountered: