BungeeCord Stops Listening for new connections

[thefourcraft]

The Problem/bug

We have a server network, our main proxy just stops excepting connections after some time (8-5 hours)
No plugins are interfering & no errors on the console are showing up, it just stops accepting new players.
Players that are online are not affected unless they want to switch servers and new players can't join.
This has started happening recently (about two weeks), also this is happing at night most of the time when no players are online
ones it happens and BungeeCord Stops Listening for new connections no one can join or do anything
Unless you kill the server and restart it

Solutions that we have tried

1. Checking all ports & DNS
   This is the first step we took, we testes all of the port and DNS
   everything is working fine
   our main proxy is running on 25565 just fine
2. Checking tunnel & firewall rules & protocols
   I thought that its a problem in some firewall rule or protocol, but all the rules are doing what their spouse to do 😄
   With the tunnel everything is good (MySQL server is able to connect with no problems)
3. installing a fresh BunngeCord server
   A new server with no plugins same result (the latest version)
4. searching Google
   We found this post on Github
   He ItsHarry, referred to a system limitation caused by max file descriptor limit, maybe this is our problem but we have been running 24/7 no problems till now if I to understand correctly this problem that he was talking about is maximum for the number of simultaneously open files at the stream I/O level. This article
   We thought that this was our problem, we deleted all the logs and cleaned up some temp files for the server PC
   As of 3.22.2022, This Problem didn't show up, but there were a lot of players online (It happens when there is almost no one online or when BungeeCord is ideal)
5. Contacting The ISP to Check if we have a limit on rate-limit per IP as mentioned by @xism4

This Might be The Problem But Still Unsure

Information & Logs

• Last 48H log: Click Me
• System specs:
• Servers - self-hosted
• OS: Win 10
• CPU: AMD Rayzen 5 3400G
• RAM: 16GB
• BunngeCord Version: This server is running BungeeCord version git:BungeeCord-Bootstrap:1.18-R0.1-SNAPSHOT:f4f94d3:1632 by md_5
• Servers: Spigot 52 behind (This server is running CraftBukkit version 3284a-Spigot-3892929-0ab8487 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) - all are the same
• This is not the only server that is running on this IP address, we have multiple servers and applications & websites in our "Facility 🏭"
• We use Cloudflared tunnel & Ngrok for our network and operation, the BunngeCord is port forwarded & has its backup tunnel

Note: We Might Have A fix because we didn't try number 5 and maybe there will be a number 6

IF ANYONE has suggestions, you are welcome to help

[Janmm14]

try jvisualvm or console program jstack to get a thread dump of your bungee when it stops accepting connections.
the thing about max file stuff in that post was about linux. it seems the default limit on windows is higher.

[xism4]

The Problem/bug

We have a server network, our main proxy just stops excepting connections after some time (8-5 hours) No plugins are interfering & no errors on the console are showing up, it just stops accepting new players. Players that are online are not affected unless they want to switch servers and new players can't join. This has started happening recently (about two weeks), also this is happing at night most of the time when no players are online ones it happens and BungeeCord Stops Listening for new connections no one can join or do anything Unless you kill the server and restart it

Solutions that we have tried

1. Checking all ports & DNS
   This is the first step we took, we testes all of the port and DNS
   everything is working fine
   our main proxy is running on 25565 just fine
2. Checking tunnel & firewall rules & protocols
   I thought that its a problem in some firewall rule or protocol, but all the rules are doing what their spouse to do 😄
   With the tunnel everything is good (MySQL server is able to connect with no problems)
3. installing a fresh bunngecord server
   A new server with no plugins same result (the latest version)
4. searching Google
   We found this post on Github
   He ItsHarry, referred to a system limitation caused by max file descriptor limit, maybe this is our problem but we have been running 24/7 no problems till now if I to understand correctly this problem that he was talking about is maximum for the number of simultaneously open files at the stream I/O level. This article

This might be the solution but I'm not sure

Information & Logs

Last 48H log: Click Me Sytme cpacs: Servers - self hosted OS: Win 10 CPU: AMD Rayzen 5 3400G RAM: 16GB BunngeCord Version: This server is running BungeeCord version git:BungeeCord-Bootstrap:1.18-R0.1-SNAPSHOT:f4f94d3:1632 by md_5 Servers: Spigot 52 behind (This server is running CraftBukkit version 3284a-Spigot-3892929-0ab8487 (MC: 1.17.1) (Implementing API version 1.17.1-R0.1-SNAPSHOT) - all are the same

Note: We Might Have A fix because we didn't try number 4 and maybe there will be a number 5

IF ANYONE has suggestions, you are welcome to help

I had that issue, it was my ratelimit per ip.
I say you this because i readed before that you are self-hosting, so i guess that you have some type of Firewall OS for your router.

[thefourcraft]

@Janmm14
You Meen VisualVM, use this to find the problem?
This could be a nice tool to monitor the servers

[thefourcraft]

@xism4
I don't think but I will check it with our ISP,
But the whole reason that this is happing is
"also this is happing at night most of the time when no players are online"

This limit might be a future problem 👍🏻 Thanks for the heads-up

[Janmm14]

@Janmm14 You Meen VisualVM, use this to find the problem? This could be a nice tool to monitor the servers

yes I meant that - i was used to unix where it is sometimes named jvisualvm.

[Outfluencer]

You can use ps -ef | grep java to get the PID of the java process and than run jstack -l -e YOUR_PID > abc.txt to get a full threaddump this could probably help

[Janmm14]

You can use ps -ef | grep java to get the PID of the java process and than run jstack -l -e YOUR_PID > abc.txt to get a full threaddump this could probably help

Well he homehosts on windows, so for the PID you go into task manager "Detail" page.

[xism4]

You can use ps -ef | grep java to get the PID of the java process and than run jstack -l -e YOUR_PID > abc.txt to get a full threaddump this could probably help

Do you have discord?

[Outfluencer]

Fabian#0554

[thefourcraft]

Testing

So, We have been tasting on different PC, and different ISP providers to check if the problem is actually us or a plugin
This will take some time, but the problem is still happening.
so far this problem is accruing only on some PC with an AMD CPU, also the problem is only happening when there are no players online, maybe it's a Java issue not sure
All the PC have the same java and latest BungeeCord...
I have seen this problem in a couple of places:

• https://www.spigotmc.org/threads/about-three-times-a-day-the-server-disconnects-all-players.100963/
• https://www.spigotmc.org/threads/bungee-not-listening.318094/
  Now, these are different versions of BungeeCord but they are very similar to our case

Follow up

The "BUG" BungeeCord stops listening to new connections. This happens when no one is online on the server, at night.
BUT When players or stuff want to join it doesn't let them because BungeeCord stops listening to new connections.
When there are a lot of Players (15 -60 ) online sometimes BungeeCord stops listening to new connections and new players can't join.

[xism4]

Testing

So, We have been tasting on different PC, and different ISP providers to check if the problem is actually us or a plugin This will take some time, but the problem is still happening. so far this problem is accruing only on some PC with an AMD CPU, also the problem is only happening when there are no players online, maybe it's a Java issue not sure All the PC have the same java and latest BungeeCord... I have seen this problem in a couple of places:

• https://www.spigotmc.org/threads/about-three-times-a-day-the-server-disconnects-all-players.100963/
• https://www.spigotmc.org/threads/bungee-not-listening.318094/
  Now, these are different versions of BungeeCord but they are very similar to our case

Follow up

The "BUG" BungeeCord stops listening to new connections. This happens when no one is online on the server, at night. BUT When players or stuff want to join it doesn't let them because BungeeCord stops listening to new connections. When there are a lot of Players (15 -60 ) online sometimes BungeeCord stops listening to new connections and new players can't join.

I guess that is some type of firewall or something related to that.
I saw few of software that does that to avoid hardware usage.

Is not a BungeeCord issue

[thefourcraft]

@xism4
I have turned off all firewalls, localhost doesn't work also, if maybe some firewall is blocking it then the external IP won't work
After BungeeCord stops listening to new connections even LocalHost can't Join.

"But this might be a plugin or something"
All the plugins that we use have been running fine for the last year
All are up to date 0 Bugs or problems

Plugin list

• LuckPerms
• AdvancedMOTD
• Plan - Player Analytics
• SkinRestorr

Costume Plugins

• Login System
• Language System
  Costume Plugins Have Been Running With No Problems, Our Partners are also using the same costume Plugins, no problems
  they are on a host, we are on a private server...

This might be a problem on the server PC only... But this started very recently and this server is Running Some Other Things.
Also, it Runns another small Spigot server, no problems on there.

[Janmm14]

Still no attempt at providing a thread dump when this happens :/

Please provide a thread dump when the problem appears!

[thefourcraft]

@Janmm14
We Have Made A Monitor For this, as soon as the BungeeCord stops listening to new connections this should be red
https://stats.ims-network.net/
I will also share a thread dump

[thefourcraft]

Solved 💯

Problem: The "BUG" BungeeCord stops listening to new connections. This happens when no one is online on the server, at night. BUT When players or stuff want to join it doesn't let them because BungeeCord stops listening to new connections. When there are a lot of Players (15 -60 ) online sometimes BungeeCord stops listening to new connections and new players can't join.

What Happend❓ ❔

When Running A self-hosted server you are in charge and responsible for any problems that might occur or happen to the PC.
In Windows, there is a thing called Registry. The Windows Registry is a hierarchical database that stores low-level settings for the Microsoft Windows operating system and for applications that opt to use the registry. - Wikipedia
Java is using that Registry for some logs & other things. probably when someone shut down the PC server one or more registries got broken, which affected some parts of the operating system and java fecundity.
The BunngeeCord Worked fine until java needed to use or access those broken registries, probably something to do with stand-by registries because the problem happened when no one needed to access the server.
We use open JDK... after a lot of research, we found this post on stack overflow which explained the problem of broken java registries.

How To Fix 👍🏻

To Fix Any Problems with Windows use Restoro we run a scan on all our servers and this fixed all the problems with registries.
also running a windows repair might also help to fix some problems but Restoro is also checking 3rd party apps.
Also, Restoro makes a malware scan, which is more advanced than windows because it found a RAT file on my PC. RAT is one of the most dangerous PC viruses you can get...

Network

Network and ISP can also be your problem as @Janmm14 and @xism4 stated, we checked everything all tho not related to this our BunngeeCord and spigot servers were configured wongly.
Also, we added a serverside IP backup, if the main one has restrictions or is limited the DNS & load balancing should migrate players to another available address. This is highly recommended if you have a lot of connections on one IP.

Update

Another problem that could have caused this issue is drives or faulty cables, we replaced all intended cables and updated all
BIOS, Drives, Windows, Java...

Warning 📛

we are not responsible for any damage that you do to your PC or home network,
you need to check and verify with your ISP that you are allowed to make a server, ISP might change some firewall rules or ask you to change your internet plan.
This is a specific problem that happened on our servers the cause for your problem might be different.

Thanks For All The Help 💘 🤟🏻