Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxy seconds behind! (Time not synchronized or server down?) spamming console #1

Closed
TheMagBum opened this issue Apr 16, 2021 · 15 comments · Fixed by #2
Closed

proxy seconds behind! (Time not synchronized or server down?) spamming console #1

TheMagBum opened this issue Apr 16, 2021 · 15 comments · Fixed by #2
Assignees
Labels
bug Something isn't working completed Finished! good first issue Good for newcomers

Comments

@TheMagBum
Copy link

i have this error since some times:
13:23:35 [SEVERE] [RedisBungee] proxy-1_ZPu1cV is 15971 seconds behind! (Time not synchronized or server down?)
the server is offline and i will that it is offline

@ghost
Copy link

ghost commented Apr 17, 2021

what do you mean by "the server is offline and i will that it is offline"?

@TheMagBum
Copy link
Author

i have turn off the server, but the error come too

@ghost
Copy link

ghost commented Apr 17, 2021

have you turned off the server nicely, with "end" command?

@TheMagBum
Copy link
Author

No, I think the server was frozen and I terminated it

@ghost
Copy link

ghost commented Apr 17, 2021

in that case it is normal, you need to stop it nicely in order to not have these warnings

@TheMagBum
Copy link
Author

The problem is that I am using Redis with a cloud and I can't recreate the serverid

@ghost
Copy link

ghost commented Apr 17, 2021

I see, let's see what @ham1255 thinks about this

@ham1255
Copy link
Member

ham1255 commented Apr 17, 2021

Never seen this error before, make sure your system time is right + and shutdown every bungee cord nicely

@ham1255 ham1255 self-assigned this Apr 17, 2021
@ham1255
Copy link
Member

ham1255 commented Apr 17, 2021

and also you could ignore this since this isn't harmful i think btw will check code if there some changes needed.

@mattmalec
Copy link

I've seen this before with my Kubernetes infrastructure. It occurs when RedisBungee doesn't have the chance to remove itself from the "heartbeats" key in the Redis server (i.e. not safely shutting down the server, a SIGKILL, etc.)

A heartbeat object is JSON formatted and looks like:
{ "server-id-1": "last-heartbeat-time", "server-id-2": "last-heartbeat-time", "server-id-3": "last-heartbeat-time" }
and etc.

You could fix this by either removing the key of the offending server from that object or by completely deleting the heartbeats key from Redis. You should restart all your proxies for data consistency if you remove the heartbeat key entirely, but it's not necessary if you remove the offending server from the object.

As a more permanent fix, a config option that removes any server that have missed the heartbeat after X amount of seconds would be nice to see. Let me know if you'd like me to PR that addition.

Hope this helps. Thanks!

@ham1255
Copy link
Member

ham1255 commented Apr 21, 2021

Thanks for info!

@ham1255
Copy link
Member

ham1255 commented Apr 21, 2021

I've seen this before with my Kubernetes infrastructure. It occurs when RedisBungee doesn't have the chance to remove itself from the "heartbeats" key in the Redis server (i.e. not safely shutting down the server, a SIGKILL, etc.)

A heartbeat object is JSON formatted and looks like:

`{

"server-id-1": "last-heartbeat-time",

"server-id-2": "last-heartbeat-time",

"server-id-3": "last-heartbeat-time"

}`

and etc.

You could fix this by either removing the key of the offending server from that object or by completely deleting the heartbeats key from Redis. You should restart all your proxies for data consistency if you remove the heartbeat key entirely, but it's not necessary if you remove the offending server from the object.

As a more permanent fix, a config option that removes any server that have missed the heartbeat after X amount of seconds would be nice to see. Let me know if you'd like me to PR that addition.

Hope this helps. Thanks!

But this kind of thing is harmless but if it causing issues please tell me!

@ham1255
Copy link
Member

ham1255 commented Apr 21, 2021

And also when that error happens what i can do is delete the server

@mattmalec
Copy link

mattmalec commented Apr 21, 2021

RedisBungee updates its heartbeat every 3 seconds to ensure the Redis server hasn't just randomly died. If its unable to update the heartbeat, it could be that a server has lost its connection to Redis, or simply that the server wasn't shutdown safely.

@ham1255 ham1255 added enhancement New feature or request good first issue Good for newcomers labels Apr 21, 2021
@ham1255
Copy link
Member

ham1255 commented Apr 27, 2021

i made pull request try it
#2

@ham1255 ham1255 linked a pull request Apr 27, 2021 that will close this issue
@ham1255 ham1255 changed the title Error proxy seconds behind! (Time not synchronized or server down?) spam May 13, 2021
@ham1255 ham1255 changed the title proxy seconds behind! (Time not synchronized or server down?) spam proxy seconds behind! (Time not synchronized or server down?) spamming console May 13, 2021
@ham1255 ham1255 added completed Finished! bug Something isn't working and removed enhancement New feature or request labels May 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working completed Finished! good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants