Minecraft::Packet - PacketPlayInCustomPayload

[SecretlyJealous]

Version: git-Paper-014578a7 (MC: 1.12)
Hi, I've been having a recent problem where all my creative servers have a large amount of movement lag and commands seem to be running slow. I took a timings and "Minecraft::Packet - PacketPlayInCustomPayload" is showing as extremely high and seems to be the culprit.
Timings: https://timings.aikar.co/?id=b9d5e18685fd46bfb3292670f1217d85

Not really sure what to do so I wanted to ask what the cause for this could be and the best way to prevent it? It seems as if someone is using a hacked client to spam packets or something but it's hard to say.

[electronicboy]

that would imply that something is sending a reasonably sized chunk of info into that, potentially clients or even bungeecord plugins sending data to the server. iirc, protocol lib or some plugin like that had a way to dump out what we being sent to the server so you could probably see what is being sent through that

[SecretlyJealous]

Here's a log of 10 minutes showing the packets being received, but I don't see anything unconventional going on.
https://www.dropbox.com/s/sachcdfxd97sfi1/Packets.log?dl=1

"[00:23:56] [Netty Epoll Server IO #1/INFO]: Packet received: MC|ItemName" There's lots of ItemName packets being received but I'm not sure if that would cause the type of problems I'm experiencing.

[electronicboy]

MC|ItemName is from anvils, so could potentially be somebody is using an anvil with a modded client or something to send something stupidly long, no idea if the server does anything to say in the log that they used an invalid length; the bungeecord messages also indicate that something is using the messaging channel to send messages around; Doesn't look like timings really captures what is listening on channels

[SecretlyJealous]

The amount of packets being received doesn't really explain the crazy high timings though :(

"Minecraft::Packet - PacketPlayInCustomPayloadcount(242) total(81.73% 75.449s, 715.16% of tick)avg(311.77ms per - 357.58ms/1.15 per tick"

[DrkMatr1984]

Hmm, yeah it seems to be something with packets. I would try disabling any plugins you have that use protocollib first, but it almost looks like your bungeecord is flooding your backend server with packets. Do you maybe have a voting plugin that is installed on both your bungeecord and backend servers, or a global messaging plugin that sends the messages globally across bungeecord and ur spigot server? There may be an exploit to one of those plugins that someone is aware of, or a bug in a voting plugin where the developer forgot to end a loop, and it's just sending packets over and over again from the Bungeecord to the server.

[DrkMatr1984]

Also, looking again at your packet logs, the most packets seem to be sent as soon as someone logs in, and that seems to be when your timings suffer the most. It could be you have too many plugins using the PlayerLoginEvent or BungeeCords PostLoginEvent, and whenever someone logs in, at least 5 different plugins send data from the bungeecord to the server, probably reporting things to a server plugin like votes, skins, playernames, etc. To reduce packet load like that, perhaps reduce the amount of bungeecord plugins you have that communicate with the server.

[SecretlyJealous]

Found the issue, related to chat formatting taking up a very long time but I had to use WarmRoast to identify this since the timings were very vague.