Lag on player join #59

Open
Waffleman123s opened this Issue Jun 29, 2016 · 101 comments

Projects

None yet
@Waffleman123s

I have installed the new version of ULX and ULIB and when ever a player joins the server spike lags for a few seconds and I need to know how to fix this asap!
I deleted all my addons and just left ULX and ULIB in and it still does it now when ever I delete ULX and ULIB the lag stops!!???

Please help me asap!

@sammyt291
sammyt291 commented Jun 29, 2016 edited

ULX / ULib version?
OS version?
Latest gmod version?
Errors in console?

And what have you already tried aside from removing other addons?
If nothing try deleting or temporarily renaming your ulx and ulib folders in /garrysmod/data.
Restart, and test with a fresh installation.

@Waffleman123s

0 ERRORS in console! We had the latest version of ULX and ULib and we create a fresh darkrp gamemode installed ULX and ULib FRESH and still Lags when people Connect!!!! FML

@Nayruden Nayruden changed the title from ULX and ULIB BUG to Lag on player join Jul 2, 2016
@Nayruden Nayruden added the feedback label Jul 2, 2016
@Nayruden
Member
Nayruden commented Jul 2, 2016

What OS? Are you using the latest version of GMod?

@sammaxis
sammaxis commented Jul 8, 2016

I believe I am experiencing the same issue as the OP.

It's a mild lag that only occurs on player join or leave. I have tried removing everything outside of ULX & ULIB as well as changing other settings.

Looking for some help.

@Mika32150

Same problem here, please fix Ulib and ULX. Before we doesn't have this lag when someone join / disconnect.

@Mika32150

For me i use Debian 8 with the last gmod update !

@davethestuff

Also experiencing this problem. Think it started since the latest gmod update.

@Mika32150

Someone work on the problem ?

@FlashModz

i'm some problems.

#61

Debian 8

@Nayruden
Member

I'm unable to duplicate this problem, but I've never tested on Debian before (I use Ubuntu). Will try installing Debian now.

@Nayruden Nayruden added investigation and removed feedback labels Jul 16, 2016
@Nayruden
Member

I still can't replicate this issue, and I don't see what Garry would have changed recently that would have resulted in this for Debian users. However, I took a wild stab at what the issue may be, so please try using the following slightly modified ULX and ULib versions and report back --

http://ulyssesmod.net/test_ulib.zip
http://ulyssesmod.net/test_ulx.zip

Note: due to the changes in these, there may be some errors. I tried to clean up any obvious problems, but this should work for this experiment.

@Nayruden Nayruden added feedback and removed investigation labels Jul 16, 2016
@sammaxis

I'm not using Debian. We're hosted on NFO and I'm experiencing the same issue.

@uRandomAlex

@Nayruden You should create another GitHub branch and make changes there. I don't want to ruin all my ULib/ULX modifications to test this.

@uRandomAlex

@Nayruden Have you tried to play on a live server, 10+ players (not bots)? Problems starts there. Every new joined player, that's on "Sending client info" stage (PlayerAuthed/PlayerInitialSpawn hooks), freezes server for a second.

@Nayruden
Member

I do not have the ability to test on such a server, @uRandomAlex.

@FlashModz
FlashModz commented Jul 18, 2016 edited

@Nayruden actualy in the test
Come my server :) - What your Steam ?

Live Test in Progress
RM-CommUnity

@PolSpock

@Nayruden same problem with test_ulx and test_ulib
(Linux debian)

@FlashModz

@Nayruden same problems on my Debian 8... with tes_ulx and ulib.

Solution ?

@Nayruden
Member

I need someone to volunteer their server for experimentation; otherwise, I'm just making wild guesses. I'm unable to do anything for the next week or two, as I am moving my household across the country.

@sammaxis

I suppose I'd be able to do that.

http://steamcommunity.com/id/sam_maxis/

@Violat0r

I can confirm we have this issue on our servers also, mainly on DarkRP. We are running on a Windows box. No errors, latest gmod, lastest ulib/ulx. When I use an addon like FProfiler and what not, its always ulx/ulib spiking in cpu when someone joins the game.

@FlashModz

Solution ?

@Violat0r
Violat0r commented Aug 3, 2016

I can volunteer our server(s) that have the issue http://steamcommunity.com/id/violator555/

@FlashModz

I have a dedicated server I am willing to mount a dev server with all the access that will be delivered you add me on steam

FlashModz

@Nayruden
Member
Nayruden commented Aug 4, 2016

Thank you, I will certainly take someone up on that offer. The larger problem is I am working out of a hotel at the moment. I'll be in touch when I am able to debug this. :)

@FlashModz

if patched ?

@Nayruden
Member

@FlashModz see #59 (comment). I did get internet yesterday but have other tasks that take priority over ULX this weekend.

@Nayruden Nayruden self-assigned this Aug 13, 2016
@Nayruden
Member

Trying to work out what could be causing this. Can one or more of you experiencing this please answer the following questions?

How many users do you have added in ULX? A rough estimation (10's, 100's, 1000's) is fine.
Does this happen every single time someone joins and leaves?
Is it worse with more people already in the server?
Any other trends you've noticed? (certain gamemodes, maps, etc)

@uRandomAlex

@Nayruden 10 users, every single time someone joins and leaves, it's worse with more people on server.

@Nayruden
Member
Nayruden commented Aug 14, 2016 edited

@uRandomAlex, please run "lua_run_cl PrintTable(ULib.ucl)" from your client and paste the results here or send it to me directly -- megiddo@ulyssesmod.net. Note that client side lua execution has to be enabled.

@uRandomAlex
uRandomAlex commented Aug 14, 2016 edited

@Nayruden Sent in e-mail.

@FlashModz

Here are the 16 laggiest hooks found during the last 781 frames:
Identifier ------------------------------------------------------------ Impact Executions Mean t /f Max time
Event "PlayerAuthed", hook "ULibAuth" 502.40% 1 247.408us 193226us
in "addons/ulib/lua/ulib/server/ucl.lua" (lines 1093 to 1097),
not found in Workshop addons
Event "UCLChanged", hook "ulx.updateMotd.adminsChanged" 250.21% 2 246.434us 97313.6us
in "addons/ulx/lua/ulx/modules/sh/menus.lua" (lines 71 to 120),
not found in Workshop addons

10 players or more is very laggy my server
yes whenever a player joins but its not this remark that from 12 players
DarkRP - rp_rockford_french_v2b

Thanks :)

@FlashModz

As mentioned if you need a server DEV I can provide you with a test for which I will participate with pleasure :)

@PolSpock

What's up ?

@Nayruden
Member

I've carved out some time this weekend in order to look into this more. I need to get a game plan in place for how I'm going to debug on a live server, then I'll be in touch with those who volunteered server time.

@Nayruden Nayruden added a commit to TeamUlysses/ulib that referenced this issue Aug 27, 2016
@Nayruden Nayruden Testing a possible fix for TeamUlysses/ulx#59 180d737
@Nayruden
Member

I wasn't able to hail any of the volunteers, but someone please test that out and report back.

@FlashModz

i'm currently testing

@Nayruden
Member

Can you hail me on Steam? http://steamcommunity.com/id/megiddo/

@FlashModz

Yes you're already in my friends :)

@Nayruden
Member

What's your nickname on Steam

@Nayruden
Member

I'm not sure that the issue @FlashModz was having is related to this. For @FlashModz, his client lags when he joins his own server (as opposed to all currently connected clients lagging when anyone joins), but other clients join without lagging. I advised @FlashModz that he may have too many client addons installed.

I have still been unable to duplicate this issue anywhere. If anyone else is still having issues, feel free to throw your input in here. :)

@FlashModz

if that does not bother you I warn you friend to invite you to when there lags if it reproduces

@Dayornight1999

I have tried the potential fix on a live server and it didn't seem to fix the issue for me.

@Nayruden
Member

@Dayornight1999, just to make sure we are all on the same page, please describe what your issue is exactly. Please be very detailed.

@Dayornight1999

I'll try to be as detailed as possible but there isn't much to talk about. Essentially players were starting to report lag spikes every now and then. I started to investigate by looking at net_graph 4 and came to the conclusion that every time a player joins/leaves it started to lag my server for a second. I deleted the addons that could have been the issue, but nothing fixed it. So I did some more research and landed on this page. I've tried the potential fix you posted a day ago, but this doesn't seem to fix it. It also breaks something with the ranks I think.

@Nayruden
Member

What happens with your ranks?

@Dayornight1999
Dayornight1999 commented Aug 28, 2016 edited

I'm not sure really, but this is an error I just received when someone left the server after refreshing the XGUI modules and refreshing the server data.

Edit: Click on the link for the error (more organized) http://pastebin.com/vAyyhUft

@Dayornight1999
Dayornight1999 commented Aug 28, 2016 edited

From what I can tell it seems to be removing people from current groups then readding them again, but I might be wrong. It's a weird issue and I couldn't find a patern yet.

@Dayornight1999
Dayornight1999 commented Aug 28, 2016 edited

Also getting this when I switch maps: http://pastebin.com/vZAV33Z1

Edit: menu not getting loaded? https://s12.postimg.org/infybdy8t/ulib_error.png

@Violat0r

Well good news is that it did reduce the lag on the server overall but it still does lag on player join.

I am also getting a lot of errors. http://pastebin.com/JfvmbTRX

@Nayruden
Member

If you're getting errors, try removing and redownloading from Git(Hub); if you were using the zips I posted a while back, you will have conflicts without fully removing and reinstalling.

@Violat0r

I am using the one from github not the link.

@Nayruden
Member

Did you ever use the zip? If so, you need to do a full reinstall.

@Violat0r

Never used it and I always delete the old versions of addons.

@Dayornight1999

It seems ULX or ULIB wasn't the issue after all. It seems to be the Achieve! addon from scriptfodder that was causing the lag spikes. If any of you have that on the server, I recommend you remove it for now and contact the addon creator just like I will do. Nayruden, thank you for trying to help us out and sorry for wasting some of your time investigating into this problem.

~Dayornight

@Nayruden Nayruden added a commit to TeamUlysses/ulib that referenced this issue Aug 29, 2016
@Nayruden Nayruden Revert "Testing a possible fix for TeamUlysses/ulx#59"
This reverts commit 180d737.
9f459dc
@Nayruden
Member
Nayruden commented Aug 29, 2016 edited

Reverted previous attempt at fix until I can fix the bot issue

@Dayornight1999

Removing Achieve! solved this issue for me, but maybe others are still having this issue despite not having Achieve! on their servers. We'll just have to wait for a response from them.

@uRandomAlex

@Dayornight1999 I didn't even know about existance of that addon. Seems that the problem is with sending too much data to player or with too expensive calculations on player join in ULib/ULX. This issue is not affecting servers with only a few addons because they don't use CPU/network that much and therefore ULib/ULX can have this resources "right now", without need to wait for them.

@Dayornight1999
Dayornight1999 commented Aug 30, 2016 edited

Didn't the person some posts above us say he removed all his addons except for ULX/ULIB and he still got the lag spikes?

@Nayruden
Member

ULX performs no calculations on join and sends less than a kilobyte. That's
partly why I'm so confused why anyone is experiencing issues.
On Mon, Aug 29, 2016 at 20:04 Dayornight notifications@github.com wrote:

Didn't the person some posts above us say he removed all his addons except
for ULX/ULIB and he still get the lag spikes?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#59 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABAz6L7d2mgULqtgb7jLP3UeJTtQvMrks5qk4GvgaJpZM4JArNy
.

@Violat0r

I honestly think it might be darkrp or an addon for darkrp that relies on ulx/ulib for perms. We have other servers with the same issue, all darkrp gamemode. Our other servers that are not darkrp such as TTT and what have you, do not have this issue at all.

@uRandomAlex
uRandomAlex commented Aug 30, 2016 edited

@Nayruden May be ULX don't perform any calculations, but ULib does - UCL is quite heavy. I guess you should look into it.

@Nayruden
Member

I assume you meant ULib. My previous statement about ULX applies to ULib as
well.

On Tue, Aug 30, 2016 at 7:28 AM, uRandomAlex notifications@github.com
wrote:

@Nayruden https://github.com/Nayruden May be ULX don't perform any
calculations, but ULX does - UCL is quite heavy. I guess you should look
into it.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#59 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABAz0hzVtx0afHXqR5rfWSXKxBnWml-ks5qlCHYgaJpZM4JArNy
.

@uRandomAlex

@Nayruden Yeah, I've changed it but seems that GitHub doesn't send you new e-mail. Anyway, profiling with FProfiler on player join shows that UCL is most expensive.

@PolSpock

same analysis as @uRandomAlex

@SticklyMan
Contributor

Random question for all of you having issues, what is your value of 'ulx_showmotd' set to?

@FlashModz

ULX MOTD has been desactivated

@Violat0r
Violat0r commented Sep 5, 2016

Disabled, using something else.

@Nayruden
Member
Nayruden commented Sep 5, 2016

Please check if that makes any difference for you all.

@Nayruden
Member

Any luck with the changes from last week?

@sayaloli1

i fixed it with changing to an older ulx / ulib version ( 2 versions older )

@Nayruden
Member

Haven't heard anything back, is it safe to close this issue as solved?

@Violat0r

Nope still lags :(

@Nayruden
Member

I apologize this has been extremely frustrating for everyone (the developers included). It feels like we've chased a bunch of dead ends up to this point, and we still don't have any leads as to what the problem is. We'd like to see if there are any patterns among folks who are experiencing the lag -- please help us out by filling out this 2-3 minute survey https://goo.gl/forms/6y7jdpHpIfouvcHA3

@Dayornight1999

I don't know if I'm the only one with this issue, but the bring command and teleport command seem to be broken. Ulx bring doesn't work (it keeps saying it can't find a place to put the target even though there's plenty of space) and ulx teleport teleports the player into you and not where you're aiming at. Just thought i'd mention it.

@Nayruden
Member

@Dayornight1999 I'd suspect a conflicting addon there, but that's off topic of this issue.

@uRandomAlex

@Nayruden Have you tried comparing current ULX/ULib to older versions (2-3 from latest release)? People says that they don't have this issue. Try profiling both of them (old version without lag and new version) with FProfiler. I think there will be something that consumes many resources compared to old version.

@Violat0r

We broke darkrp on the server and it got rid of the lag but of course that breaks other addons. I am fairly sure it is darkrp or a darkrp addon that uses ulx for perms and what have you. I will do more testing to try and narrow it down.

@Hazbelll
Hazbelll commented Oct 19, 2016 edited

A little off-topic here, but why is ULib a thing again? Kinda feels like Useless Library.

What Developers or Server Owners use it for anything other than ULX (Which they have to due to its dependency on it)? Wouldn't it make more sense to have ULX use its own specifically tailored and optimised library and get rid of ULib?

@Nayruden
Member

That's more than just a little off-topic; if you'd like to discuss this
further on our forums (ulyssesmod.net), I'd be more than happy to share my
thoughts.

On Wed, Oct 19, 2016 at 10:12 AM, Hazbelll notifications@github.com wrote:

A little off-topic here, but why is ULib a thing again? Does it stand for
_U_seless _Lib_rary?

What Developers or Server Owners use it for anything other than ULX?
Wouldn't it make more sense to have ULX use its own specifically tailored
library and get rid of ULib?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#59 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABAz3VDzYV6lOulnaxfomM5xfl0IOzUks5q1jNYgaJpZM4JArNy
.

@roboderpy
roboderpy commented Oct 25, 2016 edited

Because of way how ULib UCL handles groups, it realoads config file on any change of groups, or player join. Because of slow KeyValues parse function, you are getting this lag. If @Nayruden would mind about remaking parse function, lags will be gone. Or, he can do even more - change how UCL handles group changes.

@Nayruden
Member

Survey results indicate that is not the problem here, though your idea may
be worth exploring as well. Sorry I haven't had a chance to continue to dig
into this more lately; I've got another idea of what may be causing this
issue, just waiting for my schedule to clear up.

On Tue, Oct 25, 2016 at 7:40 AM, DBotThePony notifications@github.com
wrote:

Because of way how ULib UCL handles groups, it realoads config file on
any
change of groups, or player join. Because of slow KeyValues parse
function, you getting this lag. If @Nayruden https://github.com/Nayruden
would mind about remaking parse functions, lags will be gone. Or, he can do
ever more - change how UCL handles group changes.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#59 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AABAz7TMLeDgaqNnr2w3OWN41anznbVsks5q3fi8gaJpZM4JArNy
.

@sweepyoface

I'm having this issue, and it's quite crippling.

@Nayruden
Member
Nayruden commented Nov 7, 2016

I'm sorry it's taken us so long to address this issue to everyone's satisfaction; I believe I will have a free day this week, and if so, it will be dedicated to tackling this. Thanks for being patient and understanding.

@sweepyoface

I'll be glad to offer you my server to test on if we haven't fixed it by that time.

@sweepyoface
sweepyoface commented Nov 7, 2016 edited

Could "[S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed." be contributing to it? I've tried for hours to fix that and I haven't been able to.

@Nayruden
Member
Nayruden commented Nov 11, 2016 edited

The survey results showed that most people having the issue are running DarkRP with 100+ addons. Even running DarkRP with lots of addons on my laptop, with eight players and other things bogging down my system, I still can't replicate this. Someone experiencing the lag please reach out to me on Steam today -- http://steamcommunity.com/id/megiddo/

@sweepyoface, to answer your question, no, your steam API error wouldn't have anything to do with this.

@Nayruden
Member
Nayruden commented Nov 11, 2016 edited

Working with @Violat0r on this issue, it seems that the lag is caused by FAdmin or an interaction between FAdmin and ULX. Disabling FAdmin seems to address the issue.

I may try removing CAMI from ULib in an experimental branch to see if that solves the problem as well; unfortunately, we've had nothing but issues from DarkRP, FAdmin, and CAMI, and I don't have the patience to keep working around them.

Another oddity observed with @Violat0r was that only one of his two DarkRP servers had this issue. The only difference noted was that one was using DarkRP configured with MySQL (the laggy one) and the other was configured for SQLite (the non-laggy one).

@sweepyoface

We seem to have solved the issue by simply renting a more powerful server.

@FPtje
Contributor
FPtje commented Nov 12, 2016

I may try removing CAMI from ULib in an experimental branch to see if that solves the problem as well; unfortunately, we've had nothing but issues from DarkRP, FAdmin, and CAMI, and I don't have the patience to keep working around them.

I've heard nothing about this for quite some time now. Where is this sudden desparation coming from? I'll look into FAdmin MySQL performance, but that shit should all be asynchronous.

What module is being used, tmysql or mysqloo? Does the lag still happen when the other module is used?

@Violat0r

tmysql, although I highly doubt it is sql that is causing the lag.

We want people to try disabling FAdmin on servers to verify our results yesterday and report back.

@FPtje
Contributor
FPtje commented Nov 12, 2016

The best way to debug lag issues is with a profiler. Just run your server as normal, preferably with all the addons. Run the profiler, serverside or clientside depending on whether the lag is happening serverside or clientside. Have someone join while the profiler is running. After the lag spike has happened, stop the profiler. Then observe the most expensive function calls (in FProfiler that should be the second tab).

Screenshot the results, post them here.

FProfiler is very likely to give you the exact source of the lag. There will be no guessing.

@Violat0r

We have done that already. I do not have the results anymore but it is very similar to this past comment

FProfiler lags our purge server to death with 40 people on when it is on so we cannot use it sadly. I used https://steamcommunity.com/sharedfiles/filedetails/?id=682765484 and it also pointed to the same hook. We have come to the conclusion that the hook spikes with darkrp as our other servers do not lag and the survey results all point to darkrp. We believe it is FAdmin which is why we want people to test it out.

@FPtje
Contributor
FPtje commented Nov 12, 2016

The profiler of the past comment only mentions ULib. If like you say your profiler also refers to the same hook, then it's clearly ULib.

Also, you don't need to run FProfiler full time. It's most efficient when you start it right before someone joins and end it right after.

I can't see the data behind the survey, but it is most likely that most of the servers reporting lag spikes are running DarkRP, maybe even 100% of the server owners run DarkRP. Not necessarily because DarkRP causes the lag spikes, but simply because it's the most popular gamemode. Also, there are many DarkRP specific addons which could also be causing the lag spikes. Of course there's going to be a strong correlation between DarkRP and these lag spikes. This doesn't imply causation.

The OP said that removing ULX and ULib solved the issue. @Nayruden is saying that removing FAdmin solves the issue. The profiler posted earlier on points at ULib. Unless REAL evidence is provided that FAdmin is directly causing a lag spike, there is NO reason to believe that it actually is. I don't trust any of you people's ability to judge whether a lag spike is "gone" when you remove an addon.

@Nayruden
Member
Nayruden commented Nov 12, 2016 edited

The profiler we ran yesterday pointed to something else entirely, and it wasn't a large enough impact to matter. True, DarkRP is a popular addon, but that's where the evidence points at the moment (100% of results). I have done everything within my power to replicate the problem and have been unable to. It could be a common addon between the lagging servers, there's just no way to know for sure with what we have so far.

Unfortunately, our advice to all our users in general is to disable FAdmin anyways, since it causes so much heartache for server owners.

@sweepyoface

I'd like to point out that the issue was still there with no addons at all (just not as bad) but vanished as soon as we upgraded the machine. That and the fact you said FProfiler lags you to death leads me to believe we're overthinking it and you just need better hardware.

@FPtje
Contributor
FPtje commented Nov 12, 2016

FProfiler can indeed be expensive when there are many players. That's because it logs every function call.

@mcNuggets1

I only see people complaining about their 100+ addons shitty optimized pay to win server lagging.
I use ULX and ULib since I started my 4 servers, I never had any negative experiences with it, besides it disabling connect/disconnect messages sometimes and sometimes not.

@FlashModz
FlashModz commented Dec 15, 2016 edited

My test Hooks : test players in join and spawn

Here are the 16 laggiest hooks found during the last 2788 frames:
Identifier ------------------------------------------------------------ Impact Executions Mean t /f Max time
Event "UCLChanged", hook "xgui_RefreshPlayerCmds" 125.05% 1 4.827us 13456.3us
in "addons/ulx/lua/ulx/xgui/commands.lua" (lines 289 to 357),
Net receiver "urpc" 45.27% 3 5.241us 14312.6us
in "addons/ulib/lua/ulib/client/cl_util.lua" (lines 7 to 23),
Default scene rendering / GM.RenderScene 17.72% 429 12401.01us 27150.9us
in "gamemodes/base/gamemode/cl_init.lua" (lines 289 to 290),
Usermessage receiver "tsayc" 12.17% 2 39.696us 8621.50us
in "addons/ulib/lua/ulib/shared/messages.lua" (lines 124 to 139),
Event "UCLChanged", hook "xgui_RefreshGroups" 07.29% 1 11.889us 5100.51us
in "addons/ulx/lua/ulx/xgui/groups.lua" (lines 1264 to 1267),

During server is Online :

Here are the 16 laggiest hooks found during the last 767 frames:
Identifier ------------------------------------------------------------ Impact Executions Mean t /f Max time
Usermessage receiver "tsayc" 22.88% 2 23.347us 9976.50us
in "addons/ulib/lua/ulib/shared/messages.lua" (lines 124 to 139),
Default scene rendering / GM.RenderScene 21.09% 767 8249.961us 24352.6us
in "gamemodes/base/gamemode/cl_init.lua" (lines 289 to 290),

Machine test =

Débian 8
64Bits / Package 32Bits added
Network : OVH 1GBPS
Type dedicated
Debian are Up2date and Online Since 2 Hours

Thanks all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment