Multiplayer Framerate

[Rashaln]

I attempted to set up a tModLoader server recently, but experienced severe frame skipping and a notable overall framerate decrease when connected to it. I have so far found no fix for it, and I can only seem to trace it to tModLoader in general. I am running a tModLoader server on a Ubuntu 16.04 Live Linux box on my home network, connecting from a Windows 10 client. No matter how far I turn down Terraria settings; tiny window, everything off(except frame skip) or turned down, I lose 10-20 FPS (With frequent stuttering from frame skip) when connecting to my server. I've done a clean install of Terraria, and tested multiple configurations; vanilla Terraria runs fine in single- and multi-player at high graphical settings, tModLoader runs fine in single-player with and without mods, at the same graphical settings, but tModLoader multiplayer, with or without mods, has significant performance problems. My server doesn't appear to be the direct cause, as it stays under 50% of either core while tModLoader is running.

Some system info:
Client:
Terraria 1.3.4.4, TML 0.9.1
Windows 10 Pro 64bit build 14393
AMD Athlon 64 X2 4600+ 2.4ghz
4GB RAM
Nvidia GeForce 9600GT

Server:
Terraria Linux server 1.3.4.4, TML Linux 0.9.1
Ubuntu 16.04 LTS 64-bit (Running off USB with persistence)
Intel Core 2 Duo E8500 3.16ghz
4GB RAM
Onboard graphics

Perhaps I'm just missing something, (I've had bad luck with troubleshooting lately) but hopefully someone can help. I will provide more detailed information if necessary.

[Chicken-Bones]

What happens when you run the server locally on your windows machine?

[Rashaln]

Ah I can't believe I forgot to test in that configuration. My computer chugged a bit, as expected, but the local game ran with far fewer hitches and NPC warping than it did when connecting to my server, and Host&play actually ran surprisingly smoothly.

[Chicken-Bones]

And last thing to test is, Could you run a windows server on another machine? (see if it's a remote/local issue or a windows/linux issue)

[Rashaln]

Connecting to the Windows server did provide a better experience than the Linux one, but there is still a disconcerting performance hit. The game didn't drop large numbers of frames, but I did still experience a 10-20 fps drop overall.

[Rashaln]

Any word on this?

[Pop000100]

try disabling Frameskip my brother had similar problem with it

[BrendanatorX]

I've got the same issue tbh.
using a few mods with tmodloader, haven't tried without, but singleplayer runs perfectly.
multiplayer opens the tmodloaderserver.exe, and while its running, fps often drops down to 30 and then things get slow mo, frame skip just makes things choppy, on or off it doesn't improve things.

One thing i did just try though was setting terraria to only my first 2 cores (i have 4 total)
Then after making a server, switched tmodloaderserver.exe's affintiy from to the other 2 cores seperate to terraria, which didn't help, but after switching both back to all 4 cores, all of a sudden there's no fps issue and its running like singleplayer, perhaps helpful info?

[Klouph]

Me and some friends had this problem too, whenever we join a steam server, steam.exe's cpu usage goes up to 30% at minimum, reaching up to 50% sometimes, with an average usage of 40%. This only happens when playing modded Terraria multiplayer; tested a few other games and it didn't happen.

[Motuzkov]

Same

[Gregxcore]

Also getting this issue!! Vanilla terraria sp fine/ vanilla mp fine. Tmod with mods sp is fine however the issues arise when using tmod loader on MP causing some kinda of performance drops especially when opening chests.

Some system info below
Win 10
Fx 6100
16gb of ram
r9 390 asus strix vid card

Hopefully this issue is fixed with the next version of tmod.

Also forgot to mention all frameskip options had really no effect.

[Gregxcore]

Just tried to get multiplayer working again but it constantly drops frames, usually around 1-3. This issue is not present in the single player aspect only the multiplayer.

[asen1987]

same issue ..

[asen1987]

But I found that ,,when I use the mouse to hold the title bar to drag the game window，CPU usage becomes normal.

[Opoz]

Having this same problem as of two days ago. Worked perfectly fine before that. Any ideas?

[AlbertThePidgey]

FUCK !

[Rashaln]

Tried running Terraria 1.3.5.3 with tModLoader 0.10.0.2 today, with the same setup otherwise, and there is no difference. No, it's not background processes hogging resources, nor frameskip causing the issue. I'd quite like it to be an issue on the user-end, leaving me looking like a whiny idiot, but I can't put my finger on anything, from my machines to my network, that would directly cause this.

[DBotThePony]

bump

[EssenceofLuna]

bump because same issue

[tkerkering]

I'm having the same problem with a friend
Singleplayer is fine but multiplayer is unplayable
Tried hosting with hamachi, port forwarding, tmodloaderserver/tmodloaderserversteamfriends and host and play.

[jurrejelle]

Bump, still have this issue

[jurrejelle]

I FOUND A SOLUTION: Putting your game to FullScreen fixes the issue for me and 4 others who had the same issue

[cryzed]

I'm also having the same issue, on Linux, latest version Terraria (Steam) and tModLoader. Singleplayer works fine, multiplayer is a nightmare, especially during boss fights with a friend where my FPS quickly drop to below 30 FPS and in some cases to 1-2 FPS and freeze my game up.

What @jurrejelle said had no effect for me at all, the game has huge framedrops with or without fullscreen, with or without desktop effects enabled.

Did you consider what @Opoz said and triage the commits during that time period? Maybe something changed that doesn't work at all well with Terraria's networking code?

[Chicken-Bones]

How many players, does it happen with just you? What about if the server is run on your machine and you connect using 127.0.0.1?

[DBotThePony]

This is still an issue. And yes, i run server o my local machine (as separate process) and connect to it using loopback address. Even without ANY mods, tModLoader WILL slowdown your client. And this is not server's issue, since server's simulation speed is almost always near 1 (unless met crazy conditions, such as insane mob spawnrates, but still maintain normal sim time most of time), and it just can't choke in extreme situations, where clients just start to freeze. And singleplayer experience no slowdown whatsoever, unless said crazy conditions are met in singleplayer.

PC performance is not the problem (Ryzen 7 1700X, OC is Windows 10)

[cryzed]

@Chicken-Bones hello, this was with 2 players only. Running the same world (and the same scenario: a small boss fight) in singeplayer mode stays at a constant 60 FPS. I can't test "Host & Play" right now (which would run the server locally and my client would connect to it, I assume), because the game crashes for some reason.

I'll look into this more tomorrow. For now I came to the conclusion though, that it might simply be related to my hardware. Trying to connect to a multiplayer server with SpiritMod/Thorium and Terraria Overhaul might simply be too much for my i7-930 (the single-core performance is abysmal when compared to newer generations i7).

To that end I'm hopefully upgrading to an i7-7770K the coming week. I'll let you know how that goes. Although I think that's just treating the symptom and not the cause, since my CPU can handle this scenario in a singleplayer scenario -- I'm not sure why there would be some strange overhead when I connect as a client to a different server; I'd think my CPU would have even less to do.

[cryzed]

Ok, I tested it again today with the new CPU. The framerate has been much improved. It's probably a combination of people underestimating what kind of system (especially single-core performance) you need to run certain Terraria mods and insufficient upload bandwidth from the server.

I still think there's something overly affecting the performance in the networking code, but the symptom has been fixed at least by upgrading my hardware.

[Chicken-Bones]

There's still something up here, tModLoader should not be slower than Terraria at a base level but we're not sure exactly what it is. The original post had the issue even on a windows client. I'm not aware of any way the server can affect the framerate of clients, except potential packet spam

[Jofairden]

I'm fairly sure certain mods absolutely destroy the network because the authors don't know what they're doing. If it's without mods, then it definitely either is a bug in tML or remnants of unloaded mods.. I guess we as developers should sometime do some tests together

[Velocco]

I got this issue too. Even with only 2 players. It makes me less motivated to play Terraria.

[Chicken-Bones]

I recently observed this on my machine, in both vanilla and modded. I've never had the issue before so I don't know what caused it, but I think my graphics card is unwilling to accelerate it in windowed mode.
I can confirm that the only thing which worked for me was going fullscreen.

Given that this is a vanilla issue, we probably won't be fixing it, but I'll leave this issue open for others who have the same problem.

[nathanfranke]

Same Issue

[junetried]

Please don't post 'same' and other comments that don't advance the issue's state or provide insight. This issue is clearly a low priority, so you will need to be patient.

[Shadowblitz16]

optimization a low priority? sound like starbound

[jaezi]

Please don't post 'same' and other comments that don't advance the issue's state or provide insight. This issue is clearly a low priority, so you will need to be patient.

how is being unable to play low priority? what kind of helper are you?

Edit: nah you aint a helper, minimod, you can't tell us to stop posting if you aint someone with authority, and even so, no one with 2 brain cells minimum would treat BEING UNABLE TO PLAY as a LOW PRIORITY post

[junetried]

I didn't decide it was low priority. I don't have such authority. However, seeing as the issue is almost three years old and no solutions have been created, one can infer that it is low priority.

[madolinn]

I'd like to chime in and say I'm experiencing similar issues. Certain monsters seem to be more heavily affected by DeSync than others.

Configurations I've tried testing on the same enemy - Queen Bee.

With tModloader

• Host & Play - Following her screen wide charge attack, teleports unexpectedly.
• Dedicated Host : Locally - Following her screen wide charge attack, teleports unexpectedly.
• Dedicated Host : Separate PC - Following her screen wide charge attack, teleports unexpectedly.
• Singleplayer - No teleporting.

Without tModloader

• Host & Play - No teleporting.
• Dedicated Host : Locally - No teleporting.
• Dedicated Host : Separate PC - No teleporting.
• Singleplayer - No teleporting.

Again. It's only a selection of monsters that are affected. Even some vanilla monsters are affected. Perhaps it's just a bad syncing with a commonly used component of these monsters?

[Jofairden]

It is not 'low priority', it is just that it's a very difficult problem to tackle.
Apparently enough people have the issue over the years, but none of us have it. We hardly have any log files, mod list and associated version numbers. When issues are very hard to reproduce they tend to become stale in terms of possible solutions. There are a number of improvements there would be in the next release so I'm really curious if the issue stays as active as it has been. My recommendation is to wait for 1.4 TML and see if the issue still arises.

[junetried]

The issue is seemingly fairly easy to reproduce. I would be happy to collect logs if you think it would be useful.

[okh4]

same issue. multiplayer is unplayable

[DBotThePony]

Forgot to mention, that when network traffic suddenly drop from server (for whatever reason), client simulation speed and framerate return back to normal. although mobs half-update themselves and, in the end, you either lose connection to server or if connection is restored, performance issue come back.

It seems that netcode is slow or doing something unexpected at updating tracked entities when playing multiplayer with tModLoader. And this, is not, in any way, a Vanilla Bug. Sure, in vanilla you get slightly more intense CPU load while playing multiplayer, but it doesn't get unplayable at any point at gameplay. Even if spawning thousands of mobs using modded server like tShock.

Could also be related to CPU latency, since if you are team Intel with their newest CPUs, you get much less latency than Ryzen (Zen, Zen+. Zen 2 (and likely upcoming Zen 3) do not suffer latency issues and are roughly equal to Intel CPUs)

Offtopic

optimization a low priority? sound like starbound

Windows is low priority*. Install any Linux distro and enjoy stable 60 FPS in Starbound in the most insane conditions.

[junetried]

This issue isn't exclusive to Windows.

[Solxanich]

There is a lot of comments here, so before I start to dig in to an independent investigation to try and figure out the causes (I'm suspecting this is a compound issue off the bat), I'd like to summarize the information that is here:

Platforms observed on: Linux, Windows

Versions of tmodloader listed: 0.9.1, 0.10.0.2. Reported as still being an issue as of Sept 2020

Clients experience frame-rate loss in both 64-bit and 32-bit versions.
Servers are assumed to be 32 bit, bc vanilla and no indication otherwise.

FrameSkip on and off doesn't fix the underlying frame rate drop. Just induces teleporting.

Gamemodes experienced in:
Multiplayer: host&play, dedicated local, dedicated remote: basically all forms

Additional details that have been mentioned (and are not necessarily limited to gamemode):

1. Per Malpheus: "High particle effect weapons such as magic and ranged weapons are particularly bad, and calamity rogue weapons such as celestus kill FPS in fights like these too."
2. Per ChickenBones & Jurrejelle: Framerate improves when going in to full screen mode. ChickenBones elaborates this was seen as a hardware acceleration issue, of which I would personally chalk up as a Vanilla gpu compat issue. Make sure graphics drivers are up to date.
3. Observed independent of installed mods - meaning even when there is no mods
4. Per BrendanatorX : "but after switching both back to all 4 cores, all of a sudden there's no fps issue and its running like singleplayer, perhaps helpful info?"

Finally, my hypotheses and my planned data to confirm or deny:

1. I anticipate that RAM handling for multiplayer terraria is significantly worse then multiplayer minecraft (ie the overhead per player is non-neglible); combining this with a 32-bit server would lead to anticipation of frame drops. Test via performance monitor recordings on RAM usage for each process of tmodloader and related server.
2. I also suspect there is a CPU handling issue; the mention that switching from 2 dedicated for each to 4 for all giving improved performance makes me suspect that something is artificially capping CPU core usage. This would mean I would see a flat line on core usage below 100% in performance monitor recordings for the given processes.
3. It also sounds like particles and AI can (unfortunately) easily slow down the game, in both singleplayer and multiplayer. In order to induce this condition, I'll use AlchemistNPC's Expiration (horrible amount of particles) + Lilith Charm (bees while using magic weapon) and GRealm's Amulet of Balance (horrible amount of bees). Again, should be useful for observing in performance monitor recordings.

Hopefully I find the cause(s) and can help devs get this addressed.

[Solxanich]

Preliminary results indicate the following (note I had to add in Fargo Souls and use Soul of eternity plus the previous three items mentioned to consistently lag the game out for my PC):

1. RAM is not the cause. I didn't see any RAM issues
2. Regarding network usage: Network usage has a high variability while frame drops/skips are happening, suggesting CPU is bottlenecking the networking response.
3. By far the most interesting, and which basically confirms that the CPU is the bottleneck is when one takes the lag inducing item combination (Soul of Eternity + Expiration + Litlith Charm + Construct of Balance) and attacks random air while watching both Task Manager's GPU performance and resource monitor Tmodloader's CPU performance. One will observe the following: CPU usage will rise, and GPU usage will drop (thus CPU is bottlenecking the GPU). For my i9 + rtx2080, the corresponding rise was 4% up to 8% on the client CPU (and I can't get CPU usage above 8% either on client) and a drop of client GPU from ~35% down to 8%. Server CPU usage goes up to 2.5%, but shows no sign of lagging (and is handling at right speed).

Continuing from point 3, this would suggest that I would be able to replicate the same CPU cap in singleplayer - and in fact, it does! Same ~ 8% cap out in singleplayer from the same item combination (although it doesn't skip as much since the starting CPU usage in SP for me is 2%).
Compare to Vanilla: SP is 2%, MP is also 2%

TL;DR: The code for tmodloader (and possibly vanilla) appears to be self-bottlenecking to default to only use one CPU core, when the client could arguably use more (roughly 2 client cores to 1 server core). This directly causes frame-skipping. Also, this means that single-core performance is what matters here.
Running tmodloader in multiplayer also uses up roughly 25% of a Client's core as compared to vanillas 0% change. So somehow tmodloader is using up a higher client percentage then it should in multiplayer which is exasperating the single-core performance limitation.

The next point is... what is tmodloader doing in multiplayer's client to hog performance? Is it running its own server and then connecting to a different one? Find out in the future

Also, as to why certain things worked for others:

1. Full-screen offloads CPU processing on background tasks, potentially freeing up space on the core (and thus resolving).
2. Upgrading your single-core performance would make it not as bad a problem.
3. If its going into multi adding 25% hog on performance... mods wouldn't come close to that easily.

[cryzed]

@Solxanich thank you for looking into this! Very interesting write-up so far.

[Mirsario]

The main reason why multiplayer lags is the way players are synchronized -- via clientClone(). We should eliminate that method and any similar practices.

[Chicken-Bones]

clientClone is a good hypothesis, for why issues only occur in multiplayer, and even in vanilla, but I want to see some profiling or instrumentation. We could add via monomod

[Solxanich]

Thanks ChickenBones and Mirsario for your input - having finished reviewing the code, I don't believe clientClone is the particular cause; rather I'm more suspicious of PlayerHooks.SendClientChanges (which is still within the same general code path, but doesn't seem to ignore sending data that hasn't changed natively). That said, I support that player syncing is a problem, as Mirsario says.

Further, I did also find the following issues within the updating structure, but doubt any are going to be the major fixer:

1. Netplay.cs.patch -> initialize server -> should be Clients[i].ReadBuffer = [ushort.MaxValue] for consistency
2. Main.cs.patch -> Update & DoUpdate -> potentially calls base.Update(gametime) twice when doesn't early return (remove call at end of DoUpdate)
3. Main.cs consumeAllMainThreadActions duplicates??? Netplay.cs on update (in all honesty I think this added code should be removed, and swap the synchronous to MainThreads. That ways it's not before, during, and after updates)
4. PlayerHook CustomBiomeMatch should be removed from "flags2" group in TrySyncingMyPlayer and moved to be part of the formal implementation "PlayerHooks.SendClientChanges"

[Solxanich]

As I predicted, 'fixing' the last four items had no real impact.
Further, playing around with the code, neither PlayerHooks nor clientClone appear to be directly responsible for the core performance issue (although both contribute a small amount).

I (unfortunately) just noticed that NetMessage (the thing called on literally everything) short-circuits in singleplayer. Meaning, can't discount the vast majority of calls as I had. I anticipate I'll need to read through more code to look for possible breakdowns.
A possible test I'll try:

1. removing the shortcircuit and seeing what performance looks like in singleplayer. Assuming it runs, should give an idea if I'm even looking in the right place. If it gets as bad as multi, at least we know it involves NetMessage, be it internal or external

[Chicken-Bones]

Try a profiler!

…

On Thu, 7 Jan 2021, 3:23 pm Solxanich, ***@***.***> wrote: As I predicted, 'fixing' the last four items had no real impact. Further, playing around with the code, neither PlayerHooks nor clientClone appear to be directly responsible for the core performance issue (although both contribute a small amount). I (unfortunately) just noticed that NetMessage (the thing called on literally everything) short-circuits in singleplayer. Meaning, can't discount the vast majority of calls as I had. I anticipate I'll need to read through more code to look for possible breakdowns. A possible test I'll try: 1. removing the shortcircuit and seeing what performance looks like in singleplayer. Assuming it runs, should give an idea if I'm even looking in the right place. If it gets as bad as multi, at least we know it involves NetMessage, be it internal or external — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#114 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC6ZCIC7YEGMMKK2TECXD3SYVAMZANCNFSM4C5UPZZA> .

[Solxanich]

@Chicken-Bones - I've tried a profiler; in general I'm missing a lot of dll in the solution to be able to determine what the 2nd unknown function is, and that has been the main reason I've avoided it and sorted code instead - with the change to so many more possibilities, seems little choice. I'm still tracking down where all the missing dlls were moved out of the VS solution, but it's pretty clear the 2nd unknown is the cause. I'll update if I get the profiling for the cause proper

Multiplayer (~80000ms cpu time)

Singleplayer (~15000ms cpu time)

[Solxanich]

Got it to load finally, after a few attempts - the heavy load path is:

Which suggests we're bottlenecking in ISocket.IsDataAvailable in TcpSocket.cs. I'll investigate further as to why we're breaking down there with calling GetStream.DataAvailable

[Solxanich]

Found it. Somebody removed the horrible implemented but needed line thread.sleep(1) in netplay.cs without actually fixing the rest of the code to allow removing it. (to be fair, that would be a lot of work to fix it, and is a Vanilla code change at that.)
That line is basically saying, instead of letting the thread that checks for new data continually run on the CPU (thus taking away time from other processes), make the thread sleep for 1 millisecond between calls.
Personally, I'd rather have significantly more CPU availability than 1ms less latency...

I'll make a quick pull request to add the line back, for now. @Chicken-Bones - 1.3 or just 1.4?

[tkerkering]

@Solxanich You are a gift sent from the gods in heaven if that fixes the issue. 👍

[IsaacElenbaas]

I can confirm that the revert fixes the issue. It was alright aside from boss fights (with six players) until I had more particles coming off my player while running, at which point it became unplayable. Running in one direction for ten seconds would either slow the game to a snail's pace, or with frame skip on display various graphical glitches (half-drawn frames) and make cursor movement jittery while giving less fps. I am on an i5-4670k and 1070ti, Arch Linux.

A note, for anyone building the fix (as I doubt another Steam release for 1.3 will happen): build and replace the server too if you're hosting. Replacing just the client showed zero difference with just me online (and, as my effects seem to be more extreme than others', may have been the main issue in the first place).

[Mirsario]

I can confirm that the revert fixes the issue. It was alright aside from boss fights (with six players) until I had more particles coming off my player while running, at which point it became unplayable. Running in one direction for ten seconds would either slow the game to a snail's pace, or with frame skip on display various graphical glitches (half-drawn frames) and make cursor movement jittery while giving less fps. I am on an i5-4670k and 1070ti, Arch Linux.

A note, for anyone building the fix (as I doubt another Steam release for 1.3 will happen): build and replace the server too if you're hosting. Replacing just the client showed zero difference with just me online (and, as my effects seem to be more extreme than others', may have been the main issue in the first place).

If the fix is making a big difference, jopo will likely do another release soon-ish.