stutter issues in linux

[senecaso]

Your system information

• System information from steam in a gist: System Information
• Have you checked for system updates?: Yes

Please describe your issue in as much detail as possible:

I frequently get "stutter" problems when playing CS on my machine. This manifests itself in my FPS (as reported by cl_showfps 1) dropping suddenly and sharply. I will be happily playing with 175 FPS, and suddenly it will drop to 15-30 FPS and stay there for 5-30s. This occurs about once every 15mins or so. I have googled around and seen that a large number of other users are experiencing the same issues, but I have not seen a solid conclusion to address the issue yet. After doing some tracing through the issue, I think I may have found a clue, but I will need help from the community to trace the issue down further.

What I have found is that when my FPS drops is when a software interrupt is being processed by the system. This results in a large spike in the amount of CPU being spent in the kernel. I can't say for sure this is the cause, but there is certainly a correlation.

Here is some output I gathered from dstat. In the output, you will notice that a software interrupt occurs at 17:33:03 (look for the siq column) and see that the SYS CPU spikes at the same time. At the same time, I also grabbed output from nvidia-smi. You can see that at approximately the same time, the GPU and memory utilization on the card hits the floor. I'm not sure how this relates, but you will also notice that just before the stutter got really bad in the game, you can see that the graphics Mhz dropped from 131 to 75 MHz and stays there until the FPS recovers around 17:33:26. I have been able to consistently reproduce this behaviour.

Unfortunately this is where my experience falls off. I'm not sure how I can gather information about what is causing the software interrupt that is causing this issue. If dtrace were available, I would use that as I suspect the issue at hand is exactly as described in that article. Does anyone out there know how to track down the source of this interrupt? I'm hoping that if we can track it down that we might be able to come up with a workaround, or perhaps Valve would be able to address it directly in the code.

I should add that I am gathering the dstat and nvidia-smi output while playing CS in a 1440x960 window with the video settings maxed out. However, I have also reproduced the issue playing fullscreen and windowed with all videos settings set to low or disabling them entirely. Whatever the issue is, its not resolved by reducing the pressure on the video system.

Steps for reproducing this issue:

1. start an offline game with bots. I have reproduced the issue on Canals, Nuke, and Inferno
2. ensure cl_showfps 1 is on (although the issue happens with it off as well)
3. play until the FPS drops and the game-play gets choppy
4. check the output of dstat and nvidia-smi for the presence of software interrupts that coincide with the poor performance

[kiroma]

Have you tried setting NICE to -19?

[senecaso]

$> sudo renice -n -19 -p $(ps aux | grep csgo_linux | grep -v grep | awk '{print $2;}')
10677 (process ID) old priority 0, new priority -19

Tried, but I'm still getting the same FPS drops when a long-running software interrupt occurs. I also noticed that the FPS drops can sometimes occur when no siq is being reported by dstat, but whenever an siq is reported by dstat the FPS is without question affected.

[senecaso]

I was able to capture dstat, nvidia-smi, and strace while the issue was occurring. While the FPS is terrible with dstat showing intermittent software interrupts, you can see the values in nvidia-smi drop, as before, but you also see that normally very fast sys calls are starting to take a long time to process. I was capturing that strace the entire time I was playing, but it wasnt until the game started getting really choppy (5-15FPS range) before the captured entries appeared.

I'm not sure what to make of this, other than "all processing done by the csgo_linux binary is suddenly slowed down".

[senecaso]

I think I may have found what the issue is!

I personally never run any production system with Transparent Huge Pages enabled, but for some reason it hadn't occurred to me to do the same for my desktop. After disabling THP, I was able to play an entire map with only 1 instance where my FPS dropped to 30 FPS, and not a single occurrence of a software interrupt showing up in the dstat output. I dont have enough time to fully test this theory tonight, but I will try to play some more when I can and report back. It would be good if someone else could try to reproduce this behaviour by enabling THP though.

If this in fact the cause of the problem, it would be good if Valve could prepare some sort of documentation indicating that THP should be disabled for linux users and make it very visible. Since its enabled by default on many distros (most? all?), I suspect that this issue is in fact impacting a large number of users who are silently (hah! silent gamers..) suffering through it.

For anyone finding this issue and looking for how to disable THP, you can find a good article on it over on the MongoDB site, or just google linux thp disable. I'm sure you'll find it.

[senecaso]

Nope, spoke too soon. After playing for a little longer, the issue has popped up again and seems no better than it was before. So now I'm back to trying to explain why the SYS CPU is so high, and why all of a sudden sys calls start getting really expensive.

$> cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

[senecaso]

Ok, I've made a little more progress here, and I think its a hardware issue.

After checking out the thermal interrupts via:

while true; do cat /proc/interrupts | grep Thermal; sleep 1; done

and keeping an eye on the current CPU frequency via:

while true; do sudo cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq  | awk '{print $1 / 1000;}'; echo ""; sleep 1; done

I was able to see that due to thermal interrupts, my 4GHz CPU was dropping to 800MHz (on all cores) because of overheating. It looks like the stock heat sink I have on there is either not enough or has somehow failed. This would very easily explain the drops in performance, and would also explain the long interrupts being processed as the CPU was likely getting extremely hot and the OS was drastically throttling the frequency back.

My next step is to buy a new heatsink and see if things improve. I'll post back when I have news.

[kiroma]

I7-4790k doesn't ship with a cooler...
Intel's box cooler is too underpowered for this processor, you NEED to buy a tower cooler.
Did you just...
How long have you been running it like this?!

[senecaso]

2yrs :) Ya, good times. Its amazing I haven't fried the CPU yet. The good news is that the OS seems to be doing a really good job of throttling to keep the temp near 80C. So.. ya.. CS has been stuttering for me since pretty much day 1, and now I know why!

[mackatozis]

@senecaso Since you found the source of the problem, and it's not related to Valve, do you mind closing this issue? Thanks.

[senecaso]

You bet. I've just been waiting for the cooler to arrive so I could perform some final testing to make sure that was the only issue before I closed it.

The cooler has now arrived (CoolMaster Hyper 212 Evo), and my CPU temp is now < 50C even with turbo enabled. After playing for a few hours, I no longer see the CPU frequency being adjusted, and the FPS while playing CS is fairly constant in the 150-200FPS range. It appears that the cause of the issue I was having was in fact hardware related and now appears to be resolved.