lgsm crashing servers? or just server bug?

[AzaZPPL]

Since of a month my server's have been crashing a lot. This happend after I ran update-functions. The error is the following:

Jan 11 02:53:11 DS19 kernel: srcds_linux[8846]: segfault at 0 ip 00000000f612cf02 sp 00000000ffcc1190 error 6 in engine.so[f5ea7000+51f000]
Jan 11 02:53:26 DS19 abrt-server: Executable '/home/kzc1/serverfiles/srcds_linux' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jan 11 02:53:26 DS19 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2016-01-11-02:53:11-8846' exited with 1
Jan 11 02:53:26 DS19 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2016-01-11-02:53:11-8846'

The same error on another server

Jan 10 21:18:09 DS19 kernel: srcds_linux[10271]: segfault at 0 ip 00000000f60fbf02 sp 00000000fff45ec0 error 6 in engine.so[f5e76000+51f000]
Jan 10 21:18:22 DS19 abrt-server: Executable '/home/kzc5/serverfiles/srcds_linux' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jan 10 21:18:22 DS19 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2016-01-10-21:18:09-10271' exited with 1
Jan 10 21:18:22 DS19 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2016-01-10-21:18:09-10271'

After googling for 10 seconds. I found this https://steamcommunity.com/app/222880/discussions/2/35220315872063271/

Any ideas?

[jaredballou]

What OS?
What game?
Any addons (SourceMod, MetaMod, Garry's Mods, etc)?
Can we get the logs 100 lines before the crash?

[AzaZPPL]

OS: Centos 7 x64
Game: CS:GO
Mods: Sourcemod, Metamod
The error above is the only error. I've witnessed it happen while I was in the console. No errors popup on the console. The console just hangs and you are unable to do anything.
lgsm logs:

Jan 11 15:25:05 csgo-server: Monitor: Querying port: 109.236.90.70:27015 : QUERYING
Jan 11 15:25:11 csgo-server: Monitor: Querying port: 109.236.90.70:27015 : ERROR: Unable to receive
Jan 11 15:25:12 csgo-server: Monitor: Waiting 30 seconds to re-query
Jan 11 15:25:42 csgo-server: Monitor: Detected gsquery.py
Jan 11 15:25:43 csgo-server: Monitor: Querying port: 109.236.90.70:27015 : QUERYING
Jan 11 15:25:49 csgo-server: Monitor: Querying port: 109.236.90.70:27015 : ERROR: Unable to receive

[UltimateByte]

Hi,

Definately a non script issue.
Could it be a CSGO update that breaks your sourcemod/metamod ?
Don't they need an update ?
Did you try validating your gamefiles ?

[AzaZPPL]

We use the same mods every other community uses. The only difference is we are using lgsm.
All of the server have been updated and validated many times, with validation completing without any mismatches.

I will try and make another server without using lgsm and see if the problem persists.

[UltimateByte]

Seems like there are more issues with CentOS than Debian or Ubuntu.
The only thing that LGSM could do is applying a fix that is no longer required on CentOS and then cause the issue.
Could you please make a new user and a blank server (with different ports) and see if it works ?
Otherwise if you can't figure out, i'd recommand making new servers (with lgsm) and transfer the config to it. But that would be the last option. We're gonna try to figure this out before.

[AzaZPPL]

Should I install the new server with the same mods?

[UltimateByte]

First try it without mods, then with them. That way you have more chance to isolate then find the cause. :)

[jaredballou]

What do the performance metrics look like? Are you running out of memory? Is the usage of the server scaling up over time till it dies? If you can run "sm plugins list" for us as well that might shed some light. SourceMod is lousy with awful developers who create monsters of allocation without freeing memory. I count myself as principal among the offenders. I've had a lot of issues when I start getting memory contention on my machines running SM, and generally when I disable it my issues (and gameplay mods) disappear.

[AzaZPPL]

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 9956 kzc70     20   0  996184 753936  18976 S  24.2  4.7 379:26.79 srcds_linux
11037 kzc7      20   0  378100 223232   5396 S   3.4  1.4 220:25.58 srcds_linux
 2052 kzc3      20   0  341260 171460  18424 S   2.0  1.1   0:05.75 srcds_linux
28271 kzc1      20   0  597868 403248  19104 S   2.0  2.5  53:11.08 srcds_linux
15472 kzc6      20   0  767928 539276   8408 S   0.7  3.4 214:42.51 srcds_linux
27686 kzc5      20   0  812496 505204  19100 S   0.7  3.1  61:02.26 srcds_linux

[kzc70@DS19 ~]$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15676        3188         293         446       12195       11945
Swap:          7999         518        7481

When the server's console hangs up, %CPU is at 100%. If I'm not running monitor every 1 minute the server won't restart and stay like that until I manually kill it. When the CPU is at 100% the memory stays the same.

[SM] Listing 36 plugins:
  01 "Fun Votes" (1.7.3-dev+5280) by AlliedModders LLC
  02 "Web Shortcuts CS:GO version" (2.3) by Franc1sco franug and James "sslice" Gray
  03 "SourceSleuth" ((SB++) 1.5.4-dev) by ecca, Sarabveer(VEER™)
  04 "Rock The Vote Extended" (1.10.0) by Powerlord and AlliedModders LLC
  05 "SM ARMCHOOSER" (1.1) by Andi67
  06 "[Any] Chat Logging Redux + RTF Support" (1.1.1) by McFlurry, Keith Warren (Drixevel)
  07 "Basic Chat" (1.7.3-dev+5280) by AlliedModders LLC
  08 "Basic Votes" (1.7.3-dev+5280) by AlliedModders LLC
  09 "Admin Help" (1.7.3-dev+5280) by AlliedModders LLC
  10 "radio" (0.01) by AzaZPPL
  11 "Mirror" (1.0) by Franc1sco franug and Nanochip
  12 "KZErrorBoxFixer" (1.1) by 1NutWunDeR
  13 "SourceBans++" (1.5.4-dev) by SourceBans Development Team, Sarabveer(VEER™)
  14 "SourceComms" ((SB++) 1.5.4-dev) by Alex, Sarabveer(VEER™)
  15 "SM SKINCHOOSER" (2.7) by Andi67
  16 "Reserved Slots" (1.7.3-dev+5280) by AlliedModders LLC
  17 "Admin Menu" (1.7.3-dev+5280) by AlliedModders LLC
  18 "Fun Commands" (1.7.3-dev+5280) by AlliedModders LLC
  19 "Map Nominations Extended" (1.10.0) by Powerlord and AlliedModders LLC
  20 "Player Commands" (1.7.3-dev+5280) by AlliedModders LLC
  21 "Anti-Flood" (1.7.3-dev+5280) by AlliedModders LLC
  22 "[ANY] Player Analytics" (1.3.1) by Dr. McKay
  23 "MapChooser Extended" (1.10.2) by Powerlord, Zuko, and AlliedModders LLC
  24 "Basic Info Triggers" (1.7.3-dev+5280) by AlliedModders LLC
  25 "SourceBans++: Admin Config Loader" (1.5.4-dev) by AlliedModders LLC, Sarabveer(VEER™)
  26 "SourceBans Checker" ((SB++) 1.5.4-dev) by psychonic, Ca$h Munny, Sarabveer(VEER™)
  27 "Nextmap" (1.7.3-dev+5280) by AlliedModders LLC
  28 "Basic Comm Control" (1.7.3-dev+5280) by AlliedModders LLC
  29 "Knife Upgrade" (2.5.5) by Klexen edit by: AzaZPPL
  30 "Basic Commands" (1.7.3-dev+5280) by AlliedModders LLC
  31 "Thirdperson" (1.6 CS:GO edition) by shavit and Franc1sco franug
  32 "Sound Commands" (1.7.3-dev+5280) by AlliedModders LLC
  33 "Client Preferences" (1.7.3-dev+5280) by AlliedModders LLC
  34 "Admin File Reader" (1.7.3-dev+5280) by AlliedModders LLC
  35 "SM CS:GO Weapon Paints" (2.8.2 public version) by Franc1sco franug
  36 "KZTimer" (1.81 Global) by 1NutWunDeR

The server's main mod is KZTimer all else secondary.

[jaredballou]

When it gets jammed up, does it still respond to clients? What sort of stuff do you see in the logs/console? How many instances are running on this box? Any changes (upgrades, new plugins) lately?

[AzaZPPL]

Jammed up issues where the cpu was at 100% has been fixed by updating the server (There was a bug with srcds_linux with very heavy mods which caused this). However the server still crashes! Since CentOS is on tmux 1.8 I can't see console logs. After taking a look into the console at just the right moment I saw the crash. But it wasn't a crash it just executed the following commands:

quit 
restart

Why would it do that? Than I remembered I have enabled monitor in crontab. Disabled it from crontab. 2/3 hours in and no crashes.

[UltimateByte]

Recently in gmod, the server crashed when you runned the command "quit" into the console if the workshop was enabled on the server.
They corrected it in the last update.
Could it be possible that this issue occurs with csgo too ?

Do you think your "monitor" command makes an issue on your server ?
What happens if you run it manually ?
./csgoserver monitor