Crashing after QL update #40
Comments
|
That's not Minqlx crashing, that's the Quake Live Dedicated Server process crashing, it's unlikely Minqlx has anything to do with this one. Minqlx doesn't print a backtrace, and from what I can see: (02:51:16) [DEBUG @ minqlx.handle_console_print] qagame: 0x7FF8CCB980B0 That's been generated by the QLDS process, not Minqlx. |
|
It could still be caused by minqlx, but I can't really tell since the offsets in the backtrace don't tell me which module the stack was into. I haven't had any issues myself and I don't think others have either, so if it's due to the update, it would have to be really rare/caused by uncommon settings. What plugins are you running? If you're running third-party plugins, you could try to disable them one by one and see if it stops. Plugins that don't use threads correctly could very easily cause random crashes like these without there being an obvious cause. Oh, and since you say these happen quite often, you could save EDIT: Huh, I had my server crash on me today. It's the first one since the update, so it's not nearly as often as you, but still. I did see someone complaining about crashes on servers without minqlx too, though. Could be some bug introduced with the update rather than minqlx. I'll keep an eye out. |
|
I am using only included plugins, except branding: plugin_manager, essentials, motd, permission, ban, silence, clan, names, log, workshop, balance, branding It has happened on two different hosts/vps, but same config. Ok, gotcha on your server crashing too :( Ok, I will gather more information. The only significant thing I can find so far, it randomly happens on a map change. I do not see crashes during a match. The server attempts to load a new map, the map screen comes up, but then the server performs a shutdown. |
|
I get this randomly on my servers as well. Same as what Li-nt mentioned. It seems to happen randomly on map changes, Server will crash and restart (thnx to supervisor). Doesn't seems to be any pattern to it. Can happen when changing to any map from any map. I only noticed this starting to happen since the previous qlds update (1068). Never noticed it happening before that. Can be quite annoying when there is 16 ppl on the server playing and the server crashes at the end of a match on map change and it essentially kills off further play because most people don't bother to reconnect. This is an example of what I see in the stdout - qagame: 0x7F8E45970180 ==== ShutdownGame ====Sys_Error: signal caught: 'Segmentation fault', si_code 1 |
|
On Bus Station we also used to have frequent crashes, but we found out they were caused by 'bad' workshop items. I.e. some custom maps would cause the server to crash when they were selected for the vote-next-map-out-of-3-options thing. |
|
I think what @dsverdlo says has merit to it, some workshop items have indeed crashed my servers and client in some cases. |
|
This is with any map, including maps like campgrounds, overkill, hiddenfortress and all the rest of the popular maps. The server can crash when changing to any map. Seems to happen more often on the duel servers but that is probably because map changes are more frequent on the duel servers. And yes, I am aware some workshop maps can cause problems. I have had to remove some workshop maps due to them causing all client to get disconnected at the end of a match when playing them but they did not cause the server to crash. |
|
Then it's very likely it's an internal Quake Live problem and nothing to do with Minqlx, I run a few vanilla servers and they suffer from the issue as well. |
|
Cage is correct - it happens on non-workshop maps (QL maps). I eliminated this in the beginning, as I removed all custom maps and ran the server for a week. It continued to crash. From what I've noticed, the crash occurs more commonly once the process has been running for a while, and only on map changes. I was going to remove minqlx for testing, but I like it too much to part with it, even for a day. However, it sounds like Thomas is seeing crashes on non-minqlx servers. If so, then it would be attributed to the April QL update, not minqlx. I have my servers setup to alert me when they crash or restart. Prior to the April update, I did not experience these crashes. Sadly, I doubt this will be addressed anytime soon by sponge, unless we all report the same issue to him. |
|
I've heard sponge has been 'active' in the QL discord channel, but I reckon he will be quite preoccupied with QuakeCon 2016 at the moment. |
|
I would just like to point out that I wasn't saying it was definitely something to do with Minqlx because I honestly have no idea. I just thought it would be a good idea to bring it up here in case it did have something to do with a change in a recent qlds update that broke something with minqlx. From the feedback though it seems that this issue is still present for servers that don't even run minqlx so it sounds like a qlds issue and nothing to do with minqlx. Thankfully its not a super major issue. At least it doesn't do it mid-game. Just annoying though for players to get dropped randomly at the end of a match during map changes, Hopefully Sponge or whoever looks after the QL development/bug fixes will get a chance to look into it at some point. I would also like to give my appreciation to MinoMino for his time and effort in developing minqlx in the first place and keeping it up to date with each qlds version. Without it there would be no way to add functionality to QL at all. Thanks MinoMino, much appreciated. |
|
Issue still ongoing from 2016 to present, we have never been able to determine a resolution to this. However, we have had a lot of time to diagnose, research and troubleshoot the problem. We have found that a large number of players in the server will cause a crash, which is likely due to the server-side buffer (MAX_RELIABLE_COMMANDS 64) in the quake code (and in quake_common.h in minqlx). Other games that are based on the same engine have reported that they compiled new server-side executables that increase the client buffer from 64 to 128 on the server-side (CL_MAX_RELIABLE_COMMANDS 128), which resolved the issue. Honestly, we have come to the realization that Bethesda is not going to address the issue, but I think there may be a way to recompile minqlx to patch this issue. If anyone out there reads this and has ideas or wants to help, you can find me on discord at http://discord.pillclan.com. Until then, we continue to reload our server after crashes and/or have players reconnect. We love Quake Live, and will play regardless. (21:32:41) [DEBUG @ minqlx.handle_console_print] qagame: 0x7F0AF14F9180 |
Asked SyncError in official quake discord channel to increase constant |
|
@em92 Thank you. There are apparently two buffers, one server side (MAX_RELIABLE_COMMANDS) and one client side (CL_MAX_RELIABLE_COMMANDS). It seems that the server side is the problem, which can be increased to resolve issues. Adam fixed many bugs in the past for us, but I think this is an inherited bug from the old engine. |
Don't know anything about it. https://github.com/id-Software/Quake-III-Arena/search?q=CL_MAX_RELIABLE_COMMANDS&unscoped_q=CL_MAX_RELIABLE_COMMANDS |
|
@em92 Thank you, then you are right and it is just the server side MAX_RELIABLE_COMMANDS that would need to be increased. CL_MAX_RELIABLE_COMMANDS is for other games only. |
|
I put your buffer CL_MAX_RELIABLE_COMMANDS with large value in test run on starfish for couple of weeks. Will see if it does anything. But i still belive for you need to adjust: ./qzeroded.x64 +set com_hunkmegs 256 +set com_zoneMegs 512 The problem is, i have no real means to check it, as server i run restarts every night, just to get rid of all those issues. Although, before i inserted those com_ values qlds would stuck even twice a day. Now is better, but still, sometimes, ... So i will test it, and share results (if i will not forget:) ). |
How are going to increase |
|
I am not a spec I thought taking a reference library quake_common.h, would be good point to start, simply by changing CL_MAX_RELIABLE_COMMANDS and compiling it, yet it went perfectly good. So i added it to server, and started it, and it also went without any problem. I connected there, had a look, it was running, all good. GTG. But i did not do any log check, now i am checking, and see, it does not work properly. I get this: Idk what is it, a buffer overflow? It seems mino reads from other part of memory now, how is it called properly? a "sentinel value" or something similar, isn't it? And real value is somewhere else inbetween. I know, i need to learn basics. But minoqlx worked, just partially ;). Do you have any other idea? |
No. You modify structures in minqlx, that MUST match structures in compiled qlds. So you need to modify MAX_RELIABLE_COMMANDS, recompile qlds, modify this value in minqlx, recompile minqlx. But the problem is that you don't have sources of qlds. If you change MAX_RELIABLE_COMMANDS in only minqlx, you will get structure mismatch, which makes minqlx unusable. |
|
@li-nt does your server has private slots? |
I see about 2 to 3 of these crashes per day since the April QL update. QL service does a shudown and restarts.
(02:51:09) [DEBUG @ minqlx.handle_console_print] ----------------------
(02:51:09) [DEBUG @ minqlx.handle_console_print] 23613 files in pk3 files
(02:51:09) [DEBUG @ minqlx.handle_console_print] Loading dll file qagame
(02:51:09) [DEBUG @ minqlx.handle_console_print] Sys_LoadDll(qagame) found dllEntry at 0x7ff8ccb980b0
(02:51:09) [DEBUG @ minqlx.handle_console_print] Sys_LoadDll(qagame) succeeded!
(02:51:09) [DEBUG @ minqlx.handle_console_print] Sys_SetModuleOffset: qagame 0x7FF8CCB980B0
(02:51:09) [DEBUG @ minqlx.handle_console_print] ------- Game Initialization -------
(02:51:09) [DEBUG @ minqlx.handle_console_print] gamename: baseqz
(02:51:09) [DEBUG @ minqlx.handle_console_print] gamedate: Apr 11 2016
(02:51:09) [DEBUG @ minqlx.handle_console_print] initializing access list...
(02:51:09) [DEBUG @ minqlx.handle_console_print] loaded 16 steam ids into the access list
(02:51:09) [DEBUG @ minqlx.handle_console_print] Not logging to disk.
(02:51:09) [DEBUG @ minqlx.handle_console_print] 0 teams with 0 entities
(02:51:09) [DEBUG @ minqlx.handle_console_print] 24 items registered
(02:51:09) [DEBUG @ minqlx.handle_console_print] -----------------------------------
(02:51:09) [DEBUG @ minqlx.handle_console_print] -----------------------------------
(02:51:09) [DEBUG @ minqlx.dispatch] map('cliffside', 'freeze')
(02:51:09) [DEBUG @ minqlx.dispatch] new_game()
(02:51:16) [DEBUG @ minqlx.handle_console_print] qagame: 0x7FF8CCB980B0
cgame: 0x0
ui: 0x0
[0x452936]
[0x7FF8D77CE340]
[0x7FF8D6A1CA03]
[0x7FF8D6ADA235]
[0x7FF8CCB990AC]
[0x7FF8CCB7B0D6]
[0x7FF8CCB86755]
[0x43ACC0]
[0x4250CC]
[0x42522D]
[0x450E39]
[0x4078EE]
[0x7FF8D69F2EC5]
0x4079F9 [DEBUG @ minqlx.handle_console_print] version: 1068 linux-x64 Mar 4 2016 19:27:22
(02:51:16) [DEBUG @ minqlx.handle_console_print] g_version: 1068 linux-x64 Apr 11 2016 15:28:26
(02:51:16) [DEBUG @ minqlx.handle_console_print] ----- Server Shutdown -----
(02:51:16) [DEBUG @ minqlx.handle_console_print] ==== ShutdownGame ====
(02:51:16) [DEBUG @ minqlx.handle_console_print] ---------------------------
(02:51:29) [INFO @ minqlx._configure_logger] ============================= minqlx run @ 2016-05-05 02:51:29.353146 =============================
(02:51:29) [INFO @ minqlx.late_init] Loading preset plugins...
The text was updated successfully, but these errors were encountered: