Map Server keeps on crashing after a few hours

[judegimeno]

• **rAthena Hash **: b04f79e

• Client Date: : 2017-06-14bRagexeRE

• Server Mode: Renewal

• Description of Issue:

  • Result: Rejected from the server
  • Expected Result:
  • How to Reproduce: Restart the server or start the server and after a few hours (6 hours or more) the map server will crashed automatically but ftp,website,ssh is working. login and char server is working as well.
  • Official Information:

• Modifications that may affect results: I only added npc on the npc custom folder and even i restart the server with no npc its still the same. and change the welcome message as well. and turn off the git version welcome message.

[admkakaroto]

My server also hangs up randomly, it did not have that problem in the old days.

[judegimeno]

@admkakaroto yes huhu 👎 i hope this will be fixed soon!

[Akkarinage]

I'm not sure how you expect the devs to fix something if you don't actually tell them what's wrong.

If your server is crashing, we need the error message/core dump. The devs are not a team of crystal balls.

[admkakaroto]

I'll generate the error and post here.

[Jeybla]

Maybe related: #2355

Did you execute a @reloadmobdb before the server crashed?

[judegimeno]

@Akkarinage ill get the error code and pst it here right away

[Mikegyver]

@Jeybla nope.. it is happen that i notice it will consume ram bit & bit until to the max.
and then it happen the map server shut down automatically and it happen that..
if u are on putty session.. and u try to ./athena-start stop. it will tell u cannot allocate memory.
at starting. my RAM usage was just 647MB out of 1GB.. suddenly after certain duration of time. the ram usage spike up to 900MB.. and when it hang.. all services are down. like ssh, apache, phpmyadmin.

[Valorro]

unsure ,but this ever happened on my server. it caused by set npc variable on your custome script, maybe related to #1205 and https://rathena.org/board/topic/93054-memory-usage-by-script-vars/?tab=comments#comment-246980 hmm, overloop?
because if you try ./athena-start restart while it happen, it fixed, no more full usage of memory
im avoiding to use set event npc variable:
set .npcvar,0;
set .npcvar,1;
set .npcvar,0;
and use global variable instead, following https://rathena.org/board/topic/108905-variable-for-each-npc-random/?tab=comments#comment-315237 ,then no more crash. idk if this correct way.

[Jeybla]

So @Valorro and @Mikegyver confirmed that the server runs out of memory and crashes. We should quickly find the cause of it, since it's a critical bug.

@Valorro I don't think that has anything to do with the NPC scope. I still would recommend to prefer NPC variables rather than global vars, because you can quickly run into side effects. Also #1205 is already fixed since dce6ff1 (the additional allocation got removed, so there shouldn't be a new memory leak because of that commit).

[sader1992]

i use the same client and i run my Linux pc all the time and the i don't stop the server atall
(my Linux pc is 4gig ram and on home network gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4)) and i use the pc for more things then rathena almost all the time while the server runing
Git Hash 6748ff7
Packet Obfuscation Disabled
renewal
i am not loged in the server all the time it's like 3 hr in the day when i make edit test scripts but the server it self running all the time almost

[Mikegyver]

i guess u git has not affected by this map-server crash problem. seems to be this map-crash problem happen on more newer commit from urs.

like @admkakaroto said. My server also hangs up randomly, it did not have that problem in the old days.

[admkakaroto]

I have an emulator of the month 02 (February) and it does not have this problem, I have been updating it manually, but without the package overhul, it is running fine but totally out of date, but it does not have this crash problem.

[judegimeno]

Please check photo

[Valorro]

@Jeybla hmm i see, im really sorry cant reproduce the core problem, this issue happen like 1-2 month ago and everything fine now no more crash after i did changes on custome npc script var, maybe it has no correlation at all. need futher test

[sader1992]

@judegimeno is this image the error you get when the map server crash ?

[Mikegyver]

@sader1992 nope.. that is the image when he execute run in gdb map-server.

after run in gdb map-server, he execute bt full

and this is what happen when the server crash... look in the red box.

[Lemongrass3110]

@Mikegyver yours was killed with SIGKILL which cannot be handled/implemented by any application. Could it be that your operation system has some auto process killing activated when something happens?

[Mikegyver]

@Lemongrass3110 i'm not sure. but i'm using same setup on ubuntu 14.04.5 & 16.04.3. this only happen on ubuntu 16.04.3 as i mentioned to u on discord earlier... maybe the OS itself shut down the process as ram usage almost full

(Could it be that your operation system has some auto process killing activated when something happens?

i'm running again gdb map-server. i'll write another finding.. either get same result on above screenshot or not.

[Mikegyver]

on 2nd testing.. still get the same result.

[Lemongrass3110]

Looking at your screenshot it looks like you intentionally killed the process this time. This time you got SIGINT which is issued by CTRL+C. Also looks like this because you have ^C in your console.
So I'll go with the assumption that you also used kill on the process on the other screenshot and ended up with sending a SIGKILL event to the process.
Just run the server with gdb and wait until it crashes by itself. There is no other option to force it to output the correct line where the error occurs.

[Lemongrass3110]

@judegimeno
Please try it on a clean rAthena. I can tell that you are not using a clean one, because there is a script parsing issue that we can see on your screenshot.

[Mikegyver]

@Lemongrass3110 isn't CTRL + C is to stop.. then i execute c (which continue back)?

http://web.eecs.umich.edu/~sugih/pointers/gdbQS.html

To stop your program while it is running, type "(ctrl) + c" (hold down the ctrl key and press c). gdb will stop your program at whatever line it has just executed.

If you want gdb to resume normal execution, type "continue" or "c". gdb will run until your program ends, your program crashes, or gdb encounters a breakpoint.

[Everade]

On git 6f98508
Everything looks normal to me.
Running since several days.

However running rAthena + SQL database on the same system only having 1gb of ram available in total sounds like you're saving money on the wrong spot.

[admkakaroto]

My dedicated server has 32GB of RAM and even then the server randomly crashes.

[Everade]

Check the updates which took place inbetween the working and crashing git and try to find out if it may be related to customizations such as scripts or core changes.

And post crashdumps so the devs can see what's going on.

You could also check if you can reproduce the crash on a clean rathena.

Mine has a lot of customs scripts and core changes yet i don't had this issue as of yet.

[Mikegyver]

this is report on clean rAthena running on ubuntu server 16.04.3 64-bit.

the only changes had been done are modifying the IP & mysql database credential.

from time to time we can see the VPS ram usage getting increase bit by bit.

and these following images done un-comment the ShowMessage in the src/common/malloc.c as requested by @Lemongrass3110 on our discord conversation. the purpose want to see once it runs we should see where it happens. but somehow nothing is shown eventho have un-comment this ShowMessage.

anyway, thanks to @Lemongrass3110 coz willing to spend his time on this issue eventho he is at work. such a very concern dev. xD

EDIT: @Everade this is on clean rA.

[Jeybla]

from time to time we can see the VPS ram usage getting increase bit by bit.
From your screens the login-, char- and map-server are all pretty stable in memory allocation. Doesn't look like the issue already emerged there.

[Mikegyver]

@Jeybla. Yup. Correct. But once the map-server is killed.. the ram-usage fall down & not increase bit by bit. It just keep increasing when map-server is running. That is the most weird thing.

Or even when the vps is idle. Without starting athena. The ram consumption pretty stable & static.. just when the server is started. The ram usage getting increase time to time.

[Mikegyver]

as i mention on above. the MySQL ram consumption is pretty stable & static (which means doesn't keep increasing ram usage) & below are the screenshots. taking interval of 1 hour gap time.. we can see from the screenshots below, on idle, the MySQL ram consumption just increase 1MB.

as i mentioned earlier, ram consumption keep increasing only when rA is running (in this case, MySQL ram consumption). it is more likely the record/statement that is not being freed. maybe that's the reason why the ram consumption keep increasing until it causing the OS ended up with sending a SIGKILL event to the rA map-server process.

Linux OS - Ubuntu Server 16.04.3 64-bit
MySQL - MySQL version 5.7.19

[sader1992]

server hash 1fc0ed6
nothing happen
the ram dose not increase and there is no map crash and there is no problem
renewal and 2017-06-14bRagexeRE

yesterday

and today after more then 12 hr

and if we count the ones that not 2017-06-14bRagexeRE
i have server on the hash 374ecd0 2015-11-04aRagexe
and it's running for more then a week now with no map crash

[Mikegyver]

@sader1992 i guess u not read carefully.. i didn't mention somewhere on above map-server is eating the RAM. in fact this has been cleared by @Jeybla

From your screens the login-, char- and map-server are all pretty stable in memory allocation. Doesn't look like the issue already emerged there.

as in my above screenshot. the MySQL is eating the RAM. and. it happen when rA is running. i already attached 2 screenshot on above. when vps is idle. the MySQL ram consumption is at steady state. only when rA is running, MySQL getting consume a lot of RAM consumption. and i also did myself here

it is more likely the record/statement that is not being freed. maybe that's the reason why the ram consumption keep increasing until it causing the OS ended up with sending a SIGKILL event to the rA map-server process.

and also i have state my linux distro & what MySQL version i'm using. this is on VPS running on ubuntu server. while your running on Linux Desktop version. since u don't have issue on this, i politely would like to ask u don't interfere.. what i posted here, just want to update the issue thread. i already address this to @Jeybla & @Lemongrass3110 via PM at discord.

[sader1992]

@Mikegyver you want to fix the problem or just want everyone to say there is a problem ??
as i see there is no error in rathena as you say and as you refer (your is ubuntu server and my is desktop) = it's from the operating system not rathena ? [ask u don't interfere] we all here to help fix issues :P how about you don't reply on me every time i post something for no reason atall and focus on the issue

for the operating system i would ask you to reinstall ubuntu (server) with all the tools rathena needs and re try again :P that if you want to fix the problem and post everything about your operating system if you got the problem again to help fix it (it's not a private post)

my system info:
mysql Ver 14.14 Distrib 5.7.16
gcc version 5.4.0
Ubuntu 16.04.4 64bit

[Mikegyver]

@sader1992 i wouldn't post here if i have no problem after rebuild the vps.. but even after rebuild the vps the same problem ever persist.. in fact i have rebuild the vps for 8 times. and even using aws & digital ocean vps.. and yet the same problem persist.. well.. since u said no error.. why not u grab either aws or digital ocean vps & make a try... and let me know either u have that problem or not..

[Jeybla]

in fact i have rebuild the vps for 8 times
I didn't thought your server is crashing that often. Why can't you provide a crash dump then?
I still wasn't able to reproduce the crash on a similar setup.

[Mikegyver]

@Anacondaqq u might have the point... coz on vps.. it doesn't have swap partition (which act like virtual memory) unlike being installed on computer or VM... (which we been asking on allocation on size for swap partition).. most of the vps provider just making into real memory.

Try to allocate some swap, 2GB maybe will be enough.

1. sudo mkswap /swapfile 2. sudo swapon /swapfile 3. reboot

is there any chance by these commands, we can specific on partition size for the swap?

[aleos89]

@Mikegyver that's the first step in @Anacondaqq's list. The 2G signifies 2 gigabytes.

[Valorro]

maybe its MYSQL 5.7 features? default table open cache on 5.1 = 64, on 5.6 = 400, now 5.7 = 2000
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_table_open_cache
our server has around 100 table and its 2000 eats alot of RAM. according to https://dev.mysql.com/doc/refman/5.7/en/table-cache.html , 'Make sure that your operating system can handle the number of open file descriptors implied by the table_open_cache setting. If table_open_cache is set too high, MySQL may run out of file descriptors and refuse connections, fail to perform queries, and be very unreliable. ' em not sql expret, not sure about it.

[JuanMeyN]

We have this problem also in our server , does instance variable ' affect it?

[admkakaroto]

=/

[Jeybla]

@Mikegyver and @judegimeno were able to fix the issue by adding a swap space. Thanks to @Anacondaqq for mentioning it. 1 GB Ram is just not enough to run rathena (with mysql and flux).
@JuanMeyN @admkakaroto if you still encounter crashes with enough ram/swap please try to create a core dump or have a look at your kernel log. If you can reproduce the issue on a clean rathena, please open another issue with detailed informations about the issue and your system.
Closed