Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Server down/stalled... #944
I'm unable to get into the VPS to restart it and it's spinning in a web browser. NOTE: This is purely a VPS issue with our provider and not the project nor the node configuration.
Last script update on local pro at 2016-04-05T12:07:05.214Z
pushed a commit
Apr 6, 2016
Found a break in the system... either a hiccup on whatever is causing this... or my distro update on laptop which may have a compatible client to connect to the latest Debian... or sizzle... although I did try Windows VM (virtual machine) and PM (physical machine), Debian VM, ArchLinux VM, ArchLinux PM, and other Linux PM too and those failed... so not entirely sure. (too many inet issues today everywhere)
I have already seen a 503 with toobusy-js on login that is probably GH's issue (still guessing here)... leaving this open for a more detailed investigation over the next few days. Apologies for this unscheduled outage... definitely out of my control at this time.
Btw dist-upgrade yielded no further updates. :\
I know... it's going to take a bit to resolve this... something is chewing up memory and causing the VPS to crash... this was before the 503 addition of toobusy-js... I'm probably going to take the server down, do some recompilations, and see if that helps. e.g. that is why it's PENDING status right now.
Patience please. :)
Downgrading node didn't help... seems that the malloc (or whichever lowlevel lib is being used) isn't freeing up memory in the distro/VPS.
I'm going to try disabling script minification just to be sure... with an environment variable to be added... don't worry I'll have it pass-through to the unminified so it doesn't break scripts.
Still losing memory, although slower, with disabling of script minification... e.g. the VPS is going to crash again... watching it right now go down, down, up, down, down, down, up, down, down, down, etc... until eventually there is zero free memory.
So systematically ruled out our project and reaffirmed this is a distro/VPS issue. :\
@sizzlemctwizzle and anyone watching,
I've tried many different versions of node all result in the same issue with this kernel image on the VPS.. e.g. memory gets eaten up. Using the precompiled _node_s, the server lasts for less than 5 minutes... with a manual build from _node_s source I can sometimes get about 45 minutes of uptime. NEITHER OPTION IS SUITABLE as I can't babysit the server that constantly.
I've also looked into backing out/rolling back the last dist-upgrade and of course the old packages aren't available on the official repos... so that will fail.
Only three options are left that I can think of...
Just a sidenote... all script sources are intact as far as I can see in local pro. e.g. this is not a DB issue. (also made the HOST label here on GH as you might have noticed already)