Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
C stack usage error #378
I have a shiny app under development that is designed to run continuously on an Amazon Lightsail instance with 1 core and 4G memory. It's at dev.open-meta.org/app.
The app has never crashed while I'm using it, but it has now crashed several times while I'm not using it, always with a C stack usage error. The most recent crash is quite interesting, here's the message:
How can I have a 1031 Gig C Stack on an instance with only 4 Gigs of memory?
Google searches suggest the C stack usage error is usually caused by out-of-control recursion. I don't have anything like that in my code.
Is this a Shiny-Server, R, or Linux error? Do you have any further ideas for debugging what's happening?
Hi Joe, Here is some info about our system. [root@discover shiny-server]# more /etc/redhat-release CentOS release 6.10 (Final) [root@discover shiny-server]# uname -r 2.6.32-696.23.1.el6.x86_64 [root@discover shiny-server]# The file /opt/shiny-server/VERSION shows 18.104.22.1684 [root@discover shiny-server]# R --version R version 3.5.0 (2018-04-23) -- "Joy in Playing" Copyright (C) 2018 The R Foundation for Statistical Computing Platform: x86_64-redhat-linux-gnu (64-bit)
I’m not able to comment on whether or not it happens with the sample apps. This application is being used by many people, and it just seems to be random as to when things die and have to be restarted. At the moment I can’t state that it is any one thing that causes the problem. Jeff From: Joe Cheng <email@example.com> Sent: Wednesday, September 05, 2018 10:14 AM To: rstudio/shiny-server <firstname.lastname@example.org> Cc: Saxe, Jeff (NIH/NCI) [C] <email@example.com>; Comment <firstname.lastname@example.org> Subject: Re: [rstudio/shiny-server] C stack usage error (#378) What versions of Shiny, httpuv, later, and shiny server? Does it happen even with the sample apps we ship in /srv/shiny-server-client/sample-apps? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#378 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/Ai2wcFBYqTrfMkh6f_ErMqZ4XSrfPokHks5uX9wKgaJpZM4WZGG6>.
Here are the specs of my Amazon Lightsail instance (note that Amazon Linux is also CentOS 6):
This happened again and I noticed the error message sometimes includes an additional error. Here are the messages from the last three crashes:
Log name: shiny-app-dir-shiny-20180906-191655-35967
Log name: shiny-app-dir-shiny-20180828-215352-35267
Log name: shiny-app-dir-shiny-20180815-011601-39097
In each case the usage limit is hundreds to thousands times more than the 4G allocated to my Amazon instance. This might be Amazon giving the instance an extremely generous burst of memory rather than a shiny-server issue, but it seems very strange.
Here's the log from the most recent crash, shiny-app-dir-shiny-20180915-164530-33121, with a last update of 9/17/2018 8:54:34. The log basically says that I (Admin) opened a session in one browser window on 9/15 and never closed the window, so the session continued. I opened a second browser window on 9/16 and closed it 19 seconds later, ending that session. The first session ends when the program crashes.
Since this is still under development and strictly doesn't need to be live, I've temporarily changed the app to Old Faithful to see if the crashes continue with a Shiny sample app that loads no packages except "shiny".
I was looking through older logs to see if I ever had a crash when there wasn't an open session and I didn't find any. It could be important that this application uses Shiny Server's app directory, not the site directory, and a session stays open until the user sends the browser window to a different URL (or closes the window). That is, a session can be open for days for a user who just leaves browser windows open and doesn't ever power-down.
On the other hand, I did find one crash that occurred soon after starting the app and it includes a longer error message. This log (last saved at :27:17) basically says the app started less than two minutes earlier at 21:25:35. During that time the log says I went from the site home page (session 1), to the site log in page (session 2), to my personal home page (session 3), to the start page for one of my projects (session 4), to some unreported page (session 5), where I clicked on a sub-menu button..
After many weeks of urging the guys here to test their own R code with valgrind they finally did and sure enough found problems with their code. Upon fixing that the problems related to C stack usage disappeared. I would recommend that you check your code using valgrind if you haven't done that already.
On that note: I've spent a lot of time tracking down memory bugs and I created this Docker image with various builds of R to help find them:
Here's the accompanying documentation:
If you have memory bugs in compiled code, this is a very useful tool for finding them.
We, too, are seeing this error in our Shiny Server Pro application log:
The error seems to be written to the log after a closed session is detected.
This is only occurring in an app that uses the rJava and xlsx packages. And it only occurs in Pro, not the free version of Shiny Server. We use these packages to dynamically generate formatted Excel workbooks. We cannot use the openxlsx or writexl packages because those packages do not provide the formatting options we need.
I'm attaching a pared down shiny app to demonstrate.
All this app does is load the jvm, rJava package and xlsx package in global.R. It does nothing else. If you run the app you will see nothing when the app loads. When you close the session, and it is detected, the C stack error is written to the log. Since there is really no code to speak of there is no way to prevent this error by optimizing or reconfiguring code.
This is our setup:
We had a crash of a real (not sample) application using rJava and xlsx yesterday but other than the C stack error there was no other indication of why the crash occurred in the application log or the shiny-server.log. A restart of Shiny Server Pro resolved whatever problem had occurred.
Does anybody have any insight:
Thank you for any insight or help you can provide.
Hey @maryskalicky ,
I have recently spent a decent amount of time trying to debug a similar problem in a much bigger app. Regarding your questions:
Ad 1. I guess it has something to do regarding scoping when spawning two parallel R processes - Shiny Server Pro does it and it seems to be the difference maker compared to the open version.
Ad 3. The solution could be moving loading of your libraries into the server file, before the declaration of the server function. It fixed the issue for me, please let me know whether it helped in your case as well.
I would also be glad to learn more from Shiny Server team as to the possible reasons behind such behavior. There seems to be no reason why global.R shouldn't load libraries.
Thanks so much for your suggestion. I just tried it, moved the library function calls into server.R before the server function but, rats, it did not resolve the issue for me. I was so hopeful!
I tried increasing the stack soft limit to unlimited. The hard limit was already at unlimited. When I did that the C stack usage error was gone but then I got a different error:
Error in execCallbacks(timeoutSecs) :
That error seems more nefarious so I reverted the stack limit and am now back to the C stack usage error.
Thankfully the C stack usage error doesn't crash the app as far as I know. The app was running fine for 5 weeks with this error appearing but only crashed just yesterday. I don't know what caused yesterday's crash but I do know we'd like to resolve the C stack usage error if possible.
Thank you again for the suggestion.
Mary - thanks so much for keeping us up to date on this. My own app uses a package called mailR, which in turn uses rJava to send email via Amazon SES. I'm going to try removing that package and connecting to Amazon SES using its api instead.
For me, this is happening on the free Shiny-Server download, though, not Pro, so we'll see.