Obscure bug: SIGPIPE and stack smashing when cancelling file download on Flask app that uses rpy2

[Melnorme1984]

The title says it all. I have a Flask-based webapp that uses rpy2 to do some computations. The app's front end also has an unrelated download link, which I serve using Flask's send_from_directory method. If I start downloading the file and then cancel the download midway, I get this on Gunicorn:

R[write to console]: Error: ignoring SIGPIPE signal

*** stack smashing detected ***: terminated
[2021-05-19 15:03:45 +0300] [50124] [WARNING] Worker with pid 50354 was terminated due to signal 6

And this on Flask's development server:

R[write to console]: Error: ignoring SIGPIPE signal

R[write to console]: Fatal error: unable to initialize the JIT


*** stack smashing detected ***: terminated
Aborted

This doesn't happen if I allow the download to complete. I'm using the latest version of rpy2. Very strange!

[krassowski]

gunicorn uses signals to communicate between its workers: https://docs.gunicorn.org/en/stable/signals.html as means of internal communication. It appears that those get propagated to R thread (but they should not). If you can provide a minimal reproducible example that could help to fix that (which might not be easy or possible), but otherwise I would suggest to restructure you app so that R does not run in the same process as gunicorn (maybe a subprocess?) or to use a different server like Apache with mod_wsgi.

[Melnorme1984]

It also happens with Flask's development server, not just with Gunicorn. But I think only on Linux, not on Windows.

[RonanDaly]

This happens on Mac as well, with django's built-in server

#0  0x00007fff6a2a56b5 in __abort () from /usr/lib/system/libsystem_c.dylib
#1  0x00007fff6a2a5ff3 in __stack_chk_fail () from /usr/lib/system/libsystem_c.dylib
#2  0x0000000129424f58 in setup_Rmainloop () at ../../../../R-4.0.2/src/main/main.c:1092
#3  0x0000000105ecda60 in ?? ()
#4  0x0000000105ecda60 in ?? ()
#5  0x00007fd5fc3f7890 in ?? ()
#6  0x000070000424a350 in ?? ()
#7  0x0000000105ecda60 in ?? ()
#8  0x0000000105ecda60 in ?? ()
#9  0x0000000105ecda60 in ?? ()
#10 0x00000082060c8230 in ?? ()
#11 0x0000000105d7a140 in ?? () at ./Modules/posixmodule.c:1172
#12 0x0000000000000000 in ?? ()

[lgautier]

Could this be a variant of issue #769 ?

[RonanDaly]

I think what is happening is that a SIGPIPE is being generated on the python side, by the client disconnecting. However, the R thread just happens to be running at that point and the kernel delivers the SIGPIPE to it, which of course it is not ready to handle. As mentioned by @krassowski and shown here https://stackoverflow.com/a/49013048, R being in a separate process should stop this happening.

[lgautier]

By default R will not run in a thread. If Flask starts threads with connections a few things could go wrong since R cannot handle multithreading (see https://rpy2.github.io/doc/v3.4.x/html/rinterface.html#multithreading).

[RonanDaly]

I was running Django 'single-threaded', but in reality there were three threads for the process. The following snippet is single threaded. If you run it and send a SIGPIPE to the process it will exit in not exactly the same way as above.

from rpy2 import robjects as ro

while True:
    pass

$kill -PIPE <pid>

R[write to console]: Error: ignoring SIGPIPE signal

Abort trap: 6

I guess R installs a handler that gets the SIGPIPE - I don't know how this would be solved, as R and python both need to handle SIGPIPEs separately.

[lgautier]

It seems that is indeed trying to handle SIGPIPE, and this the cause of issue (e.g., rstudio/httpuv#168).

I hope I can at least prevent the stack smashing from happening though.

[lgautier]

If I understand the R doc correctly (https://cran.r-project.org/doc/manuals/r-patched/R-exts.html#Threading-issues) it should be possible to silence R's handling of SIGPIPE.

This is what line 165 in rinterface_lib/embedded.py is meant to achieve with

rlib.R_SignalHandlers = 0

So may be the issue is that instead of silencing R's handling or SIGPIPE there should be a custom handler to defers to the Python default handler.

[lgautier]

Ha! R_SignalHanders is set after R is initialized and the main loop is started. When checking src/main/main.c in the R source I see that the status of R_SignalHandlers is checked when the main loop is started.

Can someone check if revision 9de03e4 solves the issue?

[RonanDaly]

This seems to solve my issues - thanks for looking into it.

[lgautier]

The fix is in the main dev branch and in release 3.4.5 (now on pypi). Closing.