R crashes when calling stopDaemonizedServer in later::later

[thomasp85]

Following is a minimal example (will crash your session):

library(httpuv)
library(later)
server <- startDaemonizedServer('127.0.0.1', 8080L, list(call = function(req) {list(status=200L, headers=list('Content-Type' = 'text/html'), body = '')}))

later::later(function() {stopDaemonizedServer(server)}, 1)

This is observed running R 3.2.2 on macOS 10.12.5

[paul-shannon]

Hi Thomas,

I cannot answer your question directly, but having spent a few years with httpuv on macos - some Bioconductor packages I maintain depend upon it - and having just now discovered that Windows httpuv supports the daemonized server if I get the configuration just right, I offer a minimal two-file demo in the attached files below. Together (app.R and index.html) exercise the connection, stops the server, then start a new one at the next port, iteration upon iteration without error. (Many thanks to @hcorrada and @jcheng5 for making these improvements.)

To run this demo, start R, then
source("app.R")
for(i in 1:10) demo(5500 + i)

Using R 3.4.1 and chrome, on Windows 7 and macos 10.12.3, there seems to be no limit to the number of iterations one can run, each a separate websocket on a distinct port. A variety of messages are passed, both ways, in each run of the demo.

asyncHttpuvDemo.zip

[thomasp85]

Thanks - my problem is specifically related to the interplay between the httpuv the later package. I have no problems running deamonized servers in itself, but gets a segfault when stopping them within a later() callback

[hcorrada]

I haven't checked in detail, but my guess is that later was not intended to be used this way. It manages it's own 'daemonized' httpuv event loop, so it may not handle running another daemonized httpuv event loop well. I imagine that the design would be to run the callbacks you would use with the 'daemonizedServer' with 'later' calls.

[thomasp85]

Oh, I'm aware of all the good reasons why this might fail - @jcheng5 just asked me to add the issue. :-)

[wch]

Running this exact set of commands from a terminal will install later with the proper debugging info and allow setting a breakpoint. (According to @jimhester, breakpoints don't work with install_github() because they need the .o files to stick around, but install_github() puts those in a temp dir which gets deleted.)

git clone git@github.com:r-lib/later.git
cd later

R --debugger=lldb
b later_posix.cpp:76
run
devtools::clean_dll(); devtools::install()
library(httpuv)
library(later)
sessioninfo::session_info()
server <- startDaemonizedServer('127.0.0.1', 8080L, list())
later::later(function() { stopDaemonizedServer(server) }, 1)

If this all works properly, the output looks like this (on my mac):

$ R --debugger=lldb
*** Further command line arguments ('--no-save --no-restore-data --quiet ') disregarded
*** (maybe use 'run --no-save --no-restore-data --quiet ' from *inside* lldb)

(lldb) target create "/Library/Frameworks/R.framework/Resources/bin/exec/R"
Current executable set to '/Library/Frameworks/R.framework/Resources/bin/exec/R' (x86_64).
(lldb) b later_posix.cpp:76
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) run
Process 15359 launched: '/Library/Frameworks/R.framework/Resources/bin/exec/R' (x86_64)
library(httpuv)
library(later)
sessioninfo::session_info()
server <- startDaemonizedServer('127.0.0.1', 8080L, list())
later::later(function() { stopDaemonizedServer(server) }, 1)


> library(httpuv)
> library(later)
1 location added to breakpoint 1
> sessioninfo::session_info()
─ Session info ───────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 3.4.1 (2017-06-30)
 os       macOS Sierra 10.12.6        
 system   x86_64, darwin15.6.0        
 ui       X11                         
 language (EN)                        
 collate  en_US.UTF-8                 
 tz       America/Chicago             
 date     2017-09-05                  

─ Packages ───────────────────────────────────────────────────────────────────
 package     * version date       source                            
 clisymbols    1.2.0   2017-05-21 CRAN (R 3.4.0)                    
 devtools      1.13.3  2017-08-02 CRAN (R 3.4.1)                    
 digest        0.6.12  2017-01-27 CRAN (R 3.4.0)                    
 git2r         0.19.0  2017-07-19 CRAN (R 3.4.1)                    
 httpuv      * 1.3.5   2017-07-04 CRAN (R 3.4.1)                    
 knitr         1.17    2017-08-10 CRAN (R 3.4.1)                    
 later       * 0.4     2017-09-05 local (r-lib/later@NA)            
 memoise       1.1.0   2017-04-21 CRAN (R 3.4.0)                    
 Rcpp          0.12.12 2017-07-15 CRAN (R 3.4.1)                    
 sessioninfo   1.0.1   2017-09-05 Github (r-lib/sessioninfo@e813de4)
 withr         2.0.0   2017-07-28 CRAN (R 3.4.1)                    
> server <- startDaemonizedServer('127.0.0.1', 8080L, list())
> later::later(function() { stopDaemonizedServer(server) }, 1)
> 
> later.so was compiled with optimization - stepping may behave oddly; variables may not be available.
Process 15359 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x00000001077afb04 later.so`async_input_handler(data=0x0000000000000000) at later_posix.cpp:76 [opt]
   73   };
   74   
   75   static void async_input_handler(void *data) {
-> 76     if (!at_top_level()) {
   77       // It's not safe to run arbitrary callbacks when other R code
   78       // is already running. Wait until we're back at the top level.
   79       

Next, we can look at the items in the R_InputHandlers linked list:

ta v *R_InputHandlers
ta v *R_InputHandlers->next
ta v *R_InputHandlers->next->next
ta v *R_InputHandlers->next->next->next

The output is:

(lldb) ta v *R_InputHandlers
(InputHandler) *R_InputHandlers = {
  activity = 2
  fileDescriptor = 0
  handler = 0x0000000000000000
  next = 0x0000000100c4e7e0
  active = 0
  userData = 0x0000000000000000
}
(lldb) ta v *R_InputHandlers->next
(_InputHandler) *R_InputHandlers->next = {
  activity = 20
  fileDescriptor = 3
  handler = 0x0000000104fdab00 (later.so`async_input_handler(void*) at later_posix.cpp:75)
  next = 0x0000000104d00000
  active = 0
  userData = 0x0000000000000000
}
(lldb) ta v *R_InputHandlers->next->next
(_InputHandler) *R_InputHandlers->next->next = {
  activity = 55
  fileDescriptor = 13
  handler = 0x0000000107cd4360 (httpuv.so`loop_input_handler(void*) at httpuv.cpp:439)
  next = 0x0000000104d00030
  active = 0
  userData = 0x0000000000000000
}
(lldb) ta v *R_InputHandlers->next->next->next
(_InputHandler) *R_InputHandlers->next->next->next = {
  activity = 57
  fileDescriptor = 7
  handler = 0x0000000107cd4360 (httpuv.so`loop_input_handler(void*) at httpuv.cpp:439)
  next = 0x0000000000000000
  active = 0
  userData = 0x0000000000000000
}

The problem occurs in R_runHandlers, when one of the input handlers removes the next handler in the list.

The second item, later.so`async_input_handler, calls the R function passed in, which calls stopDaemonizedServer. That function in turn removes the input handlers added by httpuv (here). So the second item in the linked list removes the third and fourth item, but by the time it runs, next already points to the third item, and in the next iteration of the loop, it points at memory that has been freed.

[wch]

A smaller example that hits this issue. This sets up two handlers, handler1 and handler2. The first one removes both of the handlers; the second one prints "Running handler2". In this case, both handlers end up running, even though handler1 removes handler2 before it is run:

Rcpp::cppFunction(
  includes = '
  #include <R_ext/eventloop.h>
  #include <stdio.h>
  #include <sys/stat.h> 
  #include <fcntl.h>

  InputHandler* handler1;
  InputHandler* handler2;

  int fd = 0;

  void print_message_handler2(void* data) {
    fprintf(stderr, "Running handler2\\n");
  }

  void remove_handlers(void* data) {
    fprintf(stderr, "Removing handlers\\n");

    removeInputHandler(&R_InputHandlers, handler1);
    fprintf(stderr, "  removed handler1\\n");

    removeInputHandler(&R_InputHandlers, handler2);
    fprintf(stderr, "  removed handler2\\n");
  }
  ',
  '
  long int add_handler() {
    if (fd == 0)
      fd = open("foo.txt", O_RDONLY);

    handler1 = addInputHandler(R_InputHandlers, fd, &remove_handlers, 1000);
    handler2 = addInputHandler(R_InputHandlers, fd, &print_message_handler2, 1001);
    return (long int) handler1;
  }',
  verbose = TRUE, rebuild = TRUE
)

file.create("foo.txt")
add_handler()
# [1] 4330756464
# Removing handlers
#   removed handler1
#   removed handler2
# Running handler2

If you switch the order that the handlers are removed, it results in the same kind of segfault that we see in this bug. Running with lldb:

R --debugger=lldb
run --quiet

Rcpp::cppFunction(
  includes = '
  #include <R_ext/eventloop.h>
  #include <stdio.h>
  #include <sys/stat.h> 
  #include <fcntl.h>

  InputHandler* handler1;
  InputHandler* handler2;

  int fd = 0;

  void print_message_handler2(void* data) {
    fprintf(stderr, "Running handler2\\n");
  }

  void remove_handlers(void* data) {
    fprintf(stderr, "Removing handlers\\n");

    removeInputHandler(&R_InputHandlers, handler2);
    fprintf(stderr, "  removed handler2\\n");

    removeInputHandler(&R_InputHandlers, handler1);
    fprintf(stderr, "  removed handler1\\n");
  }
  ',
  '
  long int add_handler() {
    if (fd == 0)
      fd = open("foo.txt", O_RDONLY);

    handler1 = addInputHandler(R_InputHandlers, fd, &remove_handlers, 1000);
    handler2 = addInputHandler(R_InputHandlers, fd, &print_message_handler2, 1001);
    return (long int) handler1;
  }',
  verbose = TRUE, rebuild = TRUE
)

file.create("foo.txt")
add_handler()
# [1] 4320419344
# Removing handlers
#   removed handler2
#   removed handler1
# libR.dylib was compiled with optimization - stepping may behave oddly; variables may not be available.
# Process 17953 stopped
# * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x13)
#     frame #0: 0x000000010025f76e libR.dylib`Rstd_ReadConsole [inlined] R_runHandlers(handlers=<unavailable>, readMask=<unavailable>) at sys-std.c:380 [opt]