popupTable for many points crashes RStudio session

[tim-salabim]

The following code crashes the RStudio session when n = 523000 but doesn't if n = 522000 on my linux machine.

library(sf)
library(mapview)

n = 522000 # works
# n = 523000 # crashes R session

df = data.frame(x = runif(n, 0, 40),
                y = runif(n, 0, 40),
                z = sample.int(3, n, replace = TRUE))

df = st_as_sf(df, coords = c("x", "y"), crs = 4326)

pp = popupTable(df)

This happens somewhere in listPopupTemplates.

@fdetsch do you have any idea why this crashes? Can we make this safer so that it doesn't crash the entire session?

Note: R does not crash when run from the terminal, but the following error is emitted
Error: segfault from C stack overflow
I have not pinned down the exact n when this happens in native R, but n = 523000 works, n = 1e6 causes segfault.

> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.4 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=de_DE.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=de_DE.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=de_DE.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mapview_2.4.1 tictoc_1.0    leaflet_2.0.0 sf_0.6-2     

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.16      plyr_1.8.4        compiler_3.4.4    R.methodsS3_1.7.1
 [5] base64enc_0.1-3   R.utils_2.5.0     class_7.3-14      iterators_1.0.8  
 [9] tools_3.4.4       gdalUtils_2.0.1.7 digest_0.6.15     viridisLite_0.3.0
[13] satellite_1.0.0   lattice_0.20-35   png_0.1-7         foreach_1.4.3    
[17] shiny_1.0.4       DBI_0.8           crosstalk_1.0.0   RPostgreSQL_0.6-2
[21] rgdal_1.2-8       e1071_1.6-8       raster_2.6-7      htmlwidgets_1.0  
[25] webshot_0.4.1     classInt_0.1-24   stats4_3.4.4      grid_3.4.4       
[29] R6_2.2.2          sp_1.2-7          udunits2_0.13     magrittr_1.5     
[33] scales_0.5.0.9000 codetools_0.2-15  htmltools_0.3.6   units_0.5-1      
[37] colorspace_1.3-2  mime_0.5          xtable_1.8-2      httpuv_1.3.5     
[41] munsell_0.4.3     R.oo_1.21.0

RStudio version is 1.1.383

[tim-salabim]

On my windows machine at work the respective n values are

• fine 981000
• breaks 982000

> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252    LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mapview_2.4.2 sf_0.6-2     

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.16       plyr_1.8.4         compiler_3.5.0     later_0.7.1        R.methodsS3_1.7.1  R.utils_2.6.0      base64enc_0.1-3   
 [8] class_7.3-14       iterators_1.0.9    tools_3.5.0        gdalUtils_2.0.1.14 digest_0.6.15      viridisLite_0.3.0  satellite_1.0.1   
[15] lattice_0.20-35    png_0.1-7          foreach_1.4.4      shiny_1.0.5        DBI_0.8            crosstalk_1.0.0    RPostgreSQL_0.6-2 
[22] yaml_2.1.18        rgdal_1.2-18       spData_0.2.8.3     e1071_1.6-8        raster_2.6-7       htmlwidgets_1.2    webshot_0.5.0     
[29] classInt_0.2-3     stats4_3.5.0       leaflet_2.0.0      grid_3.5.0         R6_2.2.2           sp_1.2-7           udunits2_0.13     
[36] magrittr_1.5       scales_0.5.0       codetools_0.2-15   promises_1.0.1     htmltools_0.3.6    units_0.5-1        colorspace_1.3-2  
[43] mime_0.5           xtable_1.8-2       httpuv_1.4.1       munsell_0.4.3      R.oo_1.22.0

[tim-salabim]

On windows in R started from cmd I can happily produce 3 million popups (i.e. n = 3e6). Didn't test any further as map won't render with that many points anyway... This is going to be fun to debug :-)

[edzer]

I can confirm your finding on rstudio/ubuntu 17.10.

[edzer]

I can reproduce this in on ubuntu/bionic with R 3.4.4 (no rstudio), seeing (in valgrind):

> pp = popupTable(df)
==6048== Stack overflow in thread #1: can't grow stack to 0x1ffe801000
Error: segfault from C stack overflow
==6048== Warning: client switching stacks?  SP change: 0xca37468 --> 0x1ffefff7a0
==6048==          to suppress, use: --max-stackframe=137210135352 or greater
Execution halted

[fdetsch]

As indicated by @edzer, this seems like some memory allocation issue related to our generous use of List output in the C++ code for listPopupTemplates(). See also here and here for similar reportings.

In any case, as of 3d88f6e, List output has been replaced with StringVector and the above popupTable() call now works in RStudio on my machine (which used to crash before) with a dataset consisting of 1e6+ rows.

library(sf)
library(mapview)

n = 1e6 

df = data.frame(x = runif(n, 0, 40),
                y = runif(n, 0, 40),
                z = sample.int(3, n, replace = TRUE))

df = st_as_sf(df, coords = c("x", "y"), crs = 4326)

pp = popupTable(df)
pp[5e5]
# [1] "<html><head><link rel=\"stylesheet\" type=\"text/css\" href=\"lib/popup/popup.css\"></head><body><div class=\"scrollableContainer\"><table class=\"popup scrollable\"><table id=\"popup\"><tr class='coord'><td></td><td><b>Feature ID</b></td><td align='right'>500000&emsp;</td></tr><tr class='alt'><td>1</td><td><b>z&emsp;</b></td><td align='right'>3&emsp;</td></tr><tr><td>2</td><td><b>geometry&emsp;</b></td><td align='right'>sfc_POINT&emsp;</td></tr></table></body></html>"

Here's my sessionInfo():

> sessionInfo()
R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252    LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mapview_2.4.5 sf_0.6-2     

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.16        RColorBrewer_1.1-2  plyr_1.8.4          compiler_3.5.0      later_0.7.2         R.methodsS3_1.7.1   R.utils_2.6.0      
 [8] base64enc_0.1-3     class_7.3-14        iterators_1.0.9     tools_3.5.0         gdalUtils_2.0.1.14  digest_0.6.15       memoise_1.1.0      
[15] viridisLite_0.3.0   satellite_1.0.1     lattice_0.20-35     png_0.1-7           foreach_1.4.4       shiny_1.0.5         DBI_1.0.0          
[22] crosstalk_1.0.0     RPostgreSQL_0.6-2   yaml_2.1.19         rgdal_1.2-20        xfun_0.1            spData_0.2.8.3      e1071_1.6-8        
[29] knitr_1.20          withr_2.1.2         raster_2.6-7        devtools_1.13.5     htmlwidgets_1.2     webshot_0.5.0       classInt_0.2-3     
[36] stats4_3.5.0        leaflet_2.0.0       grid_3.5.0          R6_2.2.2            bookdown_0.7        sp_1.2-7            latticeExtra_0.6-28
[43] udunits2_0.13       magrittr_1.5        Orcs_1.0.7          scales_0.5.0        codetools_0.2-15    promises_1.0.1      htmltools_0.3.6    
[50] units_0.5-1         colorspace_1.3-2    mime_0.5            xtable_1.8-2        httpuv_1.4.2        munsell_0.4.3       R.oo_1.22.0    

[tim-salabim]

Sweet, on my linux machine, I can now run this with n = 1e7 without crashing! Thanks @fdetsch for taking care of it!
I'll test on windows tomorrow.

[tim-salabim]

Confirm working for windows, hence closing this. Thanks again @fdetsch !!