Error: protect(): protection stack overflow

[etiennebacher]

Hello, I've been following a bit your repos that use extendr since I'm trying to learn it. I ran some of your examples with more points to see if the performance gains scale well and I came across this error in h3_from_points():

library(h3o)

# works fine with 10,000 points
pnts <- tibble::tibble(
  x = runif(10000, -5, 10),
  y = runif(10000, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )

foo <- h3_from_points(pnts$geometry, 4)

# error with 100,000 points
pnts <- tibble::tibble(
  x = runif(100000, -5, 10),
  y = runif(100000, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )

foo <- h3_from_points(pnts$geometry, 4)
#> Error: protect(): protection stack overflow

I know it's still early in development and I'm not using these geospatial features anyway, but I just thought you'd want to know.

Session info if needed

─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.2.2 (2022-10-31 ucrt)
 os       Windows 10 x64 (build 19044)
 system   x86_64, mingw32
 ui       RStudio
 language (EN)
 collate  English_Europe.utf8
 ctype    English_Europe.utf8
 tz       Europe/Paris
 date     2023-03-13
 rstudio  2022.12.0+353 Elsbeth Geranium (desktop)
 pandoc   3.1 @ C:/Users/etienne/AppData/Local/Pandoc/ (via rmarkdown)
─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

package     * version    date (UTC) lib source

bench         1.1.2      2021-11-30 [1] CRAN (R 4.2.0)

callr         3.7.3      2022-11-02 [1] CRAN (R 4.2.2)

class         7.3-21     2023-01-23 [1] CRAN (R 4.2.2)

classInt      0.4-9      2023-02-28 [1] CRAN (R 4.2.2)

cli           3.6.0      2023-01-09 [1] CRAN (R 4.2.2)

clipr         0.8.0      2022-02-22 [1] CRAN (R 4.2.0)

curl          5.0.0      2023-01-12 [1] CRAN (R 4.2.2)

DBI           1.1.3      2022-06-18 [1] CRAN (R 4.2.1)

digest        0.6.31     2022-12-11 [1] CRAN (R 4.2.2)

dplyr         1.1.0      2023-01-29 [1] CRAN (R 4.2.2)

e1071         1.7-13     2023-02-01 [1] CRAN (R 4.2.2)

evaluate      0.20       2023-01-17 [1] CRAN (R 4.2.2)

fansi         1.0.4      2023-01-22 [1] CRAN (R 4.2.2)

fastmap       1.1.1      2023-02-24 [1] CRAN (R 4.2.2)

fs            1.6.0      2023-01-23 [1] CRAN (R 4.2.2)

generics      0.1.3      2022-07-05 [1] CRAN (R 4.2.1)

geojsonsf     2.0.3      2022-05-30 [1] CRAN (R 4.2.1)

glue          1.6.2      2022-02-24 [1] CRAN (R 4.2.0)

h3jsr         1.3.1      2023-01-21 [1] CRAN (R 4.2.2)

h3o         * 0.1.0      2023-03-13 [1] Github (JosiahParry/h3o@e50144c)

htmltools     0.5.4      2022-12-07 [1] CRAN (R 4.2.2)

jsonlite      1.8.4      2022-12-06 [1] CRAN (R 4.2.2)

KernSmooth    2.23-20    2021-05-03 [2] CRAN (R 4.2.2)

knitr         1.42       2023-01-25 [1] CRAN (R 4.2.2)

lifecycle     1.0.3      2022-10-07 [1] CRAN (R 4.2.1)

magrittr      2.0.3      2022-03-30 [1] CRAN (R 4.2.0)

pillar        1.8.1      2022-08-19 [1] CRAN (R 4.2.1)

pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.2.0)

processx      3.8.0      2022-10-26 [1] CRAN (R 4.2.2)

profmem       0.6.0      2020-12-13 [1] CRAN (R 4.2.0)

proxy         0.4-27     2022-06-09 [1] CRAN (R 4.2.0)

ps            1.7.2      2022-10-26 [1] CRAN (R 4.2.2)

purrr         1.0.1      2023-01-10 [1] CRAN (R 4.2.2)

R.cache       0.16.0     2022-07-21 [1] CRAN (R 4.2.1)

R.methodsS3   1.8.2      2022-06-13 [1] CRAN (R 4.2.0)

R.oo          1.25.0     2022-06-12 [1] CRAN (R 4.2.0)

R.utils       2.12.2     2022-11-11 [1] CRAN (R 4.2.1)

R6            2.5.1      2021-08-19 [1] CRAN (R 4.2.0)

rappdirs      0.3.3      2021-01-31 [1] CRAN (R 4.2.0)

Rcpp          1.0.10     2023-01-22 [1] CRAN (R 4.2.2)

reprex        2.0.2      2022-08-17 [1] CRAN (R 4.2.1)

rlang         1.0.6.9000 2023-02-20 [1] Github (r-lib/rlang@394204f)

rmarkdown     2.20       2023-01-19 [1] CRAN (R 4.2.2)

roxygen2      7.2.3.9000 2023-01-04 [1] Github (r-lib/roxygen2@39d2dbe)

rstudioapi    0.14       2022-08-22 [1] CRAN (R 4.2.1)

sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.2.0)

sf            1.0-9      2022-11-08 [1] CRAN (R 4.2.2)

shrtcts       0.1.2      2022-04-29 [1] Github (gadenbuie/shrtcts@12ef67f)

stringi       1.7.12     2023-01-11 [1] CRAN (R 4.2.2)

stringr       1.5.0      2022-12-02 [1] CRAN (R 4.2.2)

styler        1.9.1      2023-03-04 [1] CRAN (R 4.2.2)

tibble        3.1.8      2022-07-22 [1] CRAN (R 4.2.1)

tidyr         1.3.0      2023-01-24 [1] CRAN (R 4.2.2)

tidyselect    1.2.0      2022-10-10 [1] CRAN (R 4.2.1)

units         0.8-1      2022-12-10 [1] CRAN (R 4.2.2)

utf8          1.2.3      2023-01-31 [1] CRAN (R 4.2.2)

V8            4.2.2      2022-11-03 [1] CRAN (R 4.2.2)

vctrs         0.5.2.9000 2023-02-20 [1] Github (r-lib/vctrs@303b5dd)

withr         2.5.0      2022-03-03 [1] CRAN (R 4.2.0)

xfun          0.37       2023-01-31 [1] CRAN (R 4.2.2)

xml2          1.3.3      2021-11-30 [1] CRAN (R 4.2.0)

yaml          2.3.7      2023-01-23 [1] CRAN (R 4.2.2)

[1] C:/Users/etienne/AppData/Local/R/win-library/4.2

[2] C:/R/library

[JosiahParry]

Thanks @etiennebacher. I did know of this limitation but not in this context. @CGMossa thinks its related to extendr/extendr#397.

Here's something interesting. The number 50000 has something to do with it.

library(h3o)

# its exactly 50000 that causes an issue
n <- 49999
pnts <- tibble::tibble(
  x = runif(n, -5, 10),
  y = runif(n, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )


foo <- h3_from_points(pnts$geometry, 4)
#> Error: protect(): protection stack overflow


# set to 50000
n <- 50000
pnts <- tibble::tibble(
  x = runif(n, -5, 10),
  y = runif(n, 40, 50)
) |> 
  sf::st_as_sf(
    coords = c("x", "y"), 
    crs = 4326
  )


# watch error
foo <- h3_from_points(pnts$geometry, 4)
#> Error: protect(): protection stack overflow

https://github.com/extendr/extendr/blob/6e15390d2d4dbcd53878f4b31312c4fb68de25e6/extendr-api/src/ownership.rs#L34

50000 * 2 = 10000. 50000 robjs (Doubles) coming in and 50000 robjs (externalptrs) going out its exactly 10000. Can we turn that number up to 11?

[etiennebacher]

The number 50000 has something to do with it.

I don't understand why, the example also fails with 49,999

[JosiahParry]

The number can change by ± 5 or so I've noticed. But pretty reliably around that limit. But only for lists I've found.

[JosiahParry]

simple R package designed to illustrate the issue
https://github.com/JosiahParry/stackoverflow/blob/main/example.R

[rstub]

Using code from this package I have boiled this down to:

code <- 'fn stack_overflow_list(x: List) -> List {
    x
        .into_iter()
        .map(|(_, x)| {
            let dbs = Doubles::from_robj(&x).unwrap();
            [dbs[1].0, dbs[0].0]
        })
        .collect::<List>()
}'
rextendr::rust_function(code)

n <- 49984
pnts <- lapply(1:n, \(x) runif(2))
stack_overflow_list(pnts)

n <- 49993
pnts <- lapply(1:n, \(x) runif(2))
stack_overflow_list(pnts)

• Unfortunately, reprex does not render this for me.
• Interestingly, the threshold for the failure seems to be even lower.
• To me this looks like a bug in rextendr. Has this been reported there?

[CGMossa]

Hello @rstub. This is a bug in extendR. @JosiahParry have helped extensively with troubleshooting it. I wrote a PR for it here extendr/extendr#540, and it got reviewed and approved.. I just need to write one more test to merge it.
This will necessitate the use of extendr on github, in order to avoid this. I would say that you can circumvent this, if it is truly necessary.
Thanks for investigating it as well!

[JosiahParry]

Thanks both! Yeah, this bothered me for quite some time. For now, the only way I am aware of to avoid it is to pre-allocate the List and assign elements in a for loop using the .elt() method.

…

On Mon, May 8, 2023 at 04:17 CGMossa ***@***.***> wrote: Hello @rstub <https://github.com/rstub>. This is a bug in extendR. @JosiahParry <https://github.com/JosiahParry> have helped extensively with troubleshooting it. I wrote a PR for it here extendr/extendr#540 <extendr/extendr#540>, and it got reviewed and approved.. I just need to write *one more* test to merge it. This will necessitate the use of extendr on github, in order to avoid this. I would say that you *can* circumvent this, if it is truly necessary. Thanks for investigating it as well! — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADHIKLGSAEMSZDPM63MSDMTXFCTZPANCNFSM6AAAAAAVY5D5ZE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JosiahParry]

@etiennebacher can you install the new version from https://josiahparry.r-universe.dev/h3o and let me know if this is fixed?

[etiennebacher]

Hi @JosiahParry, I don't have this error anymore, thanks

[JosiahParry]

@etiennebacher brilliant :) thanks