Memory errors when using Rscript with read_tsv_chunked() with trailing tab

[Shians]

Sorry for the bizzare problem. When using read_tsv_chunked() on a large enough malformed file, R throws random memory issues. When running this in Rstudio I get the classic bomb with unexpected termination. But when running via Rscript on console I get a random sample of

 *** caught segfault ***
address 0x55fd97fb94f8, cause 'memory not mapped'

or

malloc(): unsorted double linked list corrupted

or

*** recursive gc invocation
*** recursive gc invocation
*** recursive gc invocation
*** recursive gc invocation
*** recursive gc invocation
...many more

and potentially other memory errors that I have not noted down.

The code below is the minimal triggering condition on Pop!OS 20.04 LTS (Ubuntu 20.04). Run via Rscript with R4.0.3 and readr 1.4.0.

writeLines(paste("x", 1:6, sep = "", collapse = "\t"), "test.txt")
write(
    rep(paste0(paste(letters[1:6], collapse = "\t"), "\t"), 1e7), # trailing tab
    "test.txt",
    append = TRUE
)


readr::read_tsv_chunked(
    "test.txt",
    callback = readr::SideEffectChunkCallback$new(
        function(x, i) {}
    )
)

This particular instance throws

malloc(): unsorted double linked list corrupted

• Does not trigger when file does not have trailing tabs
• Does not trigger with 1e6 rows
• Does not trigger using regular read_tsv(), though it will flag many problems

I can simply fix my file but the nature of the error message made it incredibly difficult to diagnose the root of the problem.

[jimhester]

This turned out to be a protection issue in the cpp11 package. I believe with that change this is now fixed.

Unfortunately we just had a release of cpp11 yesterday, so we will have to wait at least a week to do another CRAN release with this fix.

In the meantime you can install the development versions of cpp11 and then re-install readr.

[Shians]

Confirming fix in my test case and actual data.