Fatal error when loading workspace image with objects containing a null external pointer and a prototype from class definition

[Sciurus365]

System details

RStudio Edition : Desktop
RStudio Version : 1.4.1103
OS Version      : Win10
R Version       :  4.0.3

Steps to reproduce the problem

library(bigmemory)
setClass("testClass1",
				 slots = c(md5 = "character"),
				 contains = "big.matrix")

setClass("testClass2",
				 slots = c(md5 = "character"),
				 prototype = c(md5 = "1"),
				 contains = "big.matrix")

t1 <- as.big.matrix(matrix(1:100, nrow = 10))

t2 <- new("testClass1", address = t1@address, md5 = digest::digest(matrix(1:100, nrow = 10)))

t3 <- new("testClass2", address = t1@address, md5 = digest::digest(matrix(1:100, nrow = 10)))

Describe the problem in detail

If I run the codes above, save the workspace, restart RStudio again, it will throw a fatal error. This only occurs if t3 is in the workspace, and only occurs in RStudio (RGui doesn't have this problem).
I guess a part of this problem comes from the behavior of bigmemory package: it generates external pointers which will become null in the next time. If I click t2 with a null pointer to view it, the fatal error of R will also occur. (This is easy to avoid: as long as I don't click it, it's fine.)
But for t3, whenever I load the workspace, that fatal error occurs.

Another thing I found is that t2 is in the Data section of the variable inspector, while t3 is in the Values section. I suspect this is related to this problem, but I'm not sure.

Describe the behavior you expected

Successfully load the abovementioned objects.

• I have read the guide for submitting good bug reports.
• I have installed the latest version of RStudio, and confirmed that the issue still persists.
• If I am reporting a RStudio crash, I have included a diagnostics report.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

[ronblum]

@Sciurus365 Thank you for raising the issues! I can reproduce it in

• RStudio Desktop 1.4.1540 on MacOS 11.3 beta
• RStudio Server 1.4.1533 on Red Hat 8.3

The comparable reprex is with Session -> Restart R and waiting a moment.
We'll review this as we continue development of RStudio.

[Sciurus365]

@ronblum
Thanks! And a related issue I just found is that even if there's no prototype, if the objects are nested in a data frame, the same problem also occurs.

library(bigmemory)
setClass("testClass1",
				 slots = c(md5 = "character"),
				 contains = "big.matrix")

t1 <- as.big.matrix(matrix(1:100, nrow = 10))

tlist <- list()
for(i in 1:10){
	tlist <- append(tlist, new("testClass1", address = t1@address, md5 = digest::digest(matrix(1:100, nrow = 10))))
}

tdf <- data.frame(x = 1:10)
tdf$testClass1 <- tlist

And suppressing the variable inspector (by choosing "Manual Refresh Only" can solve this. So I'm now quite sure it's related to the variable inspector.

[ronblum]

@Sciurus365 I ran into this again, more or less by accident, and recorded it on MacOS 11.3. Is this what you're seeing?

bigmemory.mov

[Sciurus365]

@Sciurus365 I ran into this again, more or less by accident, and recorded it on MacOS 11.3. Is this what you're seeing?

bigmemory.mov

Yes exactly

[kevinushey]

I believe this is a bug that should be fixed in the bigmemory package. You can reproduce something similar in plain R, with:

library(bigmemory)
t1 <- as.big.matrix(matrix(1:100, nrow = 10))
saveRDS(t1, file = "bigmem.rds")
t1 <- readRDS("bigmem.rds")
str(t1)

The call to str() segfaults, because bigmemory fails to check for null external pointers. We could (and perhaps should) work around this by avoiding calls to str() on such objects. My suspicion is that this is happening here:

rstudio/src/cpp/session/modules/SessionEnvironment.R

Lines 45 to 51 in 4d1fc87

	.rs.addFunction("valueFromStr", function(val)
	{
	.rs.withTimeLimit(1, fail = "<truncated>", {
	capture.output(try(str(val), silent = TRUE))
	})
	})

[kevinushey]

In addition, working around this in RStudio would be quite challenging because the segfault is actually occurring in the call to nrow(), which is attempting to dispatch to a non-existent external pointer:

$ Rscript bigmem.R

 *** caught segfault ***
address 0x10, cause 'memory not mapped'

Traceback:
 1: CGetNrow(x@address)
 2: nrow(x)
 3: nrow(x)
 4: dim(x)
 5: dim(x)
 6: length(object)
 7: length(object)
 8: utils:::str.default(t1)
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault: 11

[Sciurus365]

I believe this is a bug that should be fixed in the bigmemory package. You can reproduce something similar in plain R, with:

library(bigmemory)
t1 <- as.big.matrix(matrix(1:100, nrow = 10))
saveRDS(t1, file = "bigmem.rds")
t1 <- readRDS("bigmem.rds")
str(t1)

The call to str() segfaults, because bigmemory fails to check for null external pointers. We could (and perhaps should) work around this by avoiding calls to str() on such objects. My suspicion is that this is happening here:

rstudio/src/cpp/session/modules/SessionEnvironment.R

Lines 45 to 51 in 4d1fc87

	.rs.addFunction("valueFromStr", function(val)
	{
	.rs.withTimeLimit(1, fail = "<truncated>", {
	capture.output(try(str(val), silent = TRUE))
	})
	})

@kevinushey Thanks for your reply. Indeed I can reproduce the problem with base::str(). I don't think this can be fixed by big.matrix either because there's no way to retain the external pointer after closing the session. This, however, can be solved by reattaching the external objects. This needs to be done manually before the variable inspector crashes. So I'm wondering is it possible to set an option that prevents the variable inspector to use str or nrow (and only show the variables and the details of simple objects maybe), or let it use an error-catching way that prevents the system to crash?

[kevinushey]

We could handle this on the RStudio side, but it would be nice if bigmemory instead checked that any external pointers it tried to use were still valid (or at least not null pointers).