RestartSession doesn't run the command after restarting in R3.5.1

[jlcsw]

Hi,

I am using restartSession to deal with the problem of insufficient memory and need to clear the memory after each for loop iteration. I have written a code with the intention that a for loop will continue after restarting the R session.

I am grateful for the function restartSession as it allowed my code to run correctly for a week straight before memory became a problem again.

Part of how I do this is to get the restartSession function to source the same script again. This worked for a while but now it does not work anymore. I tried running a simple print('Hello') and '1+1' as the command to run after restarting but there was no output after r restarted.

The actual code which I ran previously is simple.
restartSession(command = "source(path)")

The path is correct because if I copy out the source(path) it runs in the console correctly.

I appreciate any help to troubleshooting the problem.

[kevinushey]

I wasn't able to reproduce this -- the command runs as expected for me. Can you provide any other information / a reproducible example?

[jlcsw]

The truth is. The function worked for an entire week, I really don't know what happened and I can't reproduce it. The function works when I tried it on another computer. I am thinking it might be due to some error or crash file. I'll am going to try shutting down the computer and reinstalling R and RStudio to see if it resolves the problem. Is this the first time you have heard of this issue?

[kevinushey]

Indeed, this is the first time we've had this issue reported. I can still believe that it can happen, though :-) If you happen to figure out what might've caused this, please let us know!

[jlcsw]

The function works again! I tried everything I could think of. I deleted the history of the environment in the main folder and cleared the crash files but it didn't work however shutting the computer down and re-installing did solve the problem. The script will likely take another 2 weeks to run but I have edited the code a bit to print out the progress so I will at least know which part of the code is causing it when it occurs again. The only thing notable was the code stopped because of the lack in memory, although deleting the crash files and quitting RStudio did not solve it. I am wondering how R and RStudio handle code when it crashes.

[jlcsw]

Hi Kevin,

The problem occurred again and I think I found the problem. It isn't related to the function. Apparently, there is some hidden process that is using all of the memory in the C drive, because there is no more memory, R cannot create a temporary file for the session and the script terminates. When this happens the function no longer works regardless of what I do.

I tried to find out if it is the temporary files that are taking up the majority of the space but they were not. I have not been able to detect a virus and system restore is not the cause. Nothing I did seem to reduce the memory usage in C drive. Without reason, about 70GB of space was suddenly freed, however, the function did not work until the computer was restarted. I am not one hundred per cent sure that the disk space issue is the cause of the malfunction but it is my best guess of the issue though I have not been able to ascertain the source of the problem.

[kevinushey]

That could explain it. restartSession() needs to write some local state files to do what it needs to do, so if it was unable to do this (e.g. due to lack of disk space) that could explain what's going on.

[sjones29]

Is there a solution to this? I'm trying to iterate/save RAM in same way as @jlcsw (using restartSession with the command to source the same file). Running into the same issue where the process works fine for a while and then abruptly stops. I still can't figure out why it's happening.

[sjones29]

@kevinushey apologies forgot to tag you. Please see my previous comment.

[jlcsw]

@sjones29 I did notice that regardless of what I did, the script will terminate after 100 restarts. I am not sure if that is the limit. My solution to the problem was to minimise the number of restarts needed. I don't have a better way unfortunately. It took a lot of experimentation to find something optimal but I manage to get what I need. Hope it helps.

[sjones29]

@jlcsw thanks for the input. Interestingly, my script restarts a variable number of times before I have to manually restart the program; sometimes, I can get significantly over 100 iterations, sometimes only 40 or so, sometimes much less. I may end up trying to optimize the number of restarts as you suggested, even if it eats up more RAM. Haven't gotten any further along in terms of pinpointing the source of the restart interference, but I'll keep looking at it.

[MilesMcBain]

I've just discovered that this function preserves packages and variables in the global environment between restarts, see: #111. Accumulating variables from long running iterative processes is probably what is gobbling user's disk space.

This was certainly unexpected to me, and seemingly to the other posters in this thread.

Edit: the more I think about this the more certain I am this is a bug.

[jkpm18]

Has anyone solved this yet. I used restartSession() in a loop, and it worked for a while. However, out of the blue the function no longer seems to act on the command argument. Here is a simple example to illustrate the issue that is bothering me. Previously running the line

restartSession(command="print('Hello')")

would restart my session and then print "Hello" to the console. Now, however, running that exact same line of code will restart my session, but it will not print "Hello" any more.

[GegznaV]

I cannot reproduce this. Can you add session information?

[PetoLau]

For example, when I run:
for (x in 1:3) {
print(paste(x, "before res"))
rstudioapi::restartSession(command = "print(x)")
print(paste(x, "after res"))
}

the output is:
[1] "1 before res"
[1] "1 after res"
[1] "2 before res"
[1] "2 after res"
[1] "3 before res"
[1] "3 after res"

Restarting R session...

print(x)
[1] 3

So it restarts session only once, and runs print functions with paste at once...

Is there a way to do it correctly?

[jimwhamilton]

I am having similar issues with the command not being executed after restartSession is called.

I have created a simple example to demonstrate the issue I am having:

Saved the following code to my working directory in a file called restartSessionTest.R:

if(exists("test")) {
test = test + 1
} else {
test = 1
}
cat(paste0("Running Restart Test ", test ,"\n"))
#Sys.sleep(1)
rstudioapi::restartSession(command = "source('restartSessionTest.R')")
cat("Why can I see this?\n")

This normally runs for about ~10 restarts before it falls over giving the following error:
Error: Unable to establish connection with R session

A few other observations:

1. Why is cat statement beneath the restartSession executed? I would not expect this to occur
2. If I add a small pause e.g. Sys.sleep(1) to the line above the restartSession, I get more successful restarts (sometimes ~50)
3. As the number of restarts increases the backing up of the session appears to take longer and longer (particularly noticeable with high numbers of restarts > 30)
4. After the restart fails, if I restart the code without manually restarting my RStudio session (Ctrl-Shift-F10) then it never runs for more than 1 restart without falling over again.

Hope this helps diagnose the problem.

Thanks,
Jim

RStudio Version 1.2.5001

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8.1 x64 (build 9600)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.6.1 tools_3.6.1   

[jimwhamilton]

Could I just ask if anyone else has been able to replicate this behaviour or is it just me?

[antriksht]

Sorry about pinging this again but was anyone able to figure out a solution here?

[kevinushey]

Sorry, but we still haven't been able to reproduce and so don't know what the cause could be.

[alexmitrani]

Could I just ask if anyone else has been able to replicate this behaviour or is it just me?

I tried running your script and it seemed to run indefinitely, it didn't crash, I had to stop it.

[alexmitrani]

I needed this restart session command to be able to run a model for a large batch of runs, cleaning out the memory in a comprehensive way between each run.

I experienced all of the problems mentioned in earlier comments,

The solution I found was to save each scenario command in a separate script file and to source the script within the restart session command. I first tried it like this, but it dodn't work:

restartSession(command='source("s718.R")')
restartSession(command='source("s719.R")')
restartSession(command='source("s720.R")')
restartSession(command='source("s726.R")')
restartSession(command='source("s727.R")')
restartSession(command='source("s728.R")')
restartSession(command='source("s729.R")')
restartSession(command='source("s730.R")')
restartSession(command='source("s731.R")')

This does not work because when R restarts it seems to lose the thread of what it was doing and it just stops. So the above script ran the first scenario then stopped.

What does seem to work is the following.

helloworld <- function(counter) {

cat(yellow(paste0("hello world ", counter)))

}

hw1.R:
helloworld(1)
restartSession(command='source("hw2.R")')

hw2.R:
helloworld(2)
restartSession(command='source("hw3.R")')

... and so on.

The key seems to be to call the code that you want R to continue executing within the command string that you provide together with the restartSession command.

It's like Russian dolls, or the film "Inception"!

The above procedure worked with the simple "helloworld" function, but it does not work when I try to do my model runs with this same approach, for some reason the command passed to restartSession does not get executed.

[alexmitrani]

I tried restarting the computer as suggested by @jlcsw but unfortunately it still won't work on the second iteration.

restartSession(command='source("s718.R")')

The above line works.

s718.R:
Sys.sleep(30)
fast (scenario = 718)
restartSession(command='source("s719.R")')

The model runs, the R session restarts, but the command 'source("s719.R")' is not executed. I don't know what to do, other than perhaps calling R from a Windows batch file instead of from RStudio. Any help would be much appreciated.

[braitmaier]

@alexmitrani: I'm not sure what the issue with the restartSession() function is and it's a bit difficult to reproduce, but instead you can send the "heavy work" to a new session, e.g. via the callr package. It would be much more convenient just using the restartSession() function though...

See a toy example below:

### packages
library(callr)

### directory path
code.dir <- "C:\\Users\\..."

### directory path with doubly escaped backslash for piping
escape.path <- "C:\\\\Users\\\\..."

for (i in 1:5) {
### set up the code to be executed in a .r file (this is a simple example, calculating i^2 and storing the results in a .txt file)
file.create(paste(code.dir, "\\autocode_", i, ".r", sep = ""), overwrite = T)
sink(paste(code.dir, "\\autocode_", i, ".r", sep = ""))
cat("output <- data.frame(a=", i,"^2)", sep = "")
cat("\n")
cat("write.table(output, file = \"", escape.path, "\\\\result_", i, ".txt\", row.names=F)", sep = "")
sink()

### execute the file in new R session
callr::rscript(paste(code.dir, "\\autocode_", i, ".r", sep = ""))
}

EDIT: This doesn't wait before starting the new job, so you still need to build in a check to see if the previous job finished before the new session is started. Should be doable though.

[alexmitrani]

This is helpful, thanks very much.

[aeiche01]

I had this same problem. I was finally able to get the command to run the script after restart by clicking the broom icon above the environments pane and sweeping away everything in the global environment. Something there might have been blocking it, but I don't know why.