Update Boost library to resolve "unable to establish connection with R session" errors (previously: Save a "standard" (non R-package) project on WINDOWS network

[gwd666]

System details

RStudio Edition : Desktop
RStudio Version : 1.3.1073, 1.2.5033 
OS Version      :  MS Windows 10 Enterprise - Version 10.0.17763 Build 17763
R Version       : R-3.6.3, R-3.5.3 

Steps to reproduce the problem

• Open RStudio
• Click File -> New Project
  In New Project Wizard:
• Create Project Select 'New Directory'
• Project Type: Select 'New Project'
  
  (Alternatively selecting "Open in New Session" in this window has the same result)
• Click 'Create Project'
  RESULT
  
  After clicking OK on the two error messages
  

Describe the problem in detail

See above (above screenshot) regarding ERROR messages.

If you select Project type: R package this error does not occur.

Error also happens if I Create Project from 'Existing Directory' when selecting a directory on the P: (Network) Drive.

Error is tricky to reproduce because in "some" network folders the project will manage to get created.
As far as I have identified the difference is that this "P" drive is a form of "GLOBAL" shared - meaning that it is shared on some form of "highest level" whereas the network drives that allow to create 'New Projects' have an additional layer of hierarchy in the sharing structure eg there is a network drive (call it U:) where there is a USERS level where every user will have his user folder to which he then has access rights, on these two-level elevated access structure, so to speak, "New Project" creation succeeds.

CURRENT WORKAROUND
Create 'New Project' type project locally or on one of the drives where it works and copy/paste to the (GLOBAL\SHARED) network drive.
Then click 'Open Project' and browse to that pasted directory (on GLOBAL\SHARED) and open the project there.

Describe the behavior you expected

Creation of projects of 'New Project' type on shared network drive.

• I have read the guide to submitting good bug reports at https://github.com/rstudio/rstudio/wiki/Writing-Good-Bug-Reports .
• I have installed the latest version of RStudio and confirmed that the issue still persists.
• If I am reporting a RStudio crash, I have included a diagnostics report. https://support.rstudio.com/hc/en-us/articles/200321257-Running-a-Diagnostics-Report
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

[kevinushey]

Thanks for the bug report. It looks like the issue is also described here:

https://stackoverflow.com/questions/44393241/boostfilesystemrelative-cannot-access-the-file-because-it-is-being-used-by

Fortunately, it should be fixed in the next Boost release:

boostorg/filesystem@726d2bf

so hopefully once we take our next Boost update this issue will be resolved.

[gwd666]

@kevinushey: thank you for the quick info
Having read the SO question I agree with you that it looks very much like this issue. Especially also the behavior that this error is only happening on some but not all network drives.
As a sidenote knitting on these drives is also effected.
Will close this as soon as I install and test with that boost update - keeping this issue open as a reminder to self.
Final 2 questions when do u expect yr update and will this boost update also effect RStudio 1.2.5033? Thanks

[gwd666]

@kevinushey
I have come to notice another issue in connection with (those) Windows network paths
When opening existing projects eg from one where the above BUG strikes you with not being able to SAVE standard projects (but also from the ones where it is possible) and then when switching (either directly via the Project dropdown or via 'close' and 'open') to another existing project it is really good advise to NOT attempt that on two different drive letters - it takes AGES (approx 5-10 minutes) to complete any of these transactions eg closing a project, which by the way completely hangs itself most of the times anyway.
AND heaven forbid you attempt to quit RStudio during that (Closing project) process - I was only able to make RStudio start up and not hang itself during the next start AFTER deleting the (whole) \LOCALAPPDATA\RStudio-Desktop folder. This frequently was necessary when attempting 'Session Quit' during 'Closing project' after getting to tired to wait any longer and then killing RStudio via the WINdows taskmanager after a couple approx. 20 or more minutes).
This basically makes dev work almost impossible as soon as I have projects saved on more than one drive (letter).
I think I have to correct myself ones more here ... after closing RStudio with an 'active' project (ie one selected/showing in the Projects dropdown) RStudio will basically hang on the next startup. AFter interrupting that process (just clicking x in upper right corner) it will start with 'None'project on the second attempt.
Then I will have to activate an existing project (via dropdown) again -> this means click and wait for about 10 minutes for it to switch there. If I switch between projects three or four times I can basically skip an hour of work a day - NOT NICE!
And man is any other interaction also SLOW - after activating the project it took a minute or so to Quit the Session.
As a final test I decided to give it a try and also attempt to wait it out during opening RStudio (after quitting the session on an active project, and not closing it prematurely) to see if it was just a matter of waiting as long as during switching. And look after watching this

for 10 minutes it managed to open with the last active project. So I guess deleting LOCALAPPDATA/RStudio-Desktop folder may not have been necessary it just resulted in dropping the last active project and eased the next startup ...
So if our hopes are on boost being the cure -> when do you think that update will happen?

[kevinushey]

Final 2 questions when do u expect yr update and will this boost update also effect RStudio 1.2.5033? Thanks

Sorry, but I'm not sure -- most likely, this update won't happen until next year; perhaps with the next version of RStudio (v1.5). I'm realize this isn't what you wanted to hear, but updating dependencies is time-consuming as it takes time for the team to run down regressions and other issues that typically arise after an update.

Per your most recent comment -- this sounds extraordinarily frustrating; I'm sorry that RStudio has been causing you so much trouble here. :-/ Are you able to collect any diagnostic data on what RStudio appears to be doing during these hangs (e.g. via Process Explorer, or other similar Windows Sysinternals tools?)

I'm wondering whether the issue might be something like:

1. The rsession.exe process itself is hung, or crashed, due to the aforementioned Boost issue;
2. The rsession.exe process is stuck trying to perform Disk IO; e.g. due to latency issues when accessing these network drives.

RStudio does try to write project-specific state to the project folder's (hidden) .Rproj.user folder both while the session is running, and during close. I wonder if the attempts to read and write that state are taking a very long time.

Or, if you've configured RStudio to auto-save and auto-load the R workspace, it could also be caused by latency in attempting to serialize / deserialize that workspace (typically a file called .RData at the project root).

[gwd666]

@kevinushey
Ouch - that boost update time is really bad news for saving standard projects on network drives.
I will try to gather as much additional Infos on these delays during project (open, close, switching) transactions as possible and add them here.
I am in a Corp env here and there was a SW update (of the user workstations) shortly before this project switching performance deterioration started showing up, although the OS Version and Build stayed the same it is always hard to tell what was all changed in the background (eg on the networking side)?

[gwd666]

Screenshot of my General settings [.RData Restore unticked, also deactivated Restore most recent project now -> that HELPS with regards to starting RStudio; this also improved the 'Quitting session' behavior/time a bit and weirdly only in some instances - seemed to depend on the drive - somehow it seemed that the drives where the above ERROR [during standard new project setup] comes up were the ones where closing the project was more efficient, compared to the ones where the above error did not show up, where closing a project almost takes as long as opening the first one - see below regarding opening first project].

CURRENTLY the only way for an acceptably efficient workflow is:

• DEACTIVATE the 'Restore most recent project' setting in "Tools->General"
• once a project is open ALWAYS "Open Projects in a New Session" instead of close/open or [directly] switching (selecting another project in the dropdown)
  -> OPENING (switching to) a fresh ie the "first" project (that resides on a network drive) still takes ages; it is only after that when you open a second one via Dropdown "Open Project in New Session", even when this next project is on a different drive ("letter") where the performance is "normal" (ie only a couple of seconds to open it).
• DO NOT ATTEMPT to "Session -> Restart R" in a Project (on Network drive) this will usually result in:

`Restarting R session...

Error: Unable to establish connection with R session` 

and from there Termination is the only remaining option.

To finish this comment here is a screenshot of the taskmanager right after this crash - @kevinushey pls NOTICE: the Microsoft Robocopy process [which I have never noticed before? in the RStudio App grouping?]

It does look (after a couple of variations) as if the drives that DO NOT have the (acc to @kevinushey probably) boost related issue with saving "Standard New Projects" are the ones having major issues with opening and closing existing (in my case R package) project files and the reasons for the delays/hangups is mainly that I am switching between these two type of drives during my tests and was not so sure on which side of the process it hangs itself.
If I stick solely to projects on the "boost broken" drives - all works normal.
Any attempt of opening a (R package) project on a drive that DOES NOT have the boost related problem (of not being able to save "Standard New Projects") hangs itself while doing so (> 5 minutes to open).

Hope that somehow helps @kevinushey and others with similar issues on Windows network shares.

[gwd666]

Final 2 questions when do u expect yr update and will this boost update also effect RStudio 1.2.5033? Thanks

Sorry, but I'm not sure -- most likely, this update won't happen until next year; perhaps with the next version of RStudio (v1.5). I'm realize this isn't what you wanted to hear, but updating dependencies is time-consuming as it takes time for the team to run down regressions and other issues that typically arise after an update.

Per your most recent comment -- this sounds extraordinarily frustrating; I'm sorry that RStudio has been causing you so much trouble here. :-/ Are you able to collect any diagnostic data on what RStudio appears to be doing during these hangs (e.g. via Process Explorer, or other similar Windows Sysinternals tools?)

I'm wondering whether the issue might be something like:

1. The rsession.exe process itself is hung, or crashed, due to the aforementioned Boost issue;
2. The rsession.exe process is stuck trying to perform Disk IO; e.g. due to latency issues when accessing these network drives.

RStudio does try to write project-specific state to the project folder's (hidden) .Rproj.user folder both while the session is running, and during close. I wonder if the attempts to read and write that state are taking a very long time.

Or, if you've configured RStudio to auto-save and auto-load the R workspace, it could also be caused by latency in attempting to serialize / deserialize that workspace (typically a file called .RData at the project root).

"auto-save and auto-load the R workspace" settings are usually the first ones that get deactivated after installation of RStudio.
I have come to realize that RStudio does not totally "hang up" - I was usually just to anxious and killing the processes after looking at blank or white startup screens for over 5 minutes (usually not even that clocky circle thing rotating in the middle) and not detecting any progress. Only after I took the time (ie wait for up to 10 minutes depending on the actions taken) I have come to realize that it does not crash BUT just takes vast amounts of time to complete the tasks - for some I have already found some workarounds (eg always open the next project as an "additional" one in a New Session, change some Settings - see other comment).

[kevinushey]

It may also be worth checking whether these projects are configured specifically to save + restore the R workspace, via Tools -> Project Options... in that open project:

I am a bit surprised to see Robocopy here, though -- RStudio doesn't explicitly use it anywhere when attempting to write or copy files. Is it possible that an R package loaded in the session is trying to use robocopy to copy files? Or is there another reason why it might be in use?

The renv package does use robocopy when attempting to copy directories on Windows; are you by any chance using renv in these projects?

[gwd666]

It may also be worth checking whether these projects are configured specifically to save + restore the R workspace, via Tools -> Project Options... in that open project:

I am a bit surprised to see Robocopy here, though -- RStudio doesn't explicitly use it anywhere when attempting to write or copy files. Is it possible that an R package loaded in the session is trying to use robocopy to copy files? Or is there another reason why it might be in use?

The renv package does use robocopy when attempting to copy directories on Windows; are you by any chance using renv in these projects?

Project Options -> General are: No, No and (Default) PLUS 'Quit child processes on exit ' is activated (see pic above).
Yes the slow starting project is using "renv" (but does so since day one ie it was already initially set up this way). I could try to deactivate 'renv' and retest behavior, but I would be quite unhappy to loose renv functionality :-(

[kevinushey]

I would be unhappy as well!

We could try adjusting some of renv's configuration settings, to see if that helps out with what's going on. What version of renv are you using in these projects? Are you able to update to the latest version?

For example, you could set the following R options; e.g. in the project's .Rprofile (or even in a running session):

options(renv.config.synchronized.check = FALSE)
options(renv.config.copy.method = "r")

It might also be worth trying to generate an R profile of what's going on. You can do so with:

utils::Rprof("~/rstudio-trace.Rprof")  # or pick another appropriate path for the trace to be written

After running this, try taking the action that is slow on your system (switching projects). After that's done, you can find the file at "~/rstudio-trace.Rprof", and share it with us.

[gwd666]

I would be unhappy as well!

We could try adjusting some of renv's configuration settings, to see if that helps out with what's going on. What version of renv are you using in these projects? Are you able to update to the latest version?

For example, you could set the following R options; e.g. in the project's .Rprofile (or even in a running session):

options(renv.config.synchronized.check = FALSE)
options(renv.config.copy.method = "r")

It might also be worth trying to generate an R profile of what's going on. You can do so with:

utils::Rprof("~/rstudio-trace.Rprof")  # or pick another appropriate path for the trace to be written

After running this, try taking the action that is slow on your system (switching projects). After that's done, you can find the file at "~/rstudio-trace.Rprof", and share it with us.

@kevinushey - will give it a shot after the weekend and get back with results

[gwd666]

@kevinushey
Pls find attached the trace file approx 18836 lines of often repeating commands written during switching to the project.
I copied utils::Rprof("~/rstudio-trace.Rprof") to the projects .Rprofile file, since running it in the Console and then opening the project did not produce anything except

sample.interval=20000
"f" "doTryCatch" "tryCatchOne" "tryCatchList" "tryCatch" "withCallingHandlers" ".rs.callAs" "utils::Rprof" 

as output in it.
Closing the project again also took some time, but only the last 5 lines were written to the trace file during that process ie these ones:

"file.exists" "<Anonymous>" 
"file.exists" "<Anonymous>" 
"file.exists" "<Anonymous>" 
"paste" "<Anonymous>" 
"makeRestartList" "withRestarts" ".signalSimpleWarning" "rm" "withCallingHandlers" "suppressWarnings" "<Anonymous>" 

rstudio-trace.Rprof.log

And as a final remark
Commenting out the line
source("renv/activate.R")
in the projects ".Rprofile" file also does the trick of normalizing the switching process.
Playing around with the two options settings (neither in Console nor in projects .Rprofile) you mentioned did not change anything.

BUT it is very probably safe to say that the "renv" package has something to do with that issue?!

G, W

[kevinushey]

Thanks! From the trace, it's clear that this is being caused by renv. Calling:

utils::summaryRprof("rstudio-trace.Rprof.log")

shows that most of the time was spent in these functions:

$by.total
                                         total.time total.pct self.time
"doTryCatch"                                 375.10     99.55      0.00
"tryCatch"                                   375.10     99.55      0.00
"tryCatchList"                               375.10     99.55      0.00
"tryCatchOne"                                375.10     99.55      0.00
"eval"                                       373.90     99.23      0.08
"withCallingHandlers"                        373.90     99.23      0.00
"eval.parent"                                373.86     99.22      0.00
"local"                                      373.86     99.22      0.00
"source"                                     373.76     99.19      0.10
"withVisible"                                373.36     99.09      0.00
"renv_bootstrap_load"                        373.26     99.06      0.00
"renv::load"                                 373.10     99.02      0.00
"catch"                                      372.86     98.95      0.00
"FUN"                                        372.50     98.86      0.02
"lapply"                                     372.46     98.85      0.02
"renv_load_sandbox"                          372.34     98.82      0.00
"renv_sandbox_activate_impl"                 372.34     98.82      0.00
"renv_sandbox_activate"                      372.34     98.82      0.00
"renv_sandbox_generate"                      372.30     98.81      0.00
"enumerate"                                  371.96     98.72      0.00
"f"                                          371.96     98.72      0.00
"renv_file_link"                             371.94     98.71      0.00
"catchall"                                   369.80     98.14      0.00
"renv_file_copy"                             368.58     97.82      0.00
"renv_file_copy_dir"                         368.18     97.71      0.00
"method"                                     366.52     97.27      0.00
"renv_file_copy_dir_impl"                    366.52     97.27      0.00
"file.copy"                                  359.28     95.35    359.18

In particular, it looks like all the time is being spent trying to prepare the renv sandbox. Given that, a fix for this would be to set:

RENV_CONFIG_SANDBOX_ENABLED = FALSE

within an appropriate startup .Renviron file. For example, you could place this in the file located at:

file.path(R.home("etc"), "Renviron.site")

Or, if just for a single user, then at ~/.Renviron.

[gwd666]

@kevinushey thanks a lot - so I guess then I will bring this later issue to the attention of the renv maintainer; the problem with saving standard projects on Windows Network drives still will remain open until the boost update ...
So I guess after having moved that second part of our discussion to the renv repo I will remove/delete most of that topic from here.
Again THANKS for your help - really appreciated it!
Greetings, Walter
PS: I just noticed that you are also heavily involved with the renv package ... means I can spare me the transfer of this issue there?

[kevinushey]

Yes, I'm sure the maintainer of renv will be happy to help :-) I'll transfer this issue to that repository.

[kevinushey]

I'm realizing we should keep the Boost component of this in the RStudio repository, so I'm going to move this back to RStudio and then create an appropriate issue to track the renv issue here. Sorry for the confusion.

[gwd666]

Thanks a lot - will keep my eyes on the boost and the renv one; sorry for the mess ie lack of clear distinction between these two issues, but they somehow came up close after one another and I wasn't sure (in the beginning at least) if they arent't maybe related. I will definitely NOT argue on how you best split the issue(s) and keep track of them.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, per https://github.com/rstudio/rstudio/wiki/Issue-Grooming. Thank you for your contributions.

[gwd666]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, per https://github.com/rstudio/rstudio/wiki/Issue-Grooming. Thank you for your contributions.

Hope the fix w boost update still stands

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs, per https://github.com/rstudio/rstudio/wiki/Issue-Grooming. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.

[mikebessuille]

Putting in Spotted Wakerobin in the hope we can update Boost and resolve some of these problems, as this is a fairly common occurrence. Sorry it has taken so long to diagnose and resolve.

[mikebessuille]

See also the internal document to create a support article: https://docs.google.com/document/d/1Usih1DbYvE7qtRbTqeqtcisgSut5UZfNtL8pyrIif6o/edit#

[jgutman]

Hi @gwd666 -- Boost was upgraded to 1.78 in February, in the latest dailies for the upcoming RStudio release. You can download the latest version for your platform here: https://dailies.rstudio.com/

Would you mind installing that and letting us know if the Boost upgrade did indeed fix the issue you were having with connecting to the R session from your Windows network drive? It's difficult for us to reproduce that network environment, but we have indeed upgraded the Boost bundled with the latest RStudio builds.

[gwd666]

Would you mind installing that and letting us know if the Boost upgrade did indeed fix the issue you were having with connecting to the R session from your Windows network drive?

@jgutman I will try to test it this week; currently a bit busy and give feedback end of this week or next Monday the latest.

[gwd666]

@jgutman - first shot looks promising
Couple of things I noticed already

1. when setting up the project the info widget (about initiliazing the project etc) is somehow hidden - despite the git info widget being visible (in case I also tick the use git box)
   

2. it does take a bit longer than what I am used to ie compared to setting up a project on the local (C:) drive
   but this could be VPN (homeoffice) related - I will additionally check onsite until Friday

3. the initial startup of the newly installed daily RStudio app took quite some time (around a minute) to start initially; later startup procedure is also much slower (not as slow as the initial one but still) than for eg.
   RStudio 2021.09.1+372 "Ghost Orchid" Release (8b9ced1, 2021-11-08) for Windows
   Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.8 Chrome/69.0.3497.128 Safari/537.36
   Means couple of seconds on 2021.09 compared to 30-40 secs on 2022.06 daily until it is fully up and running.

4. Ticking the "Restore most recent project at startup" box also increases the startup time to almost a minute (will also check if this maybe is not as long when onsite)
   

Otherwise the project creation on WIN network shares seems to no longer throw the above error!
I will try to include some more info after doing some onsite/onprem tests - mainly regarding if the startup delays are less significant there.

Thanks for the effort and fix of this issue.

[jgutman]

Going to close this out as the Boost upgrade has been completed and we have some initial verification of results from the user. The startup delays I think should be unrelated -- there are a lot of other changes you've likely pulled in too, but if they persist and are really problematic you can always file a separate issue for that. You're welcome!