company object completion not working on remotes

[solmos]

While runing R on a remote server using TRAMP the following error appears in a new help[R] buffer every time I try to open the help file on some function:

+ Error in base::as.environment("ESSR") : 
  no item called "ESSR" on the search list

The problem just started after updating to ESS 20200905.822 and only when using TRAMP.

The error appears whether I type ?function_name or I call ess-display-help-on-object with my keybinding.

I am using Spacemacs.

[lionel-]

This is from #1044. I'll have to look how ESS works with Tramp. I'm surprised it can work at all without an ESSR environment. We also need to think about some Tramp tests.

[lionel-]

I see that we use a different loading approach for remote connections. Instead of calling .ess.load.ESSR(), we directly source all the files in the REPL and the ESSR utils get defined in the global environment.

A cleaner approach might be to transfer all the ESSR files into a temp directory and then call .ess.load.ESSR() on it. This would make remote and local loading consistent, would fix edge cases such as rm(list = ls(all.names = TRUE)) deleting all the ESSR utils, and we can use as.environment("ESSR") for encapsulated evaluation.

[dapritchard]

I would be willing to take a stab at implementing the approach the @lionel- proposed above as long as I wouldn't be getting in anyone's way by doing so.

[lionel-]

I already did some work on that but haven't had time to finish yet. Also I'm not sure the approach will work out in the end. I was looking into sending text data directly to remote files using this trick:

lines <- readLines(stdin())
writeLines(text = lines, file('lines.txt'))

With an EOF to signal end of transfer.

If you'd like to work on a simpler approach you're very welcome @dapritchard!

[vspinu]

We can also just source into an environment and attach. The end result just like on local machiens.

[dapritchard]

Greetings @vspinu, I'm afraid it's not clear to me what the specifics are for your suggestion. Currently, what you've written above is what's done in the local process case: each of the files in etc/ESSR/R are sourced into the ESSR R environment, and then the environment is attached to the search path.

The question for the remote process case then is how to get that source code from the local machine to the remote process. Currently what's implemented in the WIP #1107 patch is that the source files are first copied over from the local machine to a temporary directory on the remote machine, and then the sourcing + attaching occurs the same as for the local process case.

As an aside, I suppose now is as good a time as any to offer my deepest thanks to you both for your contributions to ESS and other R/Emacs projects that I use on a daily basis.

[vspinu]

@dapritchard I was addressing @lionel-'s concern regarding sourcing of ESSR files into the R global env on M-x ess-remote.

I would suggest we first dig what is the route of your problem. There is a reason why ESSR is not loaded on your remote machine. Current implementation is supper efficient and simple. The only drawback is that it needs an internet connection to github. One can also copy the ESSR.rds manually in rare cases when github is not accessible.

[vspinu]

@dapritchard you should be seeing an error message Couldn't load ESSR.rds. .... with the explanation of what happened.

[lionel-]

@vspinu How do you source into a remote? I couldn't find a way so I started working on sending the files through the process connection (super experimental approach, I'm not saying we should pursue it).

Regarding adding an ESSR environment also on remotes, I think it's necessary not only to make remotes and local connections consistent, but also I would like to add .ess.evalq() to ESSR that would evaluate its argument inside the ESSR environment. This way ESS commands are not dependent on the contents of the global env. In order to achieve that, we need an ESSR env on remotes as well.

[dapritchard]

In case some general clarification is helpful, there are three cases in the current implementation.

1. The process is on the local machine, in which case the contents of /etc/ESSR/R are sourced into the ESSR environment.
2. The process is on a remote machine and some combination of ess-r-fetch-ESSR-on-remotes/ess-remote are non-nil, in which case ESSR.rds (containing the compiled form of the contents of /etc/ESSR/R as R objects) is downloaded from GitHub, and then the ESSR environment is then created from that.
3. When (1) and (2) are not the case, then the contents of /etc/ESSR/R are streamed into the remote process using ess--inject-code-from-file with the resulting R objects being placed in the global environment (no ESSR environment is created).

I believe that @vspinu may be referring to case (2). I'm trying to address the inconsistency that occurs in case (3) where there is no ESSR environment created in the remote R process.

[solmos]

Autocompletion does not work properly with remote process over TRAMP either. I am not sure if this is related, but I guess it is.

Sorry for not being too specific about these errors but I cannot figure out when they appear and when they do not.

These are some of the errors I get when typing some functions:

ess-command: Timeout during background ESS command ‘local({ r <- .ess_get_completions("rstanarm::", 10, " = "); r[r != '...='] })’

eldoc error: (error Timeout during background ESS command ‘.ess_funargs("tidy")’)

[lionel-]

Have you updated ESS to the latest version?

[solmos]

Never mind these errors, I cannot reproduce them now.

However, here is a problem with autocompletion that I see even after installing the latest version (20210204.856). This does not happen when I run a local R process.

1. Create a simple R script where you create some object (without starting a remote R process):

my_df <- iris

2. Start a remote R process.

3. Evaluate this line of code in the remote process.

4. Start typing in the script, say:

my_other_df <- iris[1:10,]

Completion messages show up in the minibuffer after the first few characters that I type:

Forming completions for .GlobalEnv...done
Forming completions for package:stats...done
Forming completions for package:graphics...done
Forming completions for package:grDevices...done
Forming completions for package:utils...done
Forming completions for package:datasets...done
Forming completions for package:methods...done
Forming completions for Autoloads...done
Forming completions for package:base...done

5. Evaluate this new line in the remote process.

6. Start typing my_ in the script.

Autocompletion shows up as I type but only my_df is shown in the mini box at point.

7. If when typing my_ I press TAB I do get both options for autocompletion in a Helm completion-at-point transient buffer.

What could be the problem? Is it related to the original issue?

[vspinu]

The relevant logic is here.

I don't know why ess-r-fetch-ESSR-on-remotes it nil now by default. I always thought it's t. It surely was the case before #429 that we were downloading stuff. Now both TRAMP and ess-remote ESSR is sourced into the global environment, which is not good. I have not worked on remotes in ages so I have not noticed issues. Let's move it to t.

@vspinu How do you source into a remote? I couldn't find a way so I started working on sending the files through the process connection (super experimental approach, I'm not saying we should pursue it).

The reason why we have a different approach for remotes is because sourcing files through process stdin does not work reliably. I worked on it myself, and a few years back someone else tried (we have an issues somewhere but I cannot find it). The best understanding so far is that Emacs sends new lines for long input which break the code which we sent. But in my experience it's even more than that, some parts of code can be missing.

This is why I was proposing to send RDS by chunks, reconstruct on the other side and eval into ESSR environment.

Anyways, let's make ess-r-fetch-ESSR-on-remotes t by default for now. @solmos could you please try if it works reliably for you?

[dapritchard]

Just a heads up, but on my machine the ESSR fetch from GitHub is failing for me with the following error message.

<simpleError in utils::download.file(url, essr_file): cannot open URL ’https://github.com/emacs-ess/ESS/raw/ESSRv1.6/etc/ESSR.rds’>

[solmos]

Yes, same error for me:

Loading ESSR into remote ...
Couldn’t load ESSR.rds. Injecting from local.
 Error: > > + + + + + + + + + + + + + + + + + + + + + trying URL ’https://github.com/emacs-ess/ESS/raw/ESSRv1.6/etc/ESSR.rds’
<simpleError in utils::download.file(url, essr_file): cannot open URL ’https://github.com/emacs-ess/ESS/raw/ESSRv1.6/etc/ESSR.rds’>
[1] FALSE
Warning message:
In utils::download.file(url, essr_file) :
  cannot open URL ’https://github.com/emacs-ess/ESS/raw/ESSRv1.6/etc/ESSR.rds’: HTTP status was ’404 Not Found’

[vspinu]

Fixed. There was a missing ESSR tag on github. Could you please try again?

@solmos I can reproduce your error. Something is not quite right with completion on remotes indeed. Renaming the topic.

[solmos]

Now when I start a remote R process Emacs hangs for a few seconds and then the following error message appears:

ess-command: Timeout during background ESS command ‘options(STERM='iESS', str.dendrogram.last="'", editor='emacsclient', show.error.locations=TRUE)’

[dapritchard]

I think there's a problem with ess-r--fetch-ESSR-remote at this line. ess-boolean-command calls ess-command, which in turn relies on the R function .ess.command to be available in the R runtime environment. However, the ESSR functions aren't available yet, which has the result of ess-command timing out.

[dapritchard]

I think I am mistaken in my previous comment about ess-command relying on the R function .ess.command to be available in the R runtime environment.

However, I am getting the same error as @salmos, and it seems to occur due to a failure in ess-r--fetch-ESSR-remote to inject the ESSR environment followed by an attempted call to .ess.command in R through ess-command. Will continue to investigate.

[lionel-]

However, the ESSR functions aren't available yet, which has the result of ess-command timing out.

@dapritchard All the loading code is running behind ess-r--without-format-command which disables .ess.command(), for the reason that you pointed out (can't use it before it's loaded in the process).

Now when I start a remote R process Emacs hangs for a few seconds and then the following error message appears:

@solmos The timeout should be 30 seconds now. Do you see any improvement if you bump the timeout to infinity with (setq ess--command-default-timeout most-positive-fixnum)? 30s seems a rather long response time for setting an option though.

[vspinu]

@dapritchard If there are any errors during the loading of the ESSR, those should be displayed in the minibuffer with the errors. The timeout error means that you are using a week-old emacs.

[dapritchard]

@vspinu You are correct, after catching my branch up to master ess-r--fetch-ESSR-remote works flawlessly for me. Apologies for the oversight on my end.

All the loading code is running behind ess-r--without-format-command which disables .ess.command(), for the reason that you pointed out (can't use it before it's loaded in the process).

This explanation is very helpful, thank you. I hadn't dug deeply enough to understand the mechanism by which that was working.