anaconda mode lost rpc connection after some time

[ivonwrk]

Hi.
I'm constantly getting the same error: after some time (tens of minutes),
anaconda mode could not get the response. If I kill the buffer anaconda-mode,
and after that all works fine.

Below is status of buffers after failure:
anaconda-mode
anaconda_mode port 36909
127.0.0.1 - - [05/Dec/2019 15:01:08] ================================================================================
Traceback (most recent call last):
File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 101, in apply
result = method(**params)
File "", line 98, in wrapper
result = f(jedi.Script(source, line, column, path, environment=virtual_environment))
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/jedi/api/init.py", line 118, in init
cache_path=settings.cache_directory,
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/jedi/evaluate/init.py", line 388, in parse_and_get_code
return self.grammar.parse(code=code, path=path, **kwargs), code
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/grammar.py", line 76, in parse
return self._parse(code=code, **kwargs)
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/grammar.py", line 126, in _parse
new_lines=lines
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/python/diff.py", line 152, in update
self._copy_from_old_parser(line_offset, i2, j2)
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/python/diff.py", line 211, in copy_from_old_parser
from = copied_nodes[0].get_start_pos_of_prefix()[0] + line_offset
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/tree.py", line 264, in get_start_pos_of_prefix
return self.children[0].get_start_pos_of_prefix()
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/python/tree.py", line 125, in get_start_pos_of_prefix
previous_leaf = self.get_previous_leaf()
File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/tree.py", line 74, in get_previous_leaf
i = c.index(node)
ValueError: <Function: create_dirs@1168-1179> is not in list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/providers/basehttp.py", line 40, in do_POST
status, response = self.server.service(data)
File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 50, in call
result = self.apply(method, args)
File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 105, in apply
server_error(args['id'], error)
File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/errors.py", line 91, in server_error
raise ServiceException(500, dumps(response))
service_factory.exceptions.ServiceException: (500, '{"jsonrpc": "2.0", "id": 1, "error": {"code": -32000, "message": "Server error", "data": "ValueError('<Function: create_dirs@1168-1179> is not in list')"}}')

[TerryTsao]

I'm getting very similar errors recently

[CeleritasCelery]

Can you see if reverting to commit 3ede03d fixes the issue?

cc @falloutphil

[falloutphil]

Hi,

I'm the author of 1bc301b which was recently reintroduced.

I've just tested doing method look-ups on TRAMP over an approx 25 min period on a version containing 1bc301b, but without other recent changes.

Things seemed fine, but my use-case and setup could be different to others. So I'll add a bit more explanation.

My commit does 3 things - all of which are to do with the underlying creation of an ssh tunnel as a separate process for RPC calls to travel through:

• Most importantly - fixes a bug that means that the original unsecured RPC connection created which completely ignores that ssh tunnel is replaced with a secured RPC connection through the ssh tunnel. By coincidence the original code would work in some cases, but not by design and by passing cleartext over the web.
• Secures tunnel creation by invoking tunnel on remote localhost, rather than pythonic-remote-host, this is important to make sure only the creator can access the tunnel, rather than third parties.
• Introduces an optional "-J" jump proxy switch, so multihop TRAMP connections are seamlessly supported.

If you're not using TRAMP then it's hard to see how my commit would have any effect.

If you are using TRAMP, I have a theory!

• You're one of the users where the old code worked by chance, and it was simply ignoring the ssh tunnel it was meant to be using!
• But now that I've fixed the code the ssh tunnel is being used.
• Some ssh servers have a timeout, so idle tunnels could be disconnected after a while.

This scenario would match the reported symptoms, so although I'm making a pretty wild guess, it sounds plausible to me.

So if my theory is correct - all you need to do is add a keep alive to your ~/.ssh/config file - Google will help here, but something like the below should work:

Host *
     ServerAliveInterval 300
     ServerAliveCountMax 2

Note I don't want to make the keep alive part of anaconda-mode itself as this is too intrusive. The end user should configure this as required.

Also note as I've mentioned previously you also need to make sure that each remote host and proxy jump used must have been previously added to the known_hosts file. Although this isn't the issue in the case explained as nothing would work for any time period if this was the issue.

If this doesn't help, then please can you reply with more details of exactly how to recreate the problem - some helpful questions:

• Are you opening a remote python file using TRAMP?
• Does the situation exist if you only use anaconda-mode on local python files?
• If using TRAMP, are you proxying the connection via a third or fourth server or connecting directly from your local emacs to remote server?
• You're then editing a file, but after some time Anaconda functions are not working (could you give an example of something that works and then fails)?
• Are you editing files on more that one remote server at the same time?
• Can you post the contents of both anaconda-mode and anaconda-ssh buffers
• What versions of python, jedi, and service-factory do you have installed locally/remotely.
• What does "ps -ef | grep ssh" return on the local server?
• Are you using a virtual environment, and if yes, is the situation exist if you don't use a virtual environment?

Sorry for the long message - hopefully it helps!

[CeleritasCelery]

This line does not seem related to tramp. Why was it changed?

1bc301b#diff-d4bac877ea8ba1b9051ada5ac9a964baR502

[falloutphil]

This line does not seem related to tramp. Why was it changed?

1bc301b#diff-d4bac877ea8ba1b9051ada5ac9a964baR502

After creating the local tunnel you want the RPC requests to go down the tunnel, not direct to the host. The original code was completely disregarding the tunnel it had created, which I believe was a bug - it was working sometimes by fluke (when the randomly generated RPC port happened to be open on both servers), but was sending data as cleartext, which is a bad idea.

This change ensures RPC requests go down the tunnel (on port 22) and pops-out on the server where the TRAMP file is hosted. This avoids issues with firewalls blocking direct RPC requests as well being secure.

[CeleritasCelery]

Closed with #379