Workspace changes made outside of editor do not update symbol index (file watching)

[tsmaeder]

Environment data

Name: Python

• VS Code version: 1.35.1
• Extension version (available under the Extensions sidebar): 2019.6.22090
• OS and version: Fedora 30
• Python version (& distribution if applicable, e.g. Anaconda): 2.7.16
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): n/a
• Relevant/affected Python packages and their versions: ?
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): jedi.enabled=false

Expected behaviour

New Projects under the opened folder are found and indexed

Actual behaviour

They don't seem to be.

Steps to reproduce:

1. create a folder "testproject"
2. git clone https://github.com/kubernetes-client/python inside tesproject
3. Open testproject in VS Code
4. Open apiextensions-api.py
5. Open WS symbol “ApiClient”
6. Observe: you find “APIClient” class
7. git clone https://github.com/google/python-fire
8. open completion.py
9. Open WS symbol “Fire”
10. Observe: not found
11. reopen folder
12. Open WS Symbol “Fire”
13. Observe: symbol is found

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

User belongs to experiment group 'AlwaysDisplayTestExplorer - experiment'
Starting Microsoft Python language server.
[Info  - 3:23:50 PM] Analysis cache path: /home/thomas/.cache/Microsoft/Python Language Server
[Info  - 3:23:50 PM] GetCurrentSearchPaths /usr/bin/python3 
[Info  - 3:23:50 PM] Interpreter search paths:
[Info  - 3:23:50 PM]     /usr/lib64/python3.7
[Info  - 3:23:50 PM]     /usr/lib64/python3.7/lib-dynload
[Info  - 3:23:50 PM]     /home/thomas/.local/lib/python3.7/site-packages
[Info  - 3:23:50 PM]     /usr/lib64/python3.7/site-packages
[Info  - 3:23:50 PM]     /usr/lib/python3.7/site-packages
[Info  - 3:23:50 PM] User search paths:
[Info  - 3:23:51 PM] Microsoft Python Language Server version 0.3.20.0
[Info  - 3:23:51 PM] Initializing for /usr/bin/python3

[jakebailey]

I don't think we recognize that the folder has changed and needs to be re-scanned due to new contents. Does reloading VS Code after cloning work?

[tsmaeder]

Yes, it does, see step 11-13 above.

[jakebailey]

Ah, forgive me, I missed that. We don't do any sort of file watching, so I think this sort of thing is to be expected. We only open files referenced by other open files (via imports), and scan the workspace for symbols at startup. Trying to do it for user files will probably need to wait for #1076 to pin down the best way to deal with file watching.

[Jma353]

To be clear, this means that LS cannot (reliably) be used with any sort of version control? Are there plans to add file watching of user code in the future?

Are there ways of initializing the LS at it stands to get better file watching support to keep analysis up to date with code that may have changed outside the editor environment?

[Jma353]

Also, what if a file changes outside of the IDE? Couldn't the LS return out-of-date completion?

[Jma353]

Related question: is there a way to force reanalysis? Is it possible to send LS a command or other incantation to say to it "please rescan user / lib code, it's been updated".

[jakebailey]

File watching is a bit of a hard problem here. We already watch search paths that aren't user code (so pip installing things will trigger a reload, etc), but for files in the workspace, there's an interplay of what the editor presents via its file change events and what we might see on disk. For example, you may delete a file from disk, but VS Code doesn't "close" the file, and is still allowed to send file changes.

There isn't a command to ask the language server to refresh everything, no, but killing the process would at least cause the client to restart things again (or in VSC, use the reload command in the palette). There are other issues internally when it comes to forcing a reload that would need to be fixed up first; our currently live-reloading mechanism is a bit fragile.

[tsmaeder]

For example, you may delete a file from disk, but VS Code doesn't "close" the file, and is still allowed to send file changes.

But wouldn't you get a deleted notification from the file watcher API in this case? Not sure I understand the problem...

[jakebailey]

Yes, but the editor still has the file open, and the language server could be expected to serve requests based on that document. A similar scenario happens when a file is opened and modified, then changed on disk. Whose copy is the one we have to provide information for?

[Jma353]

Yea, can totally understand the problem.

Is this a priority right now for the language server (as in, file-watching, at least with a best-effort approach)? I can see various scenarios (mainly, rebases) that involve changing files outside of what's open in the IDE for people. As it stands right now (to my understanding) the LS is at risk of returning incorrect or stale information for these files if I was to import them into a Python file and attempt to get IntelliSense for them.

[jakebailey]

We're currently focusing on improving the quality of the analysis (typing improvements, fixing import issues, caching results), so this isn't as high of a priority in comparison.

We'd need to go through all of the possible cases to see how things behave before coming up with a plan for an implementation for file watching. Notably, the point I made about changing a file then modifying it on the filesystem may work itself out, depending on how VS Code prompts the user to choose which is which. The LSP also provides file watching mechanisms, which we don't use yet (and I can't say I recall exactly what they do/don't cover).

[tsmaeder]

Yes, but the editor still has the file open, and the language server could be expected to serve requests based on that document. A similar scenario happens when a file is opened and modified, then changed on disk. Whose copy is the one we have to provide information for?

I think the LSP is pretty clear on this question: if the client has a file open, that is the "source of truth" as far as the LS is concerned.