Recursive or looping symlinks can cause language server to repeatedly analyze files

[DonJayamanne]

Original issue can be found here along with the details microsoft/vscode-python#2613

I commented previously on #2297 but I found that symlinks were still causing problems after I grabbed a recent build today.

---

Environment data

• VS Code version: 1.27.2
• Extension version (available under the Extensions sidebar): 2018.9.0-alpha, retrieved at Sep 18, 10:30AM PDT.
• OS and version: Mac OS 10.13.6
• Python version (& distribution if applicable, e.g. Anaconda): 2.7.14
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): N/A
• Relevant/affected Python packages and their versions: N/A

Actual behavior

The Python language server gets caught in a super deep nest of recursive symlinks. The repro case given completes quickly (there's nothing to analyze) but still demonstrates the problem, I think.

If there is actual code to parse/analyze it pretty much goes on forever. "Analyzing workspace, #### items remaining" goes down and then quickly shoots up, and this repeats over and over.

I can workaround this by adding the offending symlinks/folder structures to a VSCode exclusion list.

Expected behavior

The Python language server should be able to handle these structures in some reasonable way.

Steps to reproduce:

1. Create a folder structure like this

/code
  /test
    /some
      test.py
      /folder
        /symlink -> ../../some
        __init__.py (empty)
        lib.py (empty)

2. test.py only needs to contain import folder.lib.
3. Enable trace output for the language server and watch the logs, it gets progressively deeper and deeper into a repeated folder/symlink/folder/symlink/ folder structure.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Initializing for /usr/local/opt/python/bin/python2.7
Loading files from /Users/username/code/test/some
Parsing document file:///Users/username/code/test/some/test.py
Parsing document file:///Users/username/code/test/some/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/__init__.py
Parse complete for file:///Users/username/code/test/some/test.py at version -1
Parse complete for file:///Users/username/code/test/some/folder/__init__.py at version -1
Analysis queued for file:///Users/username/code/test/some/test.py
Analysis queued for file:///Users/username/code/test/some/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/lib.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/test.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/test.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/lib.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/lib.py
Parse complete for file:///Users/username/code/test/some/folder/lib.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/test.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/__init__.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/__init__.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/symlink/test.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/__init__.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/test.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/lib.py
Parsing document file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/__init__.py
Parse complete for file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/lib.py at version -1
Analysis queued for file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/lib.py
<more>
Waiting for parsing to complete
Parsing complete. Waiting for analysis entries to enqueue
Enqueue complete. Waiting for analysis to complete
Received new analysis for file:///Users/username/code/test/some/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/folder/symlink/test.py

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

---

Symlinks to a common folder (e.g. two projects which have a symlink to the same folder) also get re-analyzed but that's usually okay as analysis is pretty fast in general.

[MikhailArkhipov]

Yeah, I might have solution

[MikhailArkhipov]

Most probably no longer applies, so for verification with #432

[AlexanderSher]

If we create correct PathResolverSnapshot, there should be no problem.

[jakebailey]

Tested using this specific example, the new LS does not loop infinitely. There are some other symlink oddities (long import hints, loss of import completion once a symlink is imported), but I'll file separate issues for those.

[jakebailey]

Reopening, there may be some other looping issues.

[astrozot]

I actually still experiment this issue with the latest VS Code version (1.32.3).

[Battleman]

From my side, I seem to have problems in looping too (characteristic of the "Analyzing in background, X items left"). But in my side, it's not the file/directory/workspace structure, but more the virtual environment. With a simple file (no import, one function), and using a conda env from another project I got absolutely no problem.
But when I import certain packages (I noticed sklearn poses problems), LS tries to analyses this and get in trouble. But for example, numpy (also "complex" package) poses no problem at all.

STRANGER YET, when going with a blank conda env, with only numpy and sklearn, the same import leads to no problem at all. So it's linked to packages in environment, but there also seems to be a third party factor.

Problematic case:

Environment

• VS code 1.32.3,
• Python 3.7.3 (with anaconda 3)
• Using conda env
• On Debian 10
  [edit] and extension version : 2019.3.6352 [/edit]

Description of the problem

As many others, with certain workspaces, the "Analyzing in background, X items left" appears, grows to a certain point, sometimes decreases and grows again, until being stuck indefinitely. Sucks up CPU and memory usage.

"Simple file" in question:

from sklearn.model_selection import KFold

def main():
    print("This python is not buggy")
    return True

if __name__ == "__main__":
    main()

Content of the problematic conda env (sorry it's not minimal, I didn't find yet a minimal case)

_tflow_select             2.3.0                       mkl  
absl-py                   0.7.0                    py36_0  
alembic                   1.0.6                    py36_0  
apscheduler               3.5.3                 py36_1000    conda-forge
asn1crypto                0.24.0                   py36_0  
astor                     0.7.1                    py36_0  
astroid                   2.1.0                    py36_0  
autopep8                  1.4.3                    py36_0  
backcall                  0.1.0                    py36_0  
beautifulsoup4            4.7.1                    py36_1  
blas                      1.0                         mkl  
bleach                    3.1.0                    py36_0  
c-ares                    1.15.0               h7b6447c_1  
ca-certificates           2018.12.5                     0  
certifi                   2018.11.29               py36_0  
cffi                      1.11.5           py36he75722e_1  
chardet                   3.0.4                    py36_1  
click                     7.0                      py36_0  
cloudpickle               0.6.1                    py36_0  
cryptography              2.4.2            py36h1ba5d50_0  
cycler                    0.10.0                   py36_0  
cytoolz                   0.9.0.1          py36h14c3975_1  
dask-core                 1.0.0                    py36_0  
dbus                      1.13.6               h746ee38_0  
decorator                 4.3.0                    py36_0  
entrypoints               0.3                      py36_0  
expat                     2.2.6                he6710b0_0  
flask                     1.0.2                    py36_1  
flask-migrate             2.2.1                 py36_1000    conda-forge
flask-sqlalchemy          2.3.2                    py36_0  
fontconfig                2.13.0               h9420a91_0  
freetype                  2.9.1                h8a8886c_1  
gast                      0.2.2                    py36_0  
glib                      2.56.2               hd408876_0  
gmp                       6.1.2                h6c8ec71_1  
grpcio                    1.16.1           py36hf8bcb03_1  
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.9.0            py36h7918eee_0  
hdf5                      1.10.4               hb1b8bf9_0  
icu                       58.2                 h9c2bf20_1  
idna                      2.8                      py36_0  
imageio                   2.4.1                    py36_0  
intel-openmp              2019.1                      144  
ipykernel                 5.1.0            py36h39e3cac_0  
ipython                   7.2.0            py36h39e3cac_0  
ipython_genutils          0.2.0                    py36_0  
ipywidgets                7.4.2                    py36_0  
isort                     4.3.4                    py36_0  
itsdangerous              1.1.0                    py36_0  
jedi                      0.13.2                   py36_0  
jinja2                    2.10                     py36_0  
jpeg                      9b                   h024ee3a_2  
jsonschema                2.6.0                    py36_0  
jupyter                   1.0.0                    py36_7  
jupyter_client            5.2.4                    py36_0  
jupyter_console           6.0.0                    py36_0  
jupyter_core              4.4.0                    py36_0  
keras-applications        1.0.6                    py36_0  
keras-preprocessing       1.0.5                    py36_0  
kiwisolver                1.0.1            py36hf484d3e_0  
lazy-object-proxy         1.3.1            py36h14c3975_2  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc                    7.2.0                h69d50b8_2  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libpng                    1.6.36               hbc83047_0  
libprotobuf               3.6.1                hd408876_0  
libsodium                 1.0.16               h1bed415_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
libtiff                   4.0.10            h2733197_1001  
libuuid                   1.0.3                h1bed415_2  
libxcb                    1.13                 h1bed415_1  
libxml2                   2.9.9                he19cac6_0  
mako                      1.0.7                    pypi_0    pypi
markdown                  3.0.1                    py36_0  
markupsafe                1.1.0            py36h7b6447c_0  
matplotlib                3.0.2            py36h5429711_0  
mccabe                    0.6.1                    py36_1  
mistune                   0.8.4            py36h7b6447c_0  
mkl                       2019.1                      144  
mkl_fft                   1.0.10           py36ha843d7b_0  
mkl_random                1.0.2            py36hd81dba3_0  
nbconvert                 5.3.1                    py36_0  
nbformat                  4.4.0                    py36_0  
ncurses                   6.1                  he6710b0_1  
networkx                  2.2                      py36_1  
notebook                  5.7.4                    py36_0  
numpy                     1.15.4           py36h7e9f1db_0  
numpy-base                1.15.4           py36hde5b4d6_0  
olefile                   0.46                     py36_0  
openssl                   1.1.1a               h7b6447c_0  
pandas                    0.23.4           py36h04863e7_0  
pandoc                    2.2.3.2                       0  
pandocfilters             1.4.2                    py36_1  
parso                     0.3.1                    py36_0  
pcre                      8.42                 h439df22_0  
pexpect                   4.6.0                    py36_0  
pickleshare               0.7.5                    py36_0  
pillow                    5.4.1            py36h34e0f95_0  
pip                       18.0                     pypi_0    pypi
prometheus_client         0.5.0                    py36_0  
prompt_toolkit            2.0.7                    py36_0  
protobuf                  3.6.1            py36he6710b0_0  
ptyprocess                0.6.0                    py36_0  
pycodestyle               2.4.0                    py36_0  
pycparser                 2.19                     py36_0  
pygments                  2.3.1                    py36_0  
pylint                    2.2.2                    py36_0  
pyopenssl                 18.0.0                   py36_0  
pyparsing                 2.3.1                    py36_0  
pyqt                      5.9.2            py36h05f1152_2  
pysocks                   1.6.8                    py36_0  
python                    3.6.8                h0371630_0  
python-dateutil           2.7.5                    py36_0  
python-editor             1.0.3                    py36_0  
pytz                      2018.9                   py36_0  
pywavelets                1.0.1            py36hdd07704_0  
pyyaml                    3.13             py36h14c3975_0  
pyzmq                     17.1.2           py36h14c3975_0  
qt                        5.9.7                h5867ecd_1  
qtconsole                 4.4.3                    py36_0  
readline                  7.0                  h7b6447c_5  
requests                  2.21.0                   py36_0  
scikit-image              0.14.1           py36he6710b0_0  
scikit-learn              0.20.2           py36hd81dba3_0  
scipy                     1.1.0            py36h7c811a0_2  
send2trash                1.5.0                    py36_0  
setuptools                40.6.3                   py36_0  
sip                       4.19.8           py36hf484d3e_0  
six                       1.12.0                   py36_0  
soupsieve                 1.7.1                    py36_0  
sqlalchemy                1.2.16           py36h7b6447c_0  
sqlite                    3.26.0               h7b6447c_0  
tensorboard               1.12.2           py36he6710b0_0  
tensorflow                1.12.0          mkl_py36h69b6ba0_0  
tensorflow-base           1.12.0          mkl_py36h3c3e929_0  
termcolor                 1.1.0                    py36_1  
terminado                 0.8.1                    py36_1  
testpath                  0.4.2                    py36_0  
tk                        8.6.8                hbc83047_0  
toolz                     0.9.0                    py36_0  
tornado                   5.1.1            py36h7b6447c_0  
traitlets                 4.3.2                    py36_0  
typed-ast                 1.1.0            py36h14c3975_0  
tzlocal                   1.5.1                    py36_0  
urllib3                   1.24.1                   py36_0  
wcwidth                   0.1.7                    py36_0  
webencodings              0.5.1                    py36_1  
werkzeug                  0.14.1                   py36_0  
wheel                     0.32.3                   py36_0  
widgetsnbextension        3.4.2                    py36_0  
wrapt                     1.11.0           py36h7b6447c_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zeromq                    4.2.5                hf484d3e_1  
zlib                      1.2.11               h7b6447c_3  

Extract of the python output

Analysis of IPython.core.magics.history(Library) queued
Analysis of scipy.optimize._slsqp(Compiled) completed in 47.8488 ms.
Analysis of sklearn.externals.joblib.pool(Library) queued
Analysis of _locale(CompiledBuiltin) queued
Analysis of sre_compile(Library) queued
Analysis of distutils.command.install(Stub) completed in 48.1344 ms.
Analysis of nbformat.v2.nbbase(Library) queued
Analysis of IPython.core.magics.extension(Library) queued
Analysis of IPython.utils.terminal(Library) queued
Analysis of scipy.optimize.moduleTNC(Compiled) completed in 39.3443 ms.
Analysis of nbformat.v1.convert(Library) queued
Analysis of IPython.utils.dir2(Library) queued
Analysis of numpy.testing.noseclasses(Library) queued
Analysis of email.parser(Library) queued
Analysis of IPython.utils.process(Library) queued
Analysis of nbformat.v1.nbbase(Library) queued
Analysis of nbformat.v2.nbpy(Library) queued
Analysis of prompt_toolkit.filters.app(Library) queued
Analysis of notebook.services.contents.fileio(Library) queued
Analysis of sre_parse(Stub) queued
Analysis of prompt_toolkit.win32_types(Library) queued
Analysis of pygments.formatters.latex(Library) queued
Analysis of prompt_toolkit.renderer(Library) queued
Analysis of tornado.process(Library) queued
Analysis of asyncio.transports(Stub) queued
Analysis of asyncio.streams(Library) queued
Analysis of nbformat.v1.nbjson(Library) queued
Analysis of sklearn.externals.joblib.externals.loky.process_executor(Library) queued
Analysis of dummy_threading(Library) queued
Analysis of sre_parse(Library) queued
Analysis of IPython.lib.display(Library) queued
Analysis of prompt_toolkit.eventloop.context(Library) queued
Import:  ipython_genutils.ipstruct /storage/anaconda3/envs/facesearch/lib/python3.6/site-packages/ipython_genutils/ipstruct.py 
Analysis of distutils.log(Stub) completed in 74.345 ms.
Analysis of ipython_genutils.ipstruct(Library) queued
Analysis of prompt_toolkit.enums(Library) queued
Analysis of IPython.core.magics.execution(Library) queued
Analysis of IPython.core.page(Library) queued
Analysis of nbformat.v3(Library) queued
....
...
Analysis of _heapq(Compiled) completed in 16.2774 ms.
Analysis of scipy.interpolate._bsplines(Library) completed in 17.8444 ms.
Analysis of colorsys(Library) completed in 0.6466 ms.
Analysis of IPython.core.inputtransformer2(Library) queued
Analysis of unittest(Library) completed in 31.8242 ms.
Analysis of numpy.version(Library) completed in 0.1255 ms.
Analysis of sklearn.decomposition._online_lda(Compiled) completed in 0.3446 ms.
Analysis of scipy.interpolate.interpnd(Compiled) completed in 1.2661 ms.
Analysis of scipy.interpolate._ppoly(Compiled) completed in 9.8247 ms.
Analysis of numpy.core._multiarray_tests(Compiled) completed in 0.9455 ms.
Analysis of scipy.interpolate.dfitpack(Compiled) completed in 0.3231 ms.
Analysis of multiprocessing(Library) completed in 0.1187 ms.
Analysis of sklearn.externals.joblib.externals.loky.backend._win_wait(Library) queued
Analysis of ipykernel.kernelapp(Library) queued
Analysis of multiprocessing.pool(Stub) completed in 20.9803 ms.
Analysis of multiprocessing.process(Stub) completed in 0.305 ms.
Analysis of numpy.testing._private(Library) completed in 17.188 ms.
Analysis of _multiprocessing(Compiled) completed in 10.1877 ms.
Analysis of _hashlib(Compiled) completed in 0.4461 ms.
Analysis of _sha3(Compiled) completed in 15.738 ms.
Analysis of pycparser(Library) completed in 0.6777 ms.
Analysis of sklearn.externals.joblib.externals.loky.reusable_executor(Library) queued
Analysis of IPython.core.completer(Library) queued
Analysis of IPython.core.extensions(Library) queued
Analysis of _blake2(Compiled) completed in 17.729 ms.
Analysis of _sha512(Compiled) completed in 27.4286 ms.
Analysis of _sha256(Compiled) completed in 1.6649 ms.
Analysis of _md5(Compiled) completed in 3.1989 ms.

SO, some files are queued, while some are completed in a matter of ms.
I guess that is the problem, some recursivity in specific packages. Maybe two (or more) of the packages in the env are referencing each other, creating a loop. Point is, with a conda env having only numpy and sklearn, this issue does not appear.

Sorry if I mis-followed the guidelines...

[jakebailey]

Please file a new issue; this one is closed and is probably not related.

Regarding "only numpy and sklearn", there's a lot behind those libraries which may be affected by things relating to my explanation in #832 (comment).