Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recursive symlink significantly slows collection #624

Open
pytestbot opened this issue Oct 29, 2014 · 5 comments · May be fixed by #7355
Open

recursive symlink significantly slows collection #624

pytestbot opened this issue Oct 29, 2014 · 5 comments · May be fixed by #7355

Comments

@pytestbot
Copy link
Contributor

@pytestbot pytestbot commented Oct 29, 2014

Originally reported by: Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns)


placing "ln -s . link" in a collected dir significantly slows down collection. it seems collection doesn't keep track of directories visited to prevent unnecessary recursion?


@pytestbot
Copy link
Contributor Author

@pytestbot pytestbot commented Oct 29, 2014

Original comment by Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns):


platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
@asottile
Copy link
Member

@asottile asottile commented Jul 8, 2018

I actually found a much more amusing result when attempting to reproduce this:

mkdir t
echo 'def test_one(): pass' > t/test_foo.py
cd t && ln -s . l

and then

$ ./venv/bin/pytest t
========================================= test session starts =========================================
platform linux -- Python 3.6.5, pytest-3.6.4.dev9+g42bbb4fa.d20180707, py-1.5.4, pluggy-0.6.0
rootdir: /tmp/pytest, inifile: tox.ini
collected 41 items                                                                                    

t/test_foo.py .                                                                                 [  2%]
t/l/test_foo.py .                                                                               [  4%]
t/l/l/test_foo.py .                                                                             [  7%]
t/l/l/l/test_foo.py .                                                                           [  9%]
t/l/l/l/l/test_foo.py .                                                                         [ 12%]
t/l/l/l/l/l/test_foo.py .                                                                       [ 14%]
t/l/l/l/l/l/l/test_foo.py .                                                                     [ 17%]
t/l/l/l/l/l/l/l/test_foo.py .                                                                   [ 19%]
t/l/l/l/l/l/l/l/l/test_foo.py .                                                                 [ 21%]
t/l/l/l/l/l/l/l/l/l/test_foo.py .                                                               [ 24%]
t/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                             [ 26%]
t/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                           [ 29%]
t/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                         [ 31%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                       [ 34%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                     [ 36%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                   [ 39%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                 [ 41%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                               [ 43%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                             [ 46%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                           [ 48%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                         [ 51%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                       [ 53%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                     [ 56%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                   [ 58%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                 [ 60%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                               [ 63%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                             [ 65%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                           [ 68%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                         [ 70%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                       [ 73%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                     [ 75%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                   [ 78%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                 [ 80%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .               [ 82%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .             [ 85%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .           [ 87%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .         [ 90%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .       [ 92%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .     [ 95%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .   [ 97%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py . [100%]

====================================== 41 passed in 0.22 seconds ======================================

I'm amused it stops after recusing 40 levels without erroring -- I suspect it hits ELOOP (Too many levels of symbolic links) and then stops recursing

EDIT: for best effect, I suggest making the terminal wider so you get a nice pyramid

@asottile
Copy link
Member

@asottile asottile commented Jul 8, 2018

oh right, and of course this becomes a fun exponential problem if there's more than one symlink

@asottile
Copy link
Member

@asottile asottile commented Jul 8, 2018

apparently attempting to discover 2^41 - 2 tests makes my computer sad :)

wjt added a commit to wjt/pytest that referenced this issue Oct 19, 2018
There are two parts to the fix:

* Don't visit directories which have already been visited. This fixes
  the exponential aspect of the bug.
* Don't visit files which have already been visited. This fixes the more
  minor problem that, without this additional change, test_noop would be
  run twice, once as test_noop.py and once as symlink-0/test_noop.py.

Fixes pytest-dev#624
wjt added a commit to wjt/pytest that referenced this issue Oct 19, 2018
This fixes trying to traverse exponentially many paths in the presence
of symlink loops, and trying to run any tests discovered in that tree
exponentially many times if collecting ever finishes.

I wanted to also prevent visiting files more than once, but my first
attempt broke --keep-duplicates.

Fixes pytest-dev#624
@codingkrabbe
Copy link

@codingkrabbe codingkrabbe commented Mar 20, 2020

had a similar issue, a recursive symlink was followed until it crashed...
Although strange that you have a different behaviour.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.