Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recursive symlink significantly slows collection #624

Closed
pytestbot opened this issue Oct 29, 2014 · 7 comments
Closed

recursive symlink significantly slows collection #624

pytestbot opened this issue Oct 29, 2014 · 7 comments
Labels
topic: collection related to the collection phase type: bug problem that needs to be addressed type: performance performance or memory problem/improvement

Comments

@pytestbot
Copy link
Contributor

Originally reported by: Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns)


placing "ln -s . link" in a collected dir significantly slows down collection. it seems collection doesn't keep track of directories visited to prevent unnecessary recursion?


@pytestbot
Copy link
Contributor Author

Original comment by Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns):


platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4

@pytestbot pytestbot added the type: bug problem that needs to be addressed label Jun 15, 2015
@pfctdayelise pfctdayelise added the topic: collection related to the collection phase label Jul 25, 2015
@asottile
Copy link
Member

asottile commented Jul 8, 2018

I actually found a much more amusing result when attempting to reproduce this:

mkdir t
echo 'def test_one(): pass' > t/test_foo.py
cd t && ln -s . l

and then

$ ./venv/bin/pytest t
========================================= test session starts =========================================
platform linux -- Python 3.6.5, pytest-3.6.4.dev9+g42bbb4fa.d20180707, py-1.5.4, pluggy-0.6.0
rootdir: /tmp/pytest, inifile: tox.ini
collected 41 items                                                                                    

t/test_foo.py .                                                                                 [  2%]
t/l/test_foo.py .                                                                               [  4%]
t/l/l/test_foo.py .                                                                             [  7%]
t/l/l/l/test_foo.py .                                                                           [  9%]
t/l/l/l/l/test_foo.py .                                                                         [ 12%]
t/l/l/l/l/l/test_foo.py .                                                                       [ 14%]
t/l/l/l/l/l/l/test_foo.py .                                                                     [ 17%]
t/l/l/l/l/l/l/l/test_foo.py .                                                                   [ 19%]
t/l/l/l/l/l/l/l/l/test_foo.py .                                                                 [ 21%]
t/l/l/l/l/l/l/l/l/l/test_foo.py .                                                               [ 24%]
t/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                             [ 26%]
t/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                           [ 29%]
t/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                         [ 31%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                       [ 34%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                     [ 36%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                   [ 39%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                 [ 41%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                               [ 43%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                             [ 46%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                           [ 48%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                         [ 51%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                       [ 53%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                     [ 56%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                   [ 58%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                 [ 60%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                               [ 63%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                             [ 65%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                           [ 68%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                         [ 70%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                       [ 73%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                     [ 75%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                   [ 78%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                 [ 80%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .               [ 82%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .             [ 85%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .           [ 87%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .         [ 90%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .       [ 92%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .     [ 95%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .   [ 97%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py . [100%]

====================================== 41 passed in 0.22 seconds ======================================

I'm amused it stops after recusing 40 levels without erroring -- I suspect it hits ELOOP (Too many levels of symbolic links) and then stops recursing

EDIT: for best effect, I suggest making the terminal wider so you get a nice pyramid

@asottile
Copy link
Member

asottile commented Jul 8, 2018

oh right, and of course this becomes a fun exponential problem if there's more than one symlink

@asottile
Copy link
Member

asottile commented Jul 8, 2018

apparently attempting to discover 2^41 - 2 tests makes my computer sad :)

wjt added a commit to wjt/pytest that referenced this issue Oct 19, 2018
There are two parts to the fix:

* Don't visit directories which have already been visited. This fixes
  the exponential aspect of the bug.
* Don't visit files which have already been visited. This fixes the more
  minor problem that, without this additional change, test_noop would be
  run twice, once as test_noop.py and once as symlink-0/test_noop.py.

Fixes pytest-dev#624
wjt added a commit to wjt/pytest that referenced this issue Oct 19, 2018
This fixes trying to traverse exponentially many paths in the presence
of symlink loops, and trying to run any tests discovered in that tree
exponentially many times if collecting ever finishes.

I wanted to also prevent visiting files more than once, but my first
attempt broke --keep-duplicates.

Fixes pytest-dev#624
@Zac-HD Zac-HD added the type: performance performance or memory problem/improvement label Jun 24, 2019
@codingkrabbe
Copy link

had a similar issue, a recursive symlink was followed until it crashed...
Although strange that you have a different behaviour.

@mdmintz
Copy link

mdmintz commented Feb 8, 2022

Although the pyramid that @asottile showed still happens in pytest 7.0.0, the improvements to test-collection speed make it such that tests are collected significantly faster. Not so much a performance issue anymore as a duplicated-tests issue.

@RonnyPfannschmidt
Copy link
Member

closing this one as solved then

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: collection related to the collection phase type: bug problem that needs to be addressed type: performance performance or memory problem/improvement
Projects
None yet
7 participants