New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recursive symlink significantly slows collection #624

Open
pytestbot opened this Issue Oct 29, 2014 · 4 comments

Comments

Projects
None yet
3 participants
@pytestbot

pytestbot commented Oct 29, 2014

Originally reported by: Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns)


placing "ln -s . link" in a collected dir significantly slows down collection. it seems collection doesn't keep track of directories visited to prevent unnecessary recursion?


@pytestbot

This comment has been minimized.

Show comment
Hide comment
@pytestbot

pytestbot Oct 29, 2014

Original comment by Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns):


platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4

pytestbot commented Oct 29, 2014

Original comment by Brian Kearns (BitBucket: bdkearns, GitHub: bdkearns):


platform linux2 -- Python 2.7.6 -- py-1.4.26 -- pytest-2.6.4
platform linux -- Python 3.4.0 -- py-1.4.26 -- pytest-2.6.4
@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Jul 8, 2018

Member

I actually found a much more amusing result when attempting to reproduce this:

mkdir t
echo 'def test_one(): pass' > t/test_foo.py
cd t && ln -s . l

and then

$ ./venv/bin/pytest t
========================================= test session starts =========================================
platform linux -- Python 3.6.5, pytest-3.6.4.dev9+g42bbb4fa.d20180707, py-1.5.4, pluggy-0.6.0
rootdir: /tmp/pytest, inifile: tox.ini
collected 41 items                                                                                    

t/test_foo.py .                                                                                 [  2%]
t/l/test_foo.py .                                                                               [  4%]
t/l/l/test_foo.py .                                                                             [  7%]
t/l/l/l/test_foo.py .                                                                           [  9%]
t/l/l/l/l/test_foo.py .                                                                         [ 12%]
t/l/l/l/l/l/test_foo.py .                                                                       [ 14%]
t/l/l/l/l/l/l/test_foo.py .                                                                     [ 17%]
t/l/l/l/l/l/l/l/test_foo.py .                                                                   [ 19%]
t/l/l/l/l/l/l/l/l/test_foo.py .                                                                 [ 21%]
t/l/l/l/l/l/l/l/l/l/test_foo.py .                                                               [ 24%]
t/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                             [ 26%]
t/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                           [ 29%]
t/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                         [ 31%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                       [ 34%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                     [ 36%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                   [ 39%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                 [ 41%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                               [ 43%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                             [ 46%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                           [ 48%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                         [ 51%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                       [ 53%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                     [ 56%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                   [ 58%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                 [ 60%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                               [ 63%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                             [ 65%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                           [ 68%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                         [ 70%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                       [ 73%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                     [ 75%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                   [ 78%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                 [ 80%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .               [ 82%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .             [ 85%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .           [ 87%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .         [ 90%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .       [ 92%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .     [ 95%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .   [ 97%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py . [100%]

====================================== 41 passed in 0.22 seconds ======================================

I'm amused it stops after recusing 40 levels without erroring -- I suspect it hits ELOOP (Too many levels of symbolic links) and then stops recursing

EDIT: for best effect, I suggest making the terminal wider so you get a nice pyramid

Member

asottile commented Jul 8, 2018

I actually found a much more amusing result when attempting to reproduce this:

mkdir t
echo 'def test_one(): pass' > t/test_foo.py
cd t && ln -s . l

and then

$ ./venv/bin/pytest t
========================================= test session starts =========================================
platform linux -- Python 3.6.5, pytest-3.6.4.dev9+g42bbb4fa.d20180707, py-1.5.4, pluggy-0.6.0
rootdir: /tmp/pytest, inifile: tox.ini
collected 41 items                                                                                    

t/test_foo.py .                                                                                 [  2%]
t/l/test_foo.py .                                                                               [  4%]
t/l/l/test_foo.py .                                                                             [  7%]
t/l/l/l/test_foo.py .                                                                           [  9%]
t/l/l/l/l/test_foo.py .                                                                         [ 12%]
t/l/l/l/l/l/test_foo.py .                                                                       [ 14%]
t/l/l/l/l/l/l/test_foo.py .                                                                     [ 17%]
t/l/l/l/l/l/l/l/test_foo.py .                                                                   [ 19%]
t/l/l/l/l/l/l/l/l/test_foo.py .                                                                 [ 21%]
t/l/l/l/l/l/l/l/l/l/test_foo.py .                                                               [ 24%]
t/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                             [ 26%]
t/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                           [ 29%]
t/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                         [ 31%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                       [ 34%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                     [ 36%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                   [ 39%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                                 [ 41%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                               [ 43%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                             [ 46%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                           [ 48%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                         [ 51%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                       [ 53%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                     [ 56%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                   [ 58%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                                 [ 60%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                               [ 63%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                             [ 65%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                           [ 68%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                         [ 70%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                       [ 73%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                     [ 75%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                   [ 78%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .                 [ 80%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .               [ 82%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .             [ 85%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .           [ 87%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .         [ 90%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .       [ 92%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .     [ 95%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py .   [ 97%]
t/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/l/test_foo.py . [100%]

====================================== 41 passed in 0.22 seconds ======================================

I'm amused it stops after recusing 40 levels without erroring -- I suspect it hits ELOOP (Too many levels of symbolic links) and then stops recursing

EDIT: for best effect, I suggest making the terminal wider so you get a nice pyramid

@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Jul 8, 2018

Member

oh right, and of course this becomes a fun exponential problem if there's more than one symlink

Member

asottile commented Jul 8, 2018

oh right, and of course this becomes a fun exponential problem if there's more than one symlink

@asottile

This comment has been minimized.

Show comment
Hide comment
@asottile

asottile Jul 8, 2018

Member

apparently attempting to discover 2^41 - 2 tests makes my computer sad :)

Member

asottile commented Jul 8, 2018

apparently attempting to discover 2^41 - 2 tests makes my computer sad :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment