test_walk fails on Linux

[benhoyt]

======================================================================
FAIL: test_traversal (test_walk.WalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Work\scandir\tests\test_walk.py", line 66, in test_traversal
    self.assertEqual(all[3 - 2 * flipped], sub2_tree)
AssertionError: Tuples differ: ('C:\\Work\\scandir\\tests\\te... != ('C:\\Work\\scandir\\tests\\te...

First differing element 1:
[]
['link']

- ('C:\\Work\\scandir\\tests\\temp\\TEST1\\SUB2', [], ['link', 'tmp3'])
?                                                 ----

+ ('C:\\Work\\scandir\\tests\\temp\\TEST1\\SUB2', ['link'], ['tmp3'])
?                                                        +  +

[Malizor]

Same problem on Ubuntu 13.04.

Also, the test does not work on python2:

ERROR: test_traversal (test_walk.WalkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nicolas/Bureau/scandir/tests/test_walk.py", line 48, in test_traversal
    os.symlink(os.path.abspath(t2_path), link_path, True)
TypeError: symlink() takes exactly 2 arguments (3 given)

[benhoyt]

The original issue doesn't happen anymore, closing. @Malizor, if the issue you brought up still happens on Ubuntu, please open a new issue.

[benhoyt]

Okay, the original issue as described does still happen on Linux, reopening. I'm going to edit the description of this issue to match the current symptoms.

[htoothrot]

What is the status of this issue? As moloney found, it seems the only way around this is to fallback to a symlink-following stat() call when d_type == DT_LNK (in is_dir and is_file).

Is there still some kind of decision to made about this?

[benhoyt]

I need to review and test the fix. I'm back working on scandir (particularly PEP 471 at the moment), so I'll try to get to this in the next week or two.

[benhoyt]

Note to self: this is easy to repro on Windows just by setting sys.platform to something bogus, like sys.platform = 'FOOBAR'.

[benhoyt]

This is actually happening because scandir.py's definition of walk() is not putting symlinks to directories in the right list. See also https://mail.python.org/pipermail/python-dev/2014-July/135401.html

Fix should be simple enough -- I'll commit in the next few days.

[htoothrot]

Maybe I should've fleshed out my previous comment a little more.. I began to note it was down to the difference between lstat and stat but I thought you were already aware. Ideally paying the price for a full stat() would only happen when it's a symlink.

[benhoyt]

Fixed in e601121

[htoothrot]

Ben, with that change scandir.walk will now put symlinks to files in the directories list when followlinks=False. This does not match what os.walk does.

As a side note, I think I might be of the opinion that is_file/is_dir should always use stat instead of lstat when the file is a symlink, which would match os.path.isdir/isfile. What is the reason to the current is_dir/is_file implementation?

[benhoyt]

You're dead right. Thank you.

[benhoyt]

I realized I was overthinking it. To mimic os.walk(), I just need to use entry.is_symlink() in conjunction with os.path.isdir() in the top loop -- as per os.walk(), no followlinks test needed there.

I've fixed this in 4907dc7 and added explicit unit tests to catch this.

Tested on Python 3.4 on Windows, about about to test on Python 2.6 on Linux.

Does this look right to you?

[benhoyt]

Yep, works on Linux on python 2.6 fine too. Haven't tested Python 3 on Linux, but hoping for the best. :-)

[htoothrot]

8 passed in 0.27 seconds (from 4907dc7) on both:

platform linux2 -- Python 3.2.3 -- py-1.4.20 -- pytest-2.5.2
platform linux -- Python 3.4.1 -- py-1.4.20 -- pytest-2.5.2

[benhoyt]

I think this is all fixed now, closing.