New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Additional windows attributes #22

Closed
woodlandhunter opened this Issue Jun 9, 2014 · 11 comments

Comments

Projects
None yet
2 participants
@woodlandhunter

woodlandhunter commented Jun 9, 2014

I'd like to be able to tell if a file is hidden on windows, but unless I'm mistaken this information is lost when the generic stat object is created (with the C extension at least). Is there a way I'm overlooking? Thoughts?

Thanks.

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 10, 2014

Owner

Actually, this information isn't readily available from Python without resorting to ctypes to call GetFileAttributes() (see http://stackoverflow.com/a/6365265). For better or worse, with Python's os module being quite POSIXY, os.stat() does the same thing.

Owner

benhoyt commented Jun 10, 2014

Actually, this information isn't readily available from Python without resorting to ctypes to call GetFileAttributes() (see http://stackoverflow.com/a/6365265). For better or worse, with Python's os module being quite POSIXY, os.stat() does the same thing.

@woodlandhunter

This comment has been minimized.

Show comment
Hide comment
@woodlandhunter

woodlandhunter Jun 10, 2014

But unless I'm mistaken the hidden attribute is contained in the dwFileAttributes member of our WIN32_FIND_DATA struct, which is already returned/used in the C extension. It's simply discarded here:
https://github.com/benhoyt/scandir/blob/master/_scandir.c#L265

Or am I missing something else? Thanks for your quick response.

woodlandhunter commented Jun 10, 2014

But unless I'm mistaken the hidden attribute is contained in the dwFileAttributes member of our WIN32_FIND_DATA struct, which is already returned/used in the C extension. It's simply discarded here:
https://github.com/benhoyt/scandir/blob/master/_scandir.c#L265

Or am I missing something else? Thanks for your quick response.

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 10, 2014

Owner

Yes, you're dead right. However, the API for scandir is pretty much decided now -- it yields lightweight DirEntry objects which has a name attribute and is_dir/is_file/is_symlink/lstat functions. lstat() returns a stat structure just like os.stat(), which unfortunately doesn't include extended Windows file attribute information.

You've made me think, though -- I wonder if the Windows file attribute info could be stored in one of the non-standard stat_result attributes like st_flags (see here). Or, given that there are non-standard attributes for Linux and different ones for OS X, why couldn't we have st_winattrs on Windows?

Owner

benhoyt commented Jun 10, 2014

Yes, you're dead right. However, the API for scandir is pretty much decided now -- it yields lightweight DirEntry objects which has a name attribute and is_dir/is_file/is_symlink/lstat functions. lstat() returns a stat structure just like os.stat(), which unfortunately doesn't include extended Windows file attribute information.

You've made me think, though -- I wonder if the Windows file attribute info could be stored in one of the non-standard stat_result attributes like st_flags (see here). Or, given that there are non-standard attributes for Linux and different ones for OS X, why couldn't we have st_winattrs on Windows?

@woodlandhunter

This comment has been minimized.

Show comment
Hide comment
@woodlandhunter

woodlandhunter Jun 11, 2014

That would be helpful, but I can see people aren't likely to be clamoring for it.

I know you'd like for scandir to be included in the stdlib (a fine notion), so is that question about windows-specific stat attributes (or using st_flags) a python-dev/ideas question?

(I have an app that indexes across a NFS. Using scandir made a massive improvement in initial indexing time. So, I hate to add back in an extra syscall per file to add checking for hidden files.)

woodlandhunter commented Jun 11, 2014

That would be helpful, but I can see people aren't likely to be clamoring for it.

I know you'd like for scandir to be included in the stdlib (a fine notion), so is that question about windows-specific stat attributes (or using st_flags) a python-dev/ideas question?

(I have an app that indexes across a NFS. Using scandir made a massive improvement in initial indexing time. So, I hate to add back in an extra syscall per file to add checking for hidden files.)

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 11, 2014

Owner

I've just asked about this (adding an st_winattrs attribute to stat result on Windows) on python-dev, and it looks like there's general support. It'd be great to get this into Python 3.5 too (along with scandir if possible :-). Here's the thread and my suggestion for the final "API" for it:

https://mail.python.org/pipermail/python-dev/2014-June/134990.html
https://mail.python.org/pipermail/python-dev/2014-June/135012.html

Basically we'd add a ".st_winattrs" attribute with the Win32 file attributes DWORD to the stat result object on Windows. And then we'd add the relevant Windows constants as FILE_ATTRIBUTE_* to the stat module (via _stat.c in CPython). So testing for a hidden file would become:

import os
import stat

def is_hidden(path):
    st = os.stat(path)
    return getattr(st, 'st_winattrs', 0) & stat.FILE_ATTRIBUTE_HIDDEN != 0

As the next step I've opened Issue 21719 on bugs.python.org. Feel free to provide a patch there if you're in a position to (I have to figure out how to compile CPython 3 on Windows), otherwise I hope to get to it in the next couple of weeks.

I'll keep this issue open so we remember to also add it to scandir.

Owner

benhoyt commented Jun 11, 2014

I've just asked about this (adding an st_winattrs attribute to stat result on Windows) on python-dev, and it looks like there's general support. It'd be great to get this into Python 3.5 too (along with scandir if possible :-). Here's the thread and my suggestion for the final "API" for it:

https://mail.python.org/pipermail/python-dev/2014-June/134990.html
https://mail.python.org/pipermail/python-dev/2014-June/135012.html

Basically we'd add a ".st_winattrs" attribute with the Win32 file attributes DWORD to the stat result object on Windows. And then we'd add the relevant Windows constants as FILE_ATTRIBUTE_* to the stat module (via _stat.c in CPython). So testing for a hidden file would become:

import os
import stat

def is_hidden(path):
    st = os.stat(path)
    return getattr(st, 'st_winattrs', 0) & stat.FILE_ATTRIBUTE_HIDDEN != 0

As the next step I've opened Issue 21719 on bugs.python.org. Feel free to provide a patch there if you're in a position to (I have to figure out how to compile CPython 3 on Windows), otherwise I hope to get to it in the next couple of weeks.

I'll keep this issue open so we remember to also add it to scandir.

@woodlandhunter

This comment has been minimized.

Show comment
Hide comment
@woodlandhunter

woodlandhunter Jun 13, 2014

Fantastical. Thank you for taking the time to do that. I take it once the specifics there are hammered out scandir can grow support to make use of already having that data.

If I take a whack at a patch, I'll be sure to post to the issue

woodlandhunter commented Jun 13, 2014

Fantastical. Thank you for taking the time to do that. I take it once the specifics there are hammered out scandir can grow support to make use of already having that data.

If I take a whack at a patch, I'll be sure to post to the issue

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 13, 2014

Owner

Cool. I've got half of a patch done. Hopefully I can do the rest tomorrow or Monday and then submit it.

Owner

benhoyt commented Jun 13, 2014

Cool. I've got half of a patch done. Hopefully I can do the rest tomorrow or Monday and then submit it.

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 13, 2014

Owner

FYI, I've added a patch to add this to Python 3.5 on the Python issue: http://bugs.python.org/issue21719

Owner

benhoyt commented Jun 13, 2014

FYI, I've added a patch to add this to Python 3.5 on the Python issue: http://bugs.python.org/issue21719

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 19, 2014

Owner

This is coming to an os.stat() near you in Python 3.5! https://docs.python.org/3.5/whatsnew/3.5.html#os

I'll add this to scandir as well, but that may have to wait a couple of weeks.

Owner

benhoyt commented Jun 19, 2014

This is coming to an os.stat() near you in Python 3.5! https://docs.python.org/3.5/whatsnew/3.5.html#os

I'll add this to scandir as well, but that may have to wait a couple of weeks.

@benhoyt

This comment has been minimized.

Show comment
Hide comment
@benhoyt

benhoyt Jun 20, 2014

Owner

Done in 89ee086 and uploaded scandir 0.5 on PyPI

Owner

benhoyt commented Jun 20, 2014

Done in 89ee086 and uploaded scandir 0.5 on PyPI

@benhoyt benhoyt closed this Jun 20, 2014

@woodlandhunter

This comment has been minimized.

Show comment
Hide comment
@woodlandhunter

woodlandhunter Jun 28, 2014

contributed by Ben Hoyt in issue 21719.

So great. Thank you!

woodlandhunter commented Jun 28, 2014

contributed by Ben Hoyt in issue 21719.

So great. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment