Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Have os.unlink remove junction points #62514

Closed
KimGrsman mannequin opened this issue Jun 27, 2013 · 45 comments
Closed

Have os.unlink remove junction points #62514

KimGrsman mannequin opened this issue Jun 27, 2013 · 45 comments
Assignees
Labels
OS-windows type-feature A feature request or enhancement

Comments

@KimGrsman
Copy link
Mannequin

KimGrsman mannequin commented Jun 27, 2013

BPO 18314
Nosy @tim-one, @terryjreedy, @tjguk, @briancurtin, @zware, @eryksun
Files
  • unlink_junction.patch
  • unlink-junctions.patch
  • unlink_junctions_r2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/tjguk'
    closed_at = <Date 2014-05-05.20:01:05.083>
    created_at = <Date 2013-06-27.12:15:21.053>
    labels = ['type-feature', 'OS-windows']
    title = 'Have os.unlink remove junction points'
    updated_at = <Date 2014-05-07.00:33:11.076>
    user = 'https://bugs.python.org/KimGrsman'

    bugs.python.org fields:

    activity = <Date 2014-05-07.00:33:11.076>
    actor = 'eryksun'
    assignee = 'tim.golden'
    closed = True
    closed_date = <Date 2014-05-05.20:01:05.083>
    closer = 'tim.golden'
    components = ['Windows']
    creation = <Date 2013-06-27.12:15:21.053>
    creator = 'Kim.Gr\xc3\xa4sman'
    dependencies = []
    files = ['31915', '33288', '35095']
    hgrepos = []
    issue_num = 18314
    keywords = ['patch']
    message_count = 45.0
    messages = ['191945', '191946', '192177', '195114', '195121', '195124', '198416', '198500', '198647', '198649', '200058', '201018', '201054', '201055', '207097', '207157', '208500', '211008', '213278', '216579', '217287', '217288', '217293', '217311', '217312', '217318', '217404', '217463', '217464', '217471', '217479', '217488', '217538', '217940', '217941', '217943', '217948', '217949', '217953', '217955', '217966', '217969', '217975', '217976', '218030']
    nosy_count = 8.0
    nosy_names = ['tim.peters', 'terry.reedy', 'tim.golden', 'brian.curtin', 'python-dev', 'zach.ware', 'eryksun', 'Kim.Gr\xc3\xa4sman']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue18314'
    versions = ['Python 3.5']

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Jun 27, 2013

    os.unlink currently raises a WindowsError (Access Denied) if I attempt to unlink an NTFS junction point.

    It looks trivial to allow Py_DeleteFileW [1] to remove junction points as well as proper symbolic links, as far as I can tell.

    For example, the ntfslink-python library [2] only checks if both FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_REPARSE_POINT are set.

    RemoveDirectoryW is documented to handle junction points transparently, so it should just be a matter of passing the path on if it's a junction point or a symbolic link.

    My motivation for this is that I have used external tools to create junction points, and am now switching to symbolic links. When deleting a directory, I need to do:

    try:
        os.unlink(link_path)
    except WindowsError as detail:
        # BACKWARD COMPATIBILITY HACK
        if detail.winerror == 5:
            _delete_junction_point(link_path)
        else:
            raise
    

    which is a little funky. It seems like os.unlink semantics work just as well for junction points, even if they can't be created with os.symlink.

    Love it/hate it?

    [1] http://hg.python.org/cpython/file/44f455e6163d/Modules/posixmodule.c#l4105
    [2] https://github.com/Juntalis/ntfslink-python/blob/2f6ff903f9b22942de8aa93a32a3d817124f359e/ntfslink/internals/__init__.py#L32

    @KimGrsman KimGrsman mannequin added type-bug An unexpected behavior, bug, or error OS-windows labels Jun 27, 2013
    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Jun 27, 2013

    Also, I believe the reason os.unlink raises "access denied" is because a junction point does not currently qualify as a directory && link, so its path is passed directly to DeleteFileW, which in turn refuses to delete a directory.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Jul 2, 2013

    This comment outlines how to tell junction points from other mount points:
    http://www.codeproject.com/Articles/21202/Reparse-Points-in-Vista?msg=3651130#xx3651130xx

    This should port straight into Py_DeleteFileW.

    Would anyone be interested in a patch?

    @vstinner
    Copy link
    Member

    vstinner commented Aug 14, 2013

    What is "an NTFS junction point"? Is it possible to delete it in cmd.exe with the del command?

    @tim-one
    Copy link
    Member

    tim-one commented Aug 14, 2013

    Victor, Wikipedia has a readable explanation:

    http://en.wikipedia.org/wiki/NTFS_junction_point

    I haven't used them much. From cmd.exe, I've been able to delete them, not with "del" but with "rmdir". You can create one from cmd.exe with the "mklink" command.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Aug 14, 2013

    Victor,

    Junction points are like links between directories only. They've been around since the NTFS that came with Windows 2000, but integration with OS tools has been generally poor (e.g. Explorer wouldn't see the difference between a junction point and a regular folder.) As of Windows 7, this works better.

    @terryjreedy
    Copy link
    Member

    terryjreedy commented Sep 25, 2013

    I am not sure if this is a bug or enhancement. It is a moot point until there is a patch to apply.

    A patch would need a test that fails now and passes with the patch. From the Wikipedia article, it appears that a test using mklink /J would not run on XP and would have to be skipped. I would not expect an XP buildbot to necessarily have the Server 2003 Resource Kit needed for an XP test.

    Does _delete_junction_point(link_path) work on XP or should the feature be restricted to Vista+?

    @terryjreedy terryjreedy added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Sep 25, 2013
    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Sep 27, 2013

    _delete_junction_point currently shells out to a command-line tool, junction.exe, from SysInternals. That ran fine on XP.

    As I understand it, RemoveDirectoryW on XP also takes care of junction points, but I'll find a machine to verify.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Sep 29, 2013

    Attached is a patch that considers directory symlinks and junction points equivalent.

    I'm struggling with the test -- would it be acceptable to only run this test on platforms that have mklink /j (Windows Vista and higher)?

    I've looked at programmatic means of creating junction points, but it involves enough Win32 interop to make it a candidate for a module all by itself (it's REALLY messy.)

    Any ideas?

    Thanks!

    @tjguk
    Copy link
    Member

    tjguk commented Sep 29, 2013

    I'll try to pick this one up over the next few days. Feel free to ping me if it drops into silence!

    @tjguk tjguk self-assigned this Sep 29, 2013
    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Oct 16, 2013

    Gentle ping.

    @tjguk
    Copy link
    Member

    tjguk commented Oct 23, 2013

    Just picking this up. Considering testing... My current proposal is to add junction point support to _winapi, initially for the sole purpose of testing this change, but with a view to possibly supporting it formally via the os module. Any better ideas?

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Oct 23, 2013

    I didn't know about _winapi; looks like a good place!

    It looks like it exposes the Windows API pretty faithfully, but the junction point stuff feels like it would be better represented as a simple _winapi.CreateJunctionPoint(source, target) rather than attempting to expose DeviceIoControl and associated structures.

    I'll try to cook up a patch and we can argue about details based on that :-)

    Thanks!

    @tjguk
    Copy link
    Member

    tjguk commented Oct 23, 2013

    Sounds like a decent plan to me. Good luck with the buffer sizing!

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Dec 30, 2013

    I really needed the well-wishing with regard to buffer sizing :-)

    Here's a patch for a couple of fronts:

    • Teach os.unlink about junction points
    • Introduce _winapi.CreateJunction
    • Introduce a new test suite in test_os.py for junction points

    I pulled the definition of _REPARSE_DATA_BUFFER out into a new header called winreparse.h.

    I'd appreciate critical review of _winapi.CreateJunction to make sure I haven't missed anything. I'm not familiar with the Python/C interop idioms, so I might have missed something wrt arguments/return values handling.

    Happy new year!

    @tjguk
    Copy link
    Member

    tjguk commented Jan 2, 2014

    I'll have a look at this in a week or so when I'm back on-line.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Jan 19, 2014

    Thanks!

    There's another thing I would appreciate having somebody else test: creating and removing junctions in a non-elevated prompt. I haven't been able to, my IT department has trouble understanding the value of least-privilege.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Feb 11, 2014

    ping

    1 similar comment
    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Mar 12, 2014

    ping

    @tjguk
    Copy link
    Member

    tjguk commented Apr 16, 2014

    All tests pass on 3.5 and in an unelevated prompt. I'll have a closer look at the code tomorrow.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 27, 2014

    New changeset 17df50df62c7 by Tim Golden in branch 'default':
    Issue bpo-18314 os.unlink will now remove junction points on Windows. Patch by Kim Gräsman.
    http://hg.python.org/cpython/rev/17df50df62c7

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 27, 2014

    New changeset 4b97092aa4bd by Tim Golden in branch 'default':
    Issue bpo-18314 Add NEWS item.
    http://hg.python.org/cpython/rev/4b97092aa4bd

    @tjguk
    Copy link
    Member

    tjguk commented Apr 27, 2014

    Backed out the commits after all the Windows buildbots broke. Need to look further. (No problems on a Win7 or Ubuntu build here).

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 27, 2014

    Thanks for pushing this forward! Do you have links to the failing bots I could take a look at?

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 27, 2014

    Thanks!

    At first I suspected 32 vs 64 bit, but the failing bots cover both...

    One thing that stands out to me as risky is the memcmp() against "\\??\\", which could overrun a short src_path buffer. But I don't think that would fail here.

    I must have made some mistake with the REPARSE_DATA_BUFFER, but I can't see anything off hand.

    What are our debugging options?

    @tjguk
    Copy link
    Member

    tjguk commented Apr 28, 2014

    I'm just pinging #python-dev to see if there's a way to request a buildbot build from a specific server-side clone.

    Meanwhile, though, I definitely introduced a change into your code which I thought I had reverted, but clearly hadn't! The code, as committed, used PyMem_RawAlloc in place of the calloc() call you had, but didn't replace the later free() by its PyMem counterpart.

    If I don't get any joy with the clone-specific buildbot question, I'll just rebuild from your original patch, re-commit, and watch the buildbots.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 29, 2014

    Aha, that might cause trouble.

    I think you should add a memset() to sero out the newly allocated buffer also, I think I may have used calloc to be able to assume it was initialized with zeros.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 29, 2014

    Sorry, that wasn't clear. I mean if you change allocator _from_ calloc, make sure the buffer is zeroed out after allocation.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 29, 2014

    By the way, is PyMem_RawMalloc/PyMem_RawFree preferred for memory allocation across the board?

    If so, I can just prepare a new patch for you with that changed, zero-initialization in place and the prefix-overrun fixed. I might get to it tonight.

    @tjguk
    Copy link
    Member

    tjguk commented Apr 29, 2014

    Yes, now that the custom allocator / tracing stuff is in place:
    otherwise there's no way for custom allocation or tracing to occur.
    Please go ahead and rework the patch when you have the time.

    Also, since the setup of the reparse header is such an underdocumented
    nightmare, please add as much commentary as possible around the choice
    of allocations & offsets. I was reviewing your code with an eye on the
    various MSDN pages and examples and I still wasn't always able to follow
    your choices.

    (BTW I'm not convinced that the PyMem change was the problem since the
    PyMem_Raw* functions simply hand off to malloc/free unless there's a
    custom allocator. Unless of course the missing memset-0 was significant
    here).

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 29, 2014

    Also, since the setup of the reparse header is such an underdocumented
    nightmare, please add as much commentary as possible around the choice
    of allocations & offsets.

    I'll try. It might turn into a novel.

    (BTW I'm not convinced that the PyMem change was the problem since
    the PyMem_Raw* functions simply hand off to malloc/free unless
    there's a custom allocator.
    Unless of course the missing memset-0 was significant here).

    I think it might be, there was a message in the log that DeviceIoControl failed:

    stty: standard input: Inappropriate ioctl for device

    That could be attributed to garbage in the buffer.

    I'll be back with a revised patch, and we can work from there. Thanks for your help!

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented Apr 29, 2014

    Here's a new attempt, please let me know if this works out better.

    Changes:

    • Switched to CRT string functions (wcsncmp, wcscpy) instead of Windows lstrxxxW. There was no lstrncmpW.
    • Switched to PyMem_Raw(Malloc|Free) and added explicit memset after allocation
    • Better error handling (check arguments for NULL, check memory allocation)
    • Fix possible overrun when checking if src_path starts with "\??\"
    • Extensive commentary to describe the buffer sizing

    Hope this works out better. I already have ideas for improvements, but I think we can try to get this in place first.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 5, 2014

    New changeset 5c1c14ff1f13 by Tim Golden in branch 'default':
    bpo-18314 Allow unlink to remove junctions. Includes support for creating junctions. Patch by Kim Gräsman
    http://hg.python.org/cpython/rev/5c1c14ff1f13

    @zware
    Copy link
    Member

    zware commented May 5, 2014

    Just skimming, I noticed something about replacing calloc() with PyMem_RawAlloc; note that there is now PyMem_Calloc or PyMem_RawCalloc that you should be able to use if you prefer. See bpo-21233.

    @tjguk
    Copy link
    Member

    tjguk commented May 5, 2014

    Thanks, Zach. I was aware that calloc was in the air, but I wasn't sure
    if it had been committed and I'd delayed on this enough so I thought I'd
    push for now. We can always revisit.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 5, 2014

    New changeset e791a36ab6a2 by Tim Golden in branch 'default':
    bpo-18314 ACKS & NEWS
    http://hg.python.org/cpython/rev/e791a36ab6a2

    @tjguk
    Copy link
    Member

    tjguk commented May 5, 2014

    Buildbots seem happy. Thanks very much for the patches!

    @tjguk tjguk closed this as completed May 5, 2014
    @eryksun
    Copy link
    Contributor

    eryksun commented May 5, 2014

    For a junction reparse point, Sysinternals junction.exe shows the "Print Name" and "Substitute Name" are the same and it's not an NT \?? path (i.e. \DosDevices, i.e. \Global??). OTOH, the substitute name does use an NT DosDevices path when I use cmd's mklink or os.symlink to create a directory symbolic link to an absolute path.

    I modified the patch in a test program to see whether it's really necessary to include the DosDevices path in the substitute name. FYI, it seems to work fine without it, but it also doesn't seem to hurt to include it.

    Sysinternals Junction
    http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx

    @tjguk
    Copy link
    Member

    tjguk commented May 5, 2014

    Thanks for the research, eryksun. As long as it doesn't hurt let's leave
    it as is for now.

    @KimGrsman
    Copy link
    Mannequin Author

    KimGrsman mannequin commented May 6, 2014

    Thanks for helping me land this!

    eryksun: interesting, thanks! I seem to remember having problems without the \??\ prefix, but it could have been something else causing it (filling the buffer to DeviceIoControl's satisfaction was... challenging.)

    I have some ideas for small improvements, and I could try to fold the removal of \??\ into that. Do I just open a new enhancement issue?

    @terryjreedy
    Copy link
    Member

    terryjreedy commented May 6, 2014

    Yes, new issue.

    @eryksun
    Copy link
    Contributor

    eryksun commented May 6, 2014

    Nevermind, strike "seems to work"; it doesn't work without the \?? prefix. I stupidly assumed the DeviceIoControl call would validate the substitute name. Of course it doesn't; it happily creates a broken junction. Opening the junction with CreateFile fails with either ERROR_INVALID_REPARSE_DATA or ERROR_CANT_RESOLVE_FILENAME. Sorry.

    @tjguk
    Copy link
    Member

    tjguk commented May 6, 2014

    Thanks, eryksun: failed experiments are still useful data for future
    reference!

    @eryksun
    Copy link
    Contributor

    eryksun commented May 7, 2014

    For some read Sysinternals junction utility doesn't show the raw substitute name. Microsoft's fsutil shows the actual reparse data:

    C:\>mklink /J Python34 "C:\Program Files\Python34"
    Junction created for Python34 <<===>> C:\Program Files\Python34
    
    C:\>fsutil reparsepoint query Python34
    Reparse Tag Value : 0xa0000003
    Tag value: Microsoft
    Tag value: Name Surrogate
    Tag value: Mount Point
    Substitue Name offset: 0
    Substitue Name length: 58
    Print Name offset:     60
    Print Name Length:     50
    Substitute Name:       \??\C:\Program Files\Python34
    Print Name:            C:\Program Files\Python34
    
    Reparse Data Length: 0x00000078
    Reparse Data:
    0000:  00 00 3a 00 3c 00 32 00  5c 00 3f 00 3f 00 5c 00  ..:.<.2.\.?.?.\.
    0010:  43 00 3a 00 5c 00 50 00  72 00 6f 00 67 00 72 00  C.:.\.P.r.o.g.r.
    0020:  61 00 6d 00 20 00 46 00  69 00 6c 00 65 00 73 00  a.m. .F.i.l.e.s.
    0030:  5c 00 50 00 79 00 74 00  68 00 6f 00 6e 00 33 00  \.P.y.t.h.o.n.3.
    0040:  34 00 00 00 43 00 3a 00  5c 00 50 00 72 00 6f 00  4...C.:.\.P.r.o.
    0050:  67 00 72 00 61 00 6d 00  20 00 46 00 69 00 6c 00  g.r.a.m. .F.i.l.
    0060:  65 00 73 00 5c 00 50 00  79 00 74 00 68 00 6f 00  e.s.\.P.y.t.h.o.
    0070:  6e 00 33 00 34 00 00 00                           n.3.4...
    

    fsutil reparsepoint:
    http://technet.microsoft.com/en-us/library/cc785451.aspx

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    OS-windows type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants