Permission denied renaming VolFs hardlink directory

[marksteward]

On Windows 10.0.14393, directories on VolFS that contain hardlink targets cannot be moved shortly after the hardlink is created. This breaks the build process for Chromium (for example), as glibc_arm.tgz contains this structure.

To reproduce, create a hardlink in your home directory from b/2 to a/1 and then try to rename a to ax:

$ mkdir a b
$ touch a/1
$ ln a/1 b/2
$ mv a{,x}
mv: cannot move ‘a’ to ‘ax’: Permission denied

The mv here should not fail. If you exit all WSL windows and open bash again, it succeeds, suggesting the issue is related to internal state or caching. You must exit WSL after the ln step: if you exit before this step it still fails.

Renaming a/1 or b/2 doesn't make a difference, but after renaming b (even if renamed back again), it will succeed within the same session:

$ mkdir a b
$ touch a/1
$ ln a/1 b/2
$ mv b/2{,x}
$ mv a{,x}
mv: cannot move ‘a’ to ‘ax’: Permission denied
$ mv b{,x}
$ mv b{x,}
$ mv a{,x}
$ 

Running find -name nonexistent_name for a few seconds also allows it to succeed, reinforcing that it's some sort of cache problem.

It also works fine on DrvFs.

strace of the failing mv (the failing call is rename, with EACCES) here.

[jackchammons]

I am unable to reproduce this on build 14965.

@marksteward could you upgrade to a recent insider build of Windows and confirm that this issue is fixed?

[marksteward]

I've now enrolled but I guess it'll be a few days before I can confirm. Thanks!

[marksteward]

I've installed 14965 using the ISO as I didn't receive anything through the upgrade mechanism, and I can still reproduce this issue. Did you reproduce the issue on 14393?

[marksteward]

I've downgraded back to 14393 as Chrome kept crashing and there are no kernel symbols for that build on the public server. It turns out the error is a 0xc0000022 status from this call, which is turned into -EACCES:

LXCORE!LxpUtilRenameFileByHandle+0x9c:
fffff800`4a8c0d18 ff150294fcff    call    qword ptr [LXCORE!_imp_ZwSetInformationFile (fffff800`4a88a120)] ds:002b:fffff800`4a88a120={nt!ZwSetInformationFile (fffff801`edfc8190)}

SetInformationFile is passed FileRenameInformation, and the documentation calls out the following failure case:

A directory cannot be renamed if it or any of its subdirectories contains a file that has open handles (except in the batch-oplock case described earlier).

After running the repro case there are indeed open handles on both a/1 and b/2, annoyingly owned by the System process. The handles are the same on each failing call to LxpUtilRenameFileByHandle, but can be closed by the workarounds I mentioned before:

• Exiting all WSL processes:

 # RetAddr           : Args to Child                                                           : Call Site
00 fffff800`18069720 : 00000000`00000000 fffff802`2eb6f9e0 00000000`00000000 fffff802`2e95be66 : nt!ZwClose
01 fffff800`1806b22e : ffffa600`1b18c460 ffffc981`d94bb270 00000000`00000000 fffff802`2e95be66 : LXCORE!LxVolFsDirectoryEntryTreeDelete+0x18
02 fffff800`1805d97f : ffffc981`da14ce50 ffffc981`00000001 ffffc981`da14ce50 ffffa600`1b18c460 : LXCORE!LxVolFsInodeDelete+0x4e
03 fffff800`180653a3 : ffffc981`d1a1f3f0 00000054`00000001 ffffc981`d94bb270 00000000`0000000a : LXCORE!VfsDereferenceInode+0x93
04 fffff800`1806480f : ffffc981`d1a1f3f0 ffffc981`da6b6620 ffffc981`da6b6620 ffffc981`da6b6650 : LXCORE!VfsDereferenceDirectoryEntry+0x87
05 fffff800`1805ef1a : ffffc981`da6a0160 ffffc981`da6b6650 ffffc981`da6b6620 ffffc981`d129b010 : LXCORE!VfsDirectoryEntryCacheClearFsInstance+0xd7
06 fffff800`180660ae : ffffc981`da7626b0 ffffc981`d129b2e8 ffffc981`da7626b0 ffffc981`00000013 : LXCORE!VfsDereferenceFsInstance+0xaa
07 fffff800`1805f931 : ffffc981`d1a1f3b8 ffffc981`d129b010 ffffc981`d129b2e8 ffffc981`0000000b : LXCORE!VfsDereferenceMount+0x4a
08 fffff800`18031549 : ffffc981`da9845a8 ffffc981`d129b2e8 ffffc981`d129b2e8 ffffc981`d129b010 : LXCORE!VfsMountNamespaceUninitialize+0x3d
09 fffff800`18031056 : ffffc981`d129b010 ffffc981`d129b010 ffffc981`d129b010 00000000`00000050 : LXCORE!LxpInstanceUninitialize+0x12d
0a fffff800`180312f9 : ffffc981`d129b010 00000000`00000000 ffffc981`d129b068 00000000`000000a0 : LXCORE!LxpInstanceSetStopState+0x2e
0b fffff800`18030f6f : ffffc981`d129b010 ffffb98f`879e1000 ffffb98f`86d17040 fffff802`2e95b660 : LXCORE!LxpInstanceStop+0xe9

• Renaming b:

 # RetAddr           : Args to Child                                                           : Call Site
00 fffff800`18069720 : ffffc981`e24f6300 ffffc981`ddce3b90 ffffc63f`c428d556 ffffc981`dd2cac60 : nt!ZwClose
01 fffff800`1806be30 : 00000000`00000000 ffffa600`1db4f6f8 ffffc981`dd2cac00 ffffc981`ddce3b70 : LXCORE!LxVolFsDirectoryEntryTreeDelete+0x18
02 fffff800`180623d5 : ffffc981`dabffd60 ffffa600`1db4f800 00000000`00000000 00000000`00000000 : LXCORE!LxVolFsInodeRename+0x80
03 fffff800`1803afc5 : ffffc981`d1a1f380 00000000`00000000 ffffc981`dd2cac60 00000000`58445a00 : LXCORE!VfsPerformRename+0x4dd
04 fffff800`18041569 : 00000000`00000000 00007fff`fceb2903 00007fff`fceb2903 00007fff`fceb2905 : LXCORE!LxpRenameHelper+0xed
05 fffff800`1803c1c8 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : LXCORE!LxpSyscall_RENAME+0x19

• Running du -shx /:

 # RetAddr           : Args to Child                                                           : Call Site
00 fffff800`18069720 : 00000000`00000000 fffff802`2eb6f9e0 00000000`00000000 fffff802`2e95be66 : nt!ZwClose
01 fffff800`1806b22e : ffffc981`ea1dac01 ffffc981`dd677220 00000000`00000000 fffff802`2e95be66 : LXCORE!LxVolFsDirectoryEntryTreeDelete+0x18
02 fffff800`1805d97f : ffffc981`e481ce50 ffffc981`00000001 ffffc981`e481ce50 ffffc981`ea1dac01 : LXCORE!LxVolFsInodeDelete+0x4e
03 fffff800`180653a3 : ffffc981`e28b4650 ffffc981`00000001 ffffc981`dd677220 fffff802`0000000a : LXCORE!VfsDereferenceInode+0x93
04 fffff800`1806471b : ffffc981`e28b4650 ffffc981`e67cb620 ffffc981`e67cb620 ffffc981`ea1dabe0 : LXCORE!VfsDereferenceDirectoryEntry+0x87
05 fffff800`18064b6b : 00000000`00000000 ffffc981`ea1dac00 ffffc981`ea1dabe0 ffffc981`e67cb620 : LXCORE!VfsDirectoryEntryCacheAdd+0xfb
06 fffff800`17ff3695 : ffffc981`e67cb620 ffffc981`ea1dac00 ffffc981`ea1dabe0 00000000`00000000 : LXCORE!VfsCacheDirectoryEntry+0x17
07 fffff800`18065757 : ffffc981`e67cb620 ffffa600`1b4716e0 ffffc981`ea1dabe0 ffffa600`1b471640 : LXCORE!VfsLookupDirectoryEntryInternal+0x14d
08 fffff800`18065625 : ffffc981`e95d6010 00000000`00000000 ffffa600`1b4716f0 ffffa600`1b471640 : LXCORE!VfsLookupDirectoryEntryLocked+0x3b
09 fffff800`1806425e : 00000000`00000001 ffffa600`1b471669 ffffc981`e28b45e0 ffffc981`e28b45e0 : LXCORE!VfsLookupDirectoryEntry+0x41
0a fffff800`1806455e : 00000000`00000000 00000000`00dd2e78 00000000`00000007 00000000`00000000 : LXCORE!VfsLookupPath+0x1e2
0b fffff800`18061069 : ffffc981`e539d9b0 00000000`00000100 ffffc981`e28b45e0 00000000`00000000 : LXCORE!VfsLookupRelativePath+0x8a
0c fffff800`1804e548 : ffffc981`e539d9b0 00000000`00000100 00000000`00000007 fffff800`1805e653 : LXCORE!VfsGetInodeForPath+0x51
0d fffff800`1803bebe : 00000000`00000000 ffffa600`1b4719b9 00000000`00dd2de8 00000000`00000100 : LXCORE!LxpUtilGetInodeForStartingDirectoryAndPath+0xa4
0e fffff800`1803e197 : 00000000`00000000 ffffa600`1b4719b9 ffffc63f`c2db34e6 00000000`00000001 : LXCORE!LxpStatHelper+0x142
0f fffff800`1803c1b2 : 00000000`00000000 ffffa600`1b4719b9 00000000`00000000 00000000`00000003 : LXCORE!LxpSyscall_FSTATAT64+0x17

This last one shows there's a VfsDirectoryEntryCache, and that the cache has a limited size.

Without digging into the code, my guess is that VolFs resolves both paths to the same inode, which is then mapped to the same file handle twice, causing a temporary leak (after opening by path when creating the hardlink).

[sunilmut]

@SvenGroot as FYI

[marksteward]

Is this still unreproducible? Should I spend some time this weekend seeing if it's something weird on my machine?

[marksteward]

I can still reproduce this on build 14986 in a VM - can I suggest removing the fixedininsiderbuilds tag?

[dontsave]

I can also confirm this is happening in 14986. It's been causing npm installs to fail in my environment. Solution is to reboot the terminal

[marksteward]

Still happening in 16299.

[therealkenc]

Can't reproduce this on recent WSL1, where "recent" was probably circa the 9p filesystem changes that happened in 18362. If a similar problem persists don't hesitate to open a new issue.