WSL2: NTFS symlinks in $PATH cause EPERM and breaks execvp()

[jordansissel]

Please fill out the below information:

• Your Windows build number: 19.03 build 18945.1001

• What you're doing and what's happening: Type a command which is not in the linux $PATH, and path searching fails "operation not permitted". Example:

% asdfasdf
zsh: operation not permitted: asdfasdf

• What's wrong / what should be happening instead: Should be reporting "No such file or directory" or similar:

# A short linux-only path:
% PATH=/bin sh -c 'asdf'
sh: 1: asdf: not found

• Strace of the failing command, if applicable:

% strace  sh -c 'example'
...
stat("/home/jls/bin/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/home/jls/local/bin/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
<omitted for brevity>
stat("/bin/example", 0x7ffda8ae8670)    = -1 ENOENT (No such file or directory)
stat("/usr/games/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/usr/local/games/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/ConEmu/ConEmu/Scripts/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/ConEmu/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/ConEmu/ConEmu/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath/example", 0x7ffda8ae8670) = -1 EPERM (Operation not permitted)
stat("/mnt/c/WINDOWS/system32/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/WINDOWS/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
<omitted for brevity>
stat("/mnt/c/Program Files/Git/cmd/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
stat("/mnt/c/Program Files/PuTTY//example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
<omitted for brevity>
stat("/home/jls/go/bin/example", 0x7ffda8ae8670) = -1 ENOENT (No such file or directory)
write(2, "sh: 1: ", 7)                  = 7
write(2, "example: Operation not permitted", 32) = 32
write(2, "\n", 1)                       = 1
exit_group(127)                         = ?

All of these errors, 32 are ENOENT and 1 is EPERM.

The EPERM error comes from:

stat("/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath/example", 0x7ffda8ae8670) = -1 EPERM (Operation not permitted)

Inspecting this path:

% file "/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath"
/mnt/c/Program Files (x86)/Common Files/Oracle/Java//javapath: unreadable symlink `/mnt/c/Program Files (x86)/Common Files/Oracle/Java//javapath' (Operation not permitted)

% ls -l "/mnt/c/Program Files (x86)/Common Files/Oracle/Java/"
ls: cannot read symbolic link '/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath': Operation not permitted
total 0
lrwxrwxrwx 1 jls jls    0 Jul 16 11:45 javapath
dr-xr-xr-x 1 jls jls 4096 Jul 16 11:45 javapath_target_98222640/

Back on Windows, Powershell, we see this javapath is an NTFS symlink:

> Get-Item "C:\Program Files (x86)\Common Files\Oracle\Java\javapath" | %{ $_.Attributes, $_.Target }
Directory, ReparsePoint
C:\Program Files (x86)\Common Files\Oracle\Java\javapath_target_98222640

Workaround: For any affected program, using which to lookup the path works for me. For example, in GNU Make:

% make build
go build .
make: execvp: go: Operation not permitted

But in my shell, it's fine:

% go version
go version go1.12.6 linux/amd64

Workaround, use which when we know an executable exists but execvp() is otherwise failing.

# Makefile

GO := $(shell which go)
build:
  $(GO) build .

make fails in a special way because of the way it executes using execvp which aborts early upon the first non-ENOENT error:

[pid 29679] execve("/home/jls/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/home/jls/local/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/home/jls/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/home/jls/local/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/local/sbin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory) [pid 29679] execve("/usr/local/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/sbin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/sbin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/bin/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/games/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/usr/local/games/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)[pid 29679] execve("/mnt/c/Program Files/ConEmu/ConEmu/Scripts/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/mnt/c/Program Files/ConEmu/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/mnt/c/Program Files/ConEmu/ConEmu/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 ENOENT (No such file or directory)
[pid 29679] execve("/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath/go", ["go", "build", "."], 0x557311aafb10 /* 33 vars */) = -1 EPERM (Operation not permitted)
make: execvp: go: Operation not permitted

[jordansissel]

I dug into glibc (on my system, glibc 2.27). If I understand things correctly, the following:

The manpage says:

If permission is denied for a file (the attempted execve(2) failed with the error EACCES), these functions will continue searching the rest of the search path. If no other file is found, however, they will return with errno set to EACCES.

This documentation aligns with the glibc code handling EACCES.

However, in this scenario, we get EPERM, not EACCES, from the resulting execve call. EPERM triggers this default case which results in execvp aborting immediately instead of continuing to search $PATH.

Fun! ;)

[therealkenc]

Yep. Good post, your analysis is spot on. #4104 #4255 with a side order of #3886. I swear there is an older issue that describes the EPERM/EACCES difference nearly identically to your follow-up (I recognize the glibc code you linked), but I can't find it.

There are actually two problems with WSL here. (1) WSL imports paths it doesn't have access to, and (2) that errno is wrong; it should be EACCES, and the difference matters. But just changing to errno isn't a complete fix, because stuff can and does crap the bed even if it were EACCES.

[oToToT]

After I login into my fish shell I got

set: Warning: $PATH entry "/mnt/c/Program Files x86/Common Files/Oracle/Java/javapath" is not valid (No such file or directory)
Welcome to fish, the friendly interactive shell

I think it might be a problem relates to this.

[craigloewen-msft]

@oToToT the problem you posted is related to issue: #4104 and is being tracked there.

[microsoft-github-policy-service]

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!