os: FileInfo Mode on Windows incorrectly interpreting WSL symlinks (reparse points) as regular files always

[kfsone]

What version of Go are you using (go version)?

$ go version
go version go1.15.2 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\oliver\AppData\Local\go-build
set GOENV=C:\Users\oliver\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\oliver\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\oliver\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\oliver\AppData\Local\Temp\go-build844512546=/tmp/go-build -gno-record-gcc-switches

What did you do?

Called os.FileInfo.Mode.IsRegular and got the wrong result.

This is caused by a WSL symlink:

[D:\Dev\github.com\kfsone\archive\smugl-src]
> gci compile | format-list

    Directory: D:\Dev\github.com\kfsone\archive\smugl-src

Name           : compile
Length         : 0
CreationTime   : 7/28/2019 10:36:58 PM
LastWriteTime  : 7/28/2019 10:36:58 PM
LastAccessTime : 7/28/2019 10:36:58 PM
Mode           : la---
LinkType       :
Target         :
VersionInfo    : File:             D:\Dev\github.com\kfsone\archive\smugl-src\compile
                 InternalName:
                 OriginalFilename:
                 FileVersion:
                 FileDescription:
                 Product:
                 ProductVersion:
                 Debug:            False
                 Patched:          False
                 PreRelease:       False
                 PrivateBuild:     False
                 SpecialBuild:     False
                 Language:

Running the following go code:

package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    path := "D:/dev/github.com/kfsone/archive/smugl-src/compile"
    filepath.Walk(path, func (path string, info os.FileInfo, err error) error {
        fmt.Printf("%s %s %#+v\n", path, info.Mode().IsRegular(), info)
        return nil
    })
}

produces:

D:/dev/github.com/kfsone/archive/smugl-src/compile %!s(bool=true) &os.fileStat{name:"compile", FileAttributes:0x420, CreationTime:syscall.Filetime{LowDateTime:0xa3aab077, HighDateTime:0x1d545cf}, LastAccessTime:syscall.Filetime{LowDateTime:0xa3aab077, HighDateTime:0x1d545cf}, LastWriteTime:syscall.Filetime{LowDateTime:0xa3aab077, HighDateTime:0x1d545cf}, FileSizeHigh:0x0, FileSizeLow:0x0, Reserved0:0xa000001d, filetype:0x0, Mutex:sync.Mutex{state:0, sema:0x0}, path:"", vol:0x5ef35831, idxhi:0xb0000, idxlo:0x3b47, appendNameToPath:false}

Windows is definitely telling us it's a reparse point: FileAttributes: 0x420 includes the 1024 required. The issue here is the value of Reserved0, which is not being tested for:

According to https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4

0xA000001D
Used by the Windows Subsystem for Linux (WSL) to represent a UNIX symbolic link. Server-side interpretation only, not meaningful over the wire.

This appears it would require a change to src/os/types_windows.go lines 104,105.

[odeke-em]

Thank you for filing this issue @kfsone and welcome to the Go project!

Interesting bug, and thank you for the background. Please feel free to send a change to fix this as per https://golang.org/doc/contribute.html or even just a PR on Github after signing the CLA, it would be
awesome to have your contributions!

Just for full disclosure we are about 6 days away from a code freeze for Go1.16, thus if you can, please send a change,
with a regression test, and I shall also tag some Windows experts @alexbrainman @mattn. No rush though, because even
if not meeting the deadline, in about 2 months, if/when the change is accepted, we can always backport it.

[mattn]

CreateFile fail to open symlink created on WSL2 without FILE_FLAG_OPEN_REPARSE_POINT. To fix this, we should add FILE_FLAG_OPEN_REPARSE_POINT to second third argument of stat.

diff --git a/src/os/stat_windows.go b/src/os/stat_windows.go
index da4c49090e..43f176376c 100644
--- a/src/os/stat_windows.go
+++ b/src/os/stat_windows.go
@@ -103,7 +103,7 @@ func stat(funcname, name string, createFileAttrs uint32) (FileInfo, error) {
 
 // statNolog implements Stat for Windows.
 func statNolog(name string) (FileInfo, error) {
-	return stat("Stat", name, syscall.FILE_FLAG_BACKUP_SEMANTICS)
+	return stat("Stat", name, syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS)
 }
 
 // lstatNolog implements Lstat for Windows.

[mattn]

Hmm, TestDirectoryJunction fail if adding FILE_FLAG_OPEN_REPARSE_POINT. So it should do fallback if CreateFile fail.

[gopherbot]

Change https://golang.org/cl/265037 mentions this issue: os: fallback to open symlinks

[kfsone]

CreateFile fail to open symlink created on WSL2 without FILE_FLAG_OPEN_REPARSE_POINT. To fix this, we should add FILE_FLAG_OPEN_REPARSE_POINT to second third argument of stat.

diff --git a/src/os/stat_windows.go b/src/os/stat_windows.go
index da4c49090e..43f176376c 100644
--- a/src/os/stat_windows.go
+++ b/src/os/stat_windows.go
@@ -103,7 +103,7 @@ func stat(funcname, name string, createFileAttrs uint32) (FileInfo, error) {
 
 // statNolog implements Stat for Windows.
 func statNolog(name string) (FileInfo, error) {
-	return stat("Stat", name, syscall.FILE_FLAG_BACKUP_SEMANTICS)
+	return stat("Stat", name, syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS)
 }
 
 // lstatNolog implements Lstat for Windows.

Note the distinction between IO_REPARSE_TAG_SYMLINK and io_reparse_tag_LX_symlink in IO_REPARSE_TAG_LX_SYMLINK (lx for 'linux' maybe)

[kfsone]

Notes from looking at implementing handling for this:

• Microsoft themselves have not fully supported this yet:

> cd $env:TEMP ; rm -recurs -force symlinks ; cd (mkdir symlinks)
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> wsl ln -s target.txt relative.lnk
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> echo this is target >target.txt
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> get-content target.txt
this
is
target
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> get-content relative.lnk
Get-Content: The file cannot be accessed by the system. : 'C:\Users\oliver\AppData\Local\Temp\symlinks\relative.lnk'
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> wsl cat relative.lnk
this
is
target
[C:\Users\oliver\AppData\Local\Temp\symlinks]
> fsutil reparsePoint query relative.lnk
Reparse Tag Value : 0xa000001d
Tag value: Microsoft
Tag value: Name Surrogate

Reparse Data Length: 0xe
Reparse Data:
0000:  02 00 00 00 74 61 72 67  65 74 2e 74 78 74        ....target.txt

• It also appears this does not use the same type of reparse buffer that non-lx reparses do:

• reactos handling: https://doxygen.reactos.org/d9/d78/reparse_8c_source.html

• cygwin: https://www.mail-archive.com/cygwin@cygwin.com/msg164048.html, https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=44da5e4b8c89bfcee79eb83fd3b7142720489940 (I'm puzzled by the fact that the buffer seems to be char-based instead of mbcs or utf16).

Perhaps for now, rather than appearing as a symlink, these entities should at least appear as not being regular files?

[kfsone]

@odeke-em Taking a look; previously signed the CLA for grpc in 2014, setup a password. The windows tests appear to be failing tho as a result of #40604 but I also experience a seemingly obvious failure of TestLookPath:

    lp_windows_test.go:147: test={rootDir:C:\Users\oliver\AppData\Local\Temp\TestLookPath336921111\d16 PATH:p1;p2 PATHEXT:.COM;.EXE;.BAT files:[p1\a.bat p1\a.exe p2\a.bat p2\a.exe] searchFor:a fails:false} failed: expected to find "p1\\a.bat", but found "p1\\a.exe"```

I'll ticket that separately, but the error looks backwards.

[gopherbot]

Change https://go.dev/cl/516555 mentions this issue: os: Stat/LStat to treat all name surrogates as symlinks on Windows

[qmuntal]

I don't think we should special-case WSL symlinks (aka IO_REPARSE_TAG_LX_SYMLINK). I haven't seen any important framework doing so, even WSL is trying to limit their occurrence by creating normal symlinks whenever possible to improve compatibility (see WSL 17406 release notes).

Also, WSL symlinks can't be opened from Windows, only from within WSL, so if we identify them as fs.ModeSymlink, then functions like filepath.Walk would fail when trying to follow the link.

@bcmills

[qmuntal]

This issue should be fixed since go1.21 thanks to CL 516555 and CL 460595.