New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

path/filepath: EvalSymlinks of container mapped directory returns an error #23512

Open
tescherm opened this Issue Jan 22, 2018 · 19 comments

Comments

Projects
None yet
6 participants
@tescherm

tescherm commented Jan 22, 2018

What version of Go are you using (go version)?

go version go1.9.2 windows/amd64 and go version go1.10beta2 windows/amd64.

Does this issue reproduce with the latest release?

Yes, this is reproducable with go 1.10beta2 and 1.9.2

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\ContainerAdministrator\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\gopath
set GORACE=
set GOROOT=C:\go
set GOTMPDIR=
set GOTOOLDIR=C:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\ContainerAdministrator\AppData\Local\Temp\go-build022882258
=/tmp/go-build -gno-record-gcc-switches

What did you do?

Running filepath.EvalSymlinks against a bind mounted directory within a windows container (for example, \\?\ContainerMappedDirectories\D48AFB19-F38F-4DAE-A698-4D718D506B15) returns the following error:

PS C:\Users\Administrator> docker run --rm -v c:\:c:\host golang:1.10-rc-nanoserver-sac2016 go run c:\host\eval_symlinks.go
panic: GetFileAttributesEx \ContainerMappedDirectories: The system cannot find the file specified.

goroutine 1 [running]:
main.main()
        c:/host/eval_symlinks.go:11 +0xe6
exit status 2

eval_symlinks.go above is the following program:

package main

import (
	"fmt"
	"path/filepath"
)

func main() {
	path, err := filepath.EvalSymlinks("c:\\host")
	if err != nil {
		panic(err)
	}

	fmt.Printf("eval path: %s\n", path)
}

Note that the symlink is valid. For example, I can cd to it:

PS C:\gopath> cmd /c dir c:\
 Volume in drive C has no label.
 Volume Serial Number is 8E1D-692C

 Directory of c:\

01/22/2018  01:34 PM    <DIR>          go
01/08/2018  10:34 AM    <DIR>          gopath
01/22/2018  01:34 PM    <SYMLINKD>     host [\\?\ContainerMappedDirectories\007C5A5F-50A3-4019-BCF1-1BA20F4745
11/20/2016  03:32 AM             1,894 License.txt
01/22/2018  01:34 PM    <DIR>          Program Files
07/16/2016  04:09 AM    <DIR>          Program Files (x86)
01/22/2018  01:34 PM    <DIR>          Users
01/22/2018  01:34 PM    <DIR>          Windows
               1 File(s)          1,894 bytes
               7 Dir(s)  21,256,376,320 bytes free
PS C:\gopath> cd \\?\ContainerMappedDirectories\007C5A5F-50A3-4019-BCF1-1BA20F4745F4
PS Microsoft.PowerShell.Core\FileSystem::\\?\ContainerMappedDirectories\007C5A5F-50A3-4019-BCF1-1BA20F4745F4>

golang 1.9.2 returns the same error:

PS C:\Users\Administrator> docker run --rm -v c:\:c:\host golang:1.9.2-nanoserver-sac2016 go run c:\host\eval_symlinks.go
panic: GetFileAttributesEx \ContainerMappedDirectories: The system cannot find the file specified.

goroutine 1 [running]:
main.main()
        c:/host/eval_symlinks.go:11 +0xfe
exit status 2

I see that EvalSymlinks on Windows was reimplemented recently (66c03d3), maybe there was a nuance wrt the \\?\ path style that was missed?

What did you expect to see?

I expected filepath.EvalSymlinks to return the original directory (C:\host) or the directory that was linked to (\\?\ContainerMappedDirectories\007C5A5F-50A3-4019-BCF1-1BA20F4745).

What did you see instead?

The error GetFileAttributesEx \ContainerMappedDirectories: The system cannot find the file specified.

@tklauser

This comment has been minimized.

Member

tklauser commented Jan 23, 2018

@tklauser tklauser added this to the Go1.11 milestone Jan 23, 2018

@tklauser tklauser added the OS-Windows label Jan 23, 2018

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jan 24, 2018

Thank you for the report. I will investigate this when I have time.

Alex

@tescherm

This comment has been minimized.

tescherm commented Jan 24, 2018

Thanks @alexbrainman let me know if I can provide any additional info.

@tescherm

This comment has been minimized.

tescherm commented Jan 25, 2018

I did a little more investigating. This following program also fails:

package main

import "os"
import "fmt"

func main() {
    path, err := os.Readlink("C:\\host")
    if err != nil {
        panic(fmt.Errorf("os.Readlink failed: %v", err))
    }

    println("readlink path", path)

    fi, err := os.Open(path)
    if err != nil {
        panic(fmt.Errorf("os.Open failed: %v", err))
    }

    defer fi.Close()

    println("file info", fi.Name())
}
readlink path \ContainerMappedDirectories\FF9714DB-37DA-4741-AC60-876DD8CFDFF2
panic: os.Open failed: open \ContainerMappedDirectories\FF9714DB-37DA-4741-AC60-876DD8CFDFF2: The system cannot find the path specified.

However if I prepend \\? to the path returned by os.Readlink, os.Open succeeds:

package main

import "os"
import "fmt"

func main() {
    path, err := os.Readlink("C:\\host")
    if err != nil {
        panic(fmt.Errorf("os.Readlink failed: %v", err))
    }

    println("readlink path", path)

    fi, err := os.Open("\\\\?" + path)
    if err != nil {
        panic(fmt.Errorf("os.Open failed: %v", err))
    }

    defer fi.Close()

    println("file info", fi.Name())
}
readlink path \ContainerMappedDirectories\846DD71E-ABA5-4F3B-93EB-0276148F0F98
file info \\?\ContainerMappedDirectories\846DD71E-ABA5-4F3B-93EB-0276148F0F98

Is is possible that os.Readlink on Windows doesn't handle the \\?\ path style?

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Jan 25, 2018

let me know if I can provide any additional info.

I just need to be able to reproduce your problem, and I will need Docker installed for that. What is the easiest way to install Docker if I have Windows 10?

You can also try and fix this yourself. If you try, you need to be aware about forthcoming change https://go-review.googlesource.com/#/c/go/+/86556/ that might affect your problem. Also Windows paths starting with \?\ are not handled very well in many Go standard library parts. For example path/filepath does not handle these paths at all. Also see issue #22230.

Is is possible that os.Readlink on Windows doesn't handle the \?\ path style?

os.Readlink is broken on Windows. os.Readlink current implementation does not work in many situations, and I do not know of a way to implement it properly. But lets not discuss os.Readlink here. Create new issue for os.Readlink, if you think it is important.

Alex

@tescherm

This comment has been minimized.

tescherm commented Jan 25, 2018

@alexbrainman you can install Docker for Windows 10 by following the instructions here. You will want to make sure that you switch to Windows Containers (the second step).

I'll see if I can spend some time looking at this as well.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Feb 11, 2018

@alexbrainman you can install Docker for Windows 10 by following the instructions here. You will want to make sure that you switch to Windows Containers (the second step).

When I run "Docker for Windows" app, I get this message

image

but I do not want to proceed, because I do use VirtualBox. I do not want to loose ability to run VirtualBox. What should I do?

Thank you.

Alex

@as

This comment has been minimized.

Contributor

as commented Feb 11, 2018

It is specifically the hardware virtualization extensions that will not work; Hyper-V acquires them before Windows boots and Virtualbox can't use them anymore. This can be restored by disabling the hyper-v service in services.msc.

I happen to have Docker Enterprise on a W10 system and can attempt to reproduce it if that workaround isn't possible

image

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Feb 11, 2018

It is specifically the hardware virtualization extensions that will not work; Hyper-V acquires them before Windows boots and Virtualbox can't use them anymore.

I discovered that much myself.

I happen to have Docker Enterprise on a W10 system and can attempt to reproduce it if that workaround isn't possible

I will try https://cloud.google.com/compute/docs/containers/#docker_on_windows when I have time.

Alex

@tescherm

This comment has been minimized.

tescherm commented Feb 12, 2018

@alexbrainman the link above should work on Windows Server 2016, without the need to disable Virtualbox.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Feb 13, 2018

@alexbrainman the link above should work on Windows Server 2016, without the need to disable Virtualbox.

If you are referring to https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-10 then it does not help me, because I only have Windows 10 computer, not Windows Server 2016,

If you are referring to https://cloud.google.com/compute/docs/containers/#docker_on_windows then yes, it should work. But I would need to create computer from scratch and install all the tools I need. So I will need some free time.

Alex

@kolyshkin

This comment has been minimized.

Contributor

kolyshkin commented Oct 9, 2018

@tescherm can you please re-check that the issue was fixed in Go 1.11 (e83601b)?

@tescherm

This comment has been minimized.

tescherm commented Oct 10, 2018

@kolyshkin I can confirm that this is still an issue. Running docker run --rm -v c:\:c:\host golang:1.11-nanoserver-sac2016 go run c:\host\eval_symlinks.go with the program above yields:

screen shot 2018-10-09 at 5 02 06 pm

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Oct 13, 2018

@tescherm I am not surprised that os.Readlink is broken on windows. But maybe path/filepath.EvalSymlinks is OK. Please, try this program instead

package main

import (
	"fmt"
	"path/filepath"
)

func main() {
	path, err := filepath.EvalSymlinks("c:\\host")
	if err != nil {
		panic(err)
	}

	fmt.Printf("eval path: %s\n", path)
}

If it still fails, someone would have to investigate where it fails. I still do not have Windows Docker installed on my computer - I use VirtualBox, and, apparently, you cannot have both. So I would need to configure new PC somewhere (probably in the cloud), and have Go development environment installed on it before I can even start investigating this. I have very little free time nowadays, but I will do that, if nothing else.

Thank you.

Alex

@tescherm

This comment has been minimized.

tescherm commented Oct 13, 2018

@alexbrainman I tried it out, unfortunately the program also fails on 1.11:

PS C:\> docker run --rm -v c:\:c:\host golang:1.10-nanoserver-sac2016 go run c:\host\eval_symlinks.go
panic: GetFileAttributesEx \ContainerMappedDirectories: The system cannot find the file specified.

goroutine 1 [running]:
main.main()
        c:/host/eval_symlinks.go:11 +0xf0
exit status 2
PS C:\> docker run --rm -v c:\:c:\host golang:1.11-nanoserver-sac2016 go run c:\host\eval_symlinks.go
panic: FindFirstFile \ContainerMappedDirectories: The system cannot find the file specified.

goroutine 1 [running]:
main.main()
        c:/host/eval_symlinks.go:11 +0xf0
exit status 2
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Oct 14, 2018

@tescherm thanks for checking. I would have to debug this myself when I have time.

Alex

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Oct 21, 2018

@tescherm

I managed to setup appropriate environment for your test. I build this program

package main

import (
	"fmt"
	"path/filepath"
)

func main() {
	path, err := filepath.EvalSymlinks("c:\\host")
	if err != nil {
		panic(err)
	}

	fmt.Printf("eval path: %s\n", path)
}

and copied resulting exe into c:\test.exe, and then run this command

docker run --rm -v c:\:c:\host microsoft/nanoserver c:\host\test.exe

I can reproduce your error now.

Looking at the code, we call Windows API GetFinalPathNameByHandle with c:\host and VOLUME_NAME_DOS inside the container, and that call fails with ERROR_FILE_NOT_FOUND. I tried all VOLUME_NAME_DOS alternatives, and only VOLUME_NAME_NT works - it converts c:\host into \\Device\\HarddiskVolume1\\.

If we adjust this code to return \\Device\\HarddiskVolume1\\ in your situation, would that help you? How? Please explain in details, what it would give you.

path/filepath.EvalSymlinks is started as code to follow symlinks on Unix. I am not sure what we have here is covered by that description.

Thank you.

Alex

@tescherm

This comment has been minimized.

tescherm commented Oct 21, 2018

@alexbrainman great, yes, I believe that returning \\Device\\HarddiskVolume1\\ in the example above would help. The goal would be that this path resolves with other fs system calls. For example:

package main

import "os"
import "fmt"

func main() {
    path, err := filepath.EvalSymlinks("c:\\host")
    if err != nil {
        panic(err)
    }

    fmt.Printf("eval path: %s\n", path)

    // open resolves correctly
    fi, err := os.Open(path)
    if err != nil {
        panic(fmt.Errorf("os.Open failed: %v", err))
    }

    defer fi.Close()

    println("file info", fi.Name())
}

Thanks again for drilling in.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Oct 21, 2018

I believe that returning \\Device\\HarddiskVolume1\\ in the example above would help. The goal would be that this path resolves with other fs system calls.

If we return \Device\HarddiskVolume1\, that name would refer to
directory inside you container. For example, if your current drive is
C:, then \Device\HarddiskVolume1\ would mean
C:\Device\HarddiskVolume1\. I don't think that is what you want. You
want C:\ outside of container.

The \Device\HarddiskVolume1\ returned by calling
GetFinalPathNameByHandle with VOLUME_NAME_NT, is not proper Windows
path. Search for "NT Namespaces" in
https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file
to understand what it is. I doubt that Windows CreateFile API (and other APIs that Go uses) can access "NT Namespaces" paths.

And, in #23512 (comment) , you did not explain why you need to to call filepath.EvalSymlinks? Why cannot you call os.Open("c:\host") instead?

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment