Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: ReadDir on regular file returns ERROR_PATH_NOT_FOUND instead of ENOTDIR #46734

Open
codefromthecrypt opened this issue Jun 14, 2021 · 16 comments
Open
Labels
NeedsInvestigation OS-Windows
Milestone

Comments

@codefromthecrypt
Copy link

@codefromthecrypt codefromthecrypt commented Jun 14, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.5 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\moo\AppData\Local\go-build
set GOENV=C:\Users\moo\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\moo\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\moo\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\moo\.gimme\versions\go1.16.5.windows.amd64
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\moo\.gimme\versions\go1.16.5.windows.amd64\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.5
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\moo\oss\getenvoy\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\moo\AppData\Local\Temp\go-build3391750208=/tmp/go-build -gno-record-gcc-switches

What did you do?

	files, err := os.ReadDir(filePath)

What did you expect to see?

err should be ENOTDIR

What did you see instead?

os.IsNotExist(err) matches because it was ERROR_PATH_NOT_FOUND

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Jun 14, 2021

the problem is here

if !file.isdir() {
the state check isn't valid for the call site

a lazy double-check like darwin is one way out, but up to you!

if f.dirinfo == nil {

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Jun 14, 2021

specifically, in windows isdir is only valid if the file was opened as a directory, which isn't the case when there was a file there https://github.com/golang/go/blob/master/src/os/file_windows.go#L102

@seankhliao seankhliao changed the title Windows os.ReadDir returns ERROR_PATH_NOT_FOUND instead of ENOTDIR os: ReadDir returns ERROR_PATH_NOT_FOUND instead of ENOTDIR Jun 14, 2021
@seankhliao seankhliao added NeedsInvestigation OS-Windows labels Jun 14, 2021
@networkimprov
Copy link

@networkimprov networkimprov commented Jun 21, 2021

cc @alexbrainman @bcmills @zx2c4 @mattn

@dmitshur re incomplete triage

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Jun 22, 2021

What did you do?

	files, err := os.ReadDir(filePath)

That is not enough to reproduce this issue.

@codefromthecrypt can you, please, provide complete program with steps to reproduce?

Thank you.

Alex

@iwdgo
Copy link
Contributor

@iwdgo iwdgo commented Dec 25, 2021

Reading the line below indicates that ENOTDIR is already returned.

go/src/os/dir_windows.go

Lines 14 to 15 in 2ebe77a

if !file.isdir() {
return nil, nil, nil, &PathError{Op: "readdir", Path: file.name, Err: syscall.ENOTDIR}

This error could be checked with like recommended since #41122 was closed. documentation of os.IsNotExist(err) was improved. It recommends to use errors.Is(err, fs.ErrNotExist) on new code.

if errors.Is(err, syscall.ENOTDIR) {
    // Error handling
}

If walking the directory is the issue, then fs.Walkdir was added in go1.17.

It seems that there is nothing left to do for this issue.

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Dec 26, 2021

@iwdgo the comments you made rely on file.isdir working. Specifically, check the notes here, which are that file.isdir is the problem #46734 (comment)

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Dec 27, 2021

Here's the unit test, and on windows GitHub actions runner, it fails like this, but only on windows (doesn't fail on linux or macOS). Feel free too do whatever you want, just I owed a unit test from earlier. If you think this is ok, just close the issue.

--- FAIL: TestDirError (0.00s)
    dir_test.go:25: didn't want os.IsNotExist() on os.ReadDir(C:\Users\RUNNER~1\AppData\Local\Temp\TestDirError275160222\001/foo)
func TestDirError(t *testing.T) {
	dir := t.TempDir()
	fileNotDir := path.Join(dir, "foo")
	if os.WriteFile(fileNotDir, []byte{}, 0600) != nil {
		t.Fatalf("cannot write %s", fileNotDir)
	}
	_, err := os.ReadDir(fileNotDir)
	if err == nil {
		t.Fatalf("wanted an error on os.ReadDir(%s)", fileNotDir)
	}
	pErr, ok := err.(*os.PathError)
	if !ok {
		t.Fatalf("wanted an os.PathError on os.ReadDir(%s)", fileNotDir)
	}
	if os.IsNotExist(err) {
		t.Fatalf("didn't want os.IsNotExist() on os.ReadDir(%s)", fileNotDir)
	}
	if pErr.Err != syscall.ENOTDIR {
		t.Fatalf("want syscall.ENOTDIR, have %s, on os.ReadDir(%s)", pErr.Err, fileNotDir)
	}
}

@dmitshur dmitshur changed the title os: ReadDir returns ERROR_PATH_NOT_FOUND instead of ENOTDIR os: ReadDir on regular file returns ERROR_PATH_NOT_FOUND instead of ENOTDIR Dec 27, 2021
@dmitshur dmitshur added this to the Backlog milestone Dec 27, 2021
@iwdgo
Copy link
Contributor

@iwdgo iwdgo commented Dec 29, 2021

@codefromthecrypt Thanks. The provided test fails too on windows 10 21H1.
The discussed isdir() is in line with the purpose.

func (f *file) isdir() bool { return f != nil && f.dirinfo != nil }

Some directory data is found which lets readdir() occur with a handle that triggers a misleading error. IMHO, there is something to fix. It seems related to the use of a temporary directory following some investigation.
My first idea was erroneous as this part is compile early in the toolchain and some errors are not reported as later in the toolchain.

@iwdgo
Copy link
Contributor

@iwdgo iwdgo commented Dec 30, 2021

Using == for type error is not recommended. Using something like below provides details on what is happening:

// Testing ENOTDIR first provides the detailed cause of the failure.
if !errors.Is(pErr.Err, syscall.ENOTDIR) {
		t.Fatalf("want syscall.ENOTDIR, have %s, on os.ReadDir(%s)", pErr.Err, fileNotDir)
}
if os.IsNotExist(err) {
		t.Fatalf("didn't want os.IsNotExist() on os.ReadDir(%s)", fileNotDir)
}

IsNotExist(err) reports true as the directory is not found which is ERROR_PATH_NOT_FOUND on windows.
It seems that issue can be closed.

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Dec 31, 2021

I think the investigation so far feels like someone trying hard to prove the current code is correct, but wholly missing that the overall experience isn't consistent. I would suggest another person take a look even if they also think the experience is ok to sacrifice and that the result should be telling people one by one that they shouldn't call == on error (which is done incidentally in tests etc).

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Dec 31, 2021

To put it concretely, I believe it is an experience fail to wrap errors only on windows for the same operation. Consistency is important and reduces time loss. It is true someone can say the user is wrong, but there is probably also some value in being direct and consistent in core APIs including error depth. I think this is the key issue and if someone else also thinks through this and decides it is better to close it anyway, go for it, just I would like consistency to be considered. that alone could make the efforts here worth something above zero.

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented Dec 31, 2021

This sounds like a real bug to me. Windows returns ERROR_PATH_NOT_FOUND in some cases and ERROR_FILE_NOT_FOUND in others. It seems like there's one case of ERROR_PATH_NOT_FOUND that we should be catching and turning into an ENOTDIR? If so, do you know where in the stack that check might be missing? Alternatively, should we be updating a os.IsASomething function to check for ERROR_PATH_NOT_FOUND? In which case, do you know off hand which one?

@iwdgo
Copy link
Contributor

@iwdgo iwdgo commented Dec 31, 2021

The question seems broader than ReadDir(). It is true that the provided code behaves differently on windows and *ix as Go playground shows: https://go.dev/play/p/GM8_PX2CezK

The error mapping for windows is here:

case oserror.ErrNotExist:
return e == ERROR_FILE_NOT_FOUND ||
e == _ERROR_BAD_NETPATH ||
e == ERROR_PATH_NOT_FOUND
}

And for unix,

case oserror.ErrNotExist:
return e == ENOENT

AFAIK, ERROR_FILE_NOT_FOUND is what is returned when an exact match of a file name is missing
So, ReadDir() is consistent between windows and *ix when returning ENOTDIR, but the test IsNotExist() is not.

The mapping of errors from windows to Go API is:

ENOENT Errno = ERROR_FILE_NOT_FOUND
ENOTDIR Errno = ERROR_PATH_NOT_FOUND

A solution is to remove ERROR_PATH_NOT_FOUND from ErrNotExist. It would restore some consistency. I do not know if it can be considered.

@zx2c4
Copy link
Contributor

@zx2c4 zx2c4 commented Dec 31, 2021

A solution is to remove ERROR_PATH_NOT_FOUND from ErrNotExist. It would restore some consistency. I do not know if it can be considered.

I'm not sure that works, though, because I'm pretty sure you get PATH_NOT_FOUND if you try to open "path/notexist/file", but you get FILE_NOT_FOUND if you try to open "path/dir/notexist". In both cases, there, ErrNotExist is the correct interpretation of results.

@bcmills
Copy link
Member

@bcmills bcmills commented Jan 5, 2022

Mapping PATH_NOT_FOUND to ENOTDIR seems suspect. “path not found” sounds semantically different from “path exists and is not a directory”.

@mvdan
Copy link
Member

@mvdan mvdan commented Mar 14, 2022

I've just ran into this: I am calling ioutil.ReadDir on a path, and then using errors.Is(err, fs.ErrNotExist) to notice when there's no file at that path. However, on Windows, that misbehaves, as it's also true when the file exists but is just a regular file, not a directory.

I agree with @bcmills and @zx2c4 that we can't simply change how ErrNotExist works, because PATH_NOT_FOUND is valid in some cases. However, in the already mentioned src/os/dir_windows.go we have:

func (file *File) readdir(n int, mode readdirMode) (names []string, dirents []DirEntry, infos []FileInfo, err error) {
    if !file.isdir() {
        return nil, nil, nil, &PathError{Op: "readdir", Path: file.name, Err: syscall.ENOTDIR}
    }

Which is fine on its own, except for the fact that ENOTDIR Errno = ERROR_PATH_NOT_FOUND. Could we not assign ENOTDIR to another value, so that it doesn't become one with PATH_NOT_FOUND? It's only returned in a few APIs in the os and path/filepath packages, and I can't imagine why we would want them to satisfy ErrNotExist on Windows.

Also FYI, I think cmd/go/internal/fsys has the same bug, as it similarly relies on IsNotExist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation OS-Windows
Projects
None yet
Development

No branches or pull requests

9 participants