os: ReadDir on regular file returns ERROR_PATH_NOT_FOUND instead of ENOTDIR

[codefromthecrypt]

What version of Go are you using (go version)?

$ go version
go version go1.16.5 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\moo\AppData\Local\go-build
set GOENV=C:\Users\moo\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\moo\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\moo\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Users\moo\.gimme\versions\go1.16.5.windows.amd64
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Users\moo\.gimme\versions\go1.16.5.windows.amd64\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.5
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\moo\oss\getenvoy\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\moo\AppData\Local\Temp\go-build3391750208=/tmp/go-build -gno-record-gcc-switches

What did you do?

	files, err := os.ReadDir(filePath)

What did you expect to see?

err should be ENOTDIR

What did you see instead?

os.IsNotExist(err) matches because it was ERROR_PATH_NOT_FOUND

[codefromthecrypt]

the problem is here

go/src/os/dir_windows.go

Line 14 in 2ebe77a

if !file.isdir() {

the state check isn't valid for the call site

a lazy double-check like darwin is one way out, but up to you!

go/src/os/dir_darwin.go

Line 28 in 2ebe77a

if f.dirinfo == nil {

[codefromthecrypt]

specifically, in windows isdir is only valid if the file was opened as a directory, which isn't the case when there was a file there https://github.com/golang/go/blob/master/src/os/file_windows.go#L102

[networkimprov]

cc @alexbrainman @bcmills @zx2c4 @mattn

@dmitshur re incomplete triage

[alexbrainman]

What did you do?

	files, err := os.ReadDir(filePath)

That is not enough to reproduce this issue.

@codefromthecrypt can you, please, provide complete program with steps to reproduce?

Thank you.

Alex

[iwdgo]

Reading the line below indicates that ENOTDIR is already returned.

go/src/os/dir_windows.go

Lines 14 to 15 in 2ebe77a

	if !file.isdir() {
	return nil, nil, nil, &PathError{Op: "readdir", Path: file.name, Err: syscall.ENOTDIR}

This error could be checked with like recommended since #41122 was closed. documentation of os.IsNotExist(err) was improved. It recommends to use errors.Is(err, fs.ErrNotExist) on new code.

if errors.Is(err, syscall.ENOTDIR) {
    // Error handling
}

If walking the directory is the issue, then fs.Walkdir was added in go1.17.

It seems that there is nothing left to do for this issue.

[codefromthecrypt]

@iwdgo the comments you made rely on file.isdir working. Specifically, check the notes here, which are that file.isdir is the problem #46734 (comment)

[codefromthecrypt]

Here's the unit test, and on windows GitHub actions runner, it fails like this, but only on windows (doesn't fail on linux or macOS). Feel free too do whatever you want, just I owed a unit test from earlier. If you think this is ok, just close the issue.

--- FAIL: TestDirError (0.00s)
    dir_test.go:25: didn't want os.IsNotExist() on os.ReadDir(C:\Users\RUNNER~1\AppData\Local\Temp\TestDirError275160222\001/foo)

func TestDirError(t *testing.T) {
	dir := t.TempDir()
	fileNotDir := path.Join(dir, "foo")
	if os.WriteFile(fileNotDir, []byte{}, 0600) != nil {
		t.Fatalf("cannot write %s", fileNotDir)
	}
	_, err := os.ReadDir(fileNotDir)
	if err == nil {
		t.Fatalf("wanted an error on os.ReadDir(%s)", fileNotDir)
	}
	pErr, ok := err.(*os.PathError)
	if !ok {
		t.Fatalf("wanted an os.PathError on os.ReadDir(%s)", fileNotDir)
	}
	if os.IsNotExist(err) {
		t.Fatalf("didn't want os.IsNotExist() on os.ReadDir(%s)", fileNotDir)
	}
	if pErr.Err != syscall.ENOTDIR {
		t.Fatalf("want syscall.ENOTDIR, have %s, on os.ReadDir(%s)", pErr.Err, fileNotDir)
	}
}

[iwdgo]

@codefromthecrypt Thanks. The provided test fails too on windows 10 21H1.
The discussed isdir() is in line with the purpose.

go/src/os/file_windows.go

Line 102 in 91e7821

func (f *file) isdir() bool { return f != nil && f.dirinfo != nil }

Some directory data is found which lets readdir() occur with a handle that triggers a misleading error. IMHO, there is something to fix. It seems related to the use of a temporary directory following some investigation.
My first idea was erroneous as this part is compile early in the toolchain and some errors are not reported as later in the toolchain.

[iwdgo]

Using == for type error is not recommended. Using something like below provides details on what is happening:

// Testing ENOTDIR first provides the detailed cause of the failure.
if !errors.Is(pErr.Err, syscall.ENOTDIR) {
		t.Fatalf("want syscall.ENOTDIR, have %s, on os.ReadDir(%s)", pErr.Err, fileNotDir)
}
if os.IsNotExist(err) {
		t.Fatalf("didn't want os.IsNotExist() on os.ReadDir(%s)", fileNotDir)
}

IsNotExist(err) reports true as the directory is not found which is ERROR_PATH_NOT_FOUND on windows.
It seems that issue can be closed.

[codefromthecrypt]

I think the investigation so far feels like someone trying hard to prove the current code is correct, but wholly missing that the overall experience isn't consistent. I would suggest another person take a look even if they also think the experience is ok to sacrifice and that the result should be telling people one by one that they shouldn't call == on error (which is done incidentally in tests etc).

[codefromthecrypt]

To put it concretely, I believe it is an experience fail to wrap errors only on windows for the same operation. Consistency is important and reduces time loss. It is true someone can say the user is wrong, but there is probably also some value in being direct and consistent in core APIs including error depth. I think this is the key issue and if someone else also thinks through this and decides it is better to close it anyway, go for it, just I would like consistency to be considered. that alone could make the efforts here worth something above zero.

[zx2c4]

This sounds like a real bug to me. Windows returns ERROR_PATH_NOT_FOUND in some cases and ERROR_FILE_NOT_FOUND in others. It seems like there's one case of ERROR_PATH_NOT_FOUND that we should be catching and turning into an ENOTDIR? If so, do you know where in the stack that check might be missing? Alternatively, should we be updating a os.IsASomething function to check for ERROR_PATH_NOT_FOUND? In which case, do you know off hand which one?

[iwdgo]

The question seems broader than ReadDir(). It is true that the provided code behaves differently on windows and *ix as Go playground shows: https://go.dev/play/p/GM8_PX2CezK

The error mapping for windows is here:

go/src/syscall/syscall_windows.go

Lines 155 to 159 in 6178d25

	case oserror.ErrNotExist:
	return e == ERROR_FILE_NOT_FOUND ||
	e == _ERROR_BAD_NETPATH ||
	e == ERROR_PATH_NOT_FOUND
	}

And for unix,

go/src/syscall/syscall_unix.go

Lines 133 to 134 in 6178d25

	case oserror.ErrNotExist:
	return e == ENOENT

AFAIK, ERROR_FILE_NOT_FOUND is what is returned when an exact match of a file name is missing
So, ReadDir() is consistent between windows and *ix when returning ENOTDIR, but the test IsNotExist() is not.

The mapping of errors from windows to Go API is:

go/src/syscall/zerrors_windows.go

Lines 8 to 9 in 6178d25

	ENOENT Errno = ERROR_FILE_NOT_FOUND
	ENOTDIR Errno = ERROR_PATH_NOT_FOUND

A solution is to remove ERROR_PATH_NOT_FOUND from ErrNotExist. It would restore some consistency. I do not know if it can be considered.

[zx2c4]

A solution is to remove ERROR_PATH_NOT_FOUND from ErrNotExist. It would restore some consistency. I do not know if it can be considered.

I'm not sure that works, though, because I'm pretty sure you get PATH_NOT_FOUND if you try to open "path/notexist/file", but you get FILE_NOT_FOUND if you try to open "path/dir/notexist". In both cases, there, ErrNotExist is the correct interpretation of results.

[bcmills]

Mapping PATH_NOT_FOUND to ENOTDIR seems suspect. “path not found” sounds semantically different from “path exists and is not a directory”.

[mvdan]

I've just ran into this: I am calling ioutil.ReadDir on a path, and then using errors.Is(err, fs.ErrNotExist) to notice when there's no file at that path. However, on Windows, that misbehaves, as it's also true when the file exists but is just a regular file, not a directory.

I agree with @bcmills and @zx2c4 that we can't simply change how ErrNotExist works, because PATH_NOT_FOUND is valid in some cases. However, in the already mentioned src/os/dir_windows.go we have:

func (file *File) readdir(n int, mode readdirMode) (names []string, dirents []DirEntry, infos []FileInfo, err error) {
    if !file.isdir() {
        return nil, nil, nil, &PathError{Op: "readdir", Path: file.name, Err: syscall.ENOTDIR}
    }

Which is fine on its own, except for the fact that ENOTDIR Errno = ERROR_PATH_NOT_FOUND. Could we not assign ENOTDIR to another value, so that it doesn't become one with PATH_NOT_FOUND? It's only returned in a few APIs in the os and path/filepath packages, and I can't imagine why we would want them to satisfy ErrNotExist on Windows.

Also FYI, I think cmd/go/internal/fsys has the same bug, as it similarly relies on IsNotExist.