os: RemoveAll hangs on a path longer than 260 characters on Windows

[bsamek]

On Go 1.13.5 on Windows 2008 R2, running os.RemoveAll on a directory that contains a path longer than 260 characters hangs.

In the following repro, the directory is created, and then the program hangs on os.RemoveAll.

package main

import (
	"log"
	"os"
	"path/filepath"
)

func main() {
	// make a long path
	a := ""
	b := ""
	for i := 0; i < 150; i++ {
		a += "a"
		b += "b"
	}
	wd, err := os.Getwd()
	if err != nil {
		log.Fatal(err)
	}
	err = os.MkdirAll(filepath.Join(wd, "foo", "bar", a, b), 0755)
	if err != nil {
		log.Fatal(err)
	}

	// remove the root of the long path
	err = os.RemoveAll("foo")
	if err != nil {
		log.Fatal(err)
	}
}

[ALTree]

Previously: #3358

[networkimprov]

cc @zx2c4 @alexbrainman @mattn

[alexbrainman]

@bsamek I can reproduce it here, thank you very much.

The problem is that

os.RemoveAll("foo")

calls

os.Remove(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`)

which calls

syscall.RemoveDirectory(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`)

which fails, because syscalls don't allow for relative path to be longer than 256 chars. But syscall.RemoveDirectory should succeeds, because directory exists and can be deleted.

Path is converted with os.fixLongPath in os.Remove, but os.fixLongPath does not handle relative path, and returns them as is.

Alex

[networkimprov]

Could os.RemoveAll() construct an absolute path and pass that to os.Remove() ?

[gopherbot]

Change https://golang.org/cl/214437 mentions this issue: os: handle long path in RemoveAll for windows

[networkimprov]

I don't think that fix is correct...

[gopherbot]

Change https://golang.org/cl/214598 mentions this issue: os: actually remove long path in TestRemoveAll

[gopherbot]

Change https://golang.org/cl/214601 mentions this issue: Revert "os: handle long path in RemoveAll for windows"

[ianlancetaylor]

Patch was reverted, so reopening issue.

[iWdGo]

Converting a relative path to absolute path on Windows is also discussed here.
AFAIK, syscall package is frozen for Windows and only existing calls are available.

In this peculiar case, the end of the code can become until RemoveAll is improved:

// remove the root of the long path
	fp, err := filepath.Abs("foo")
	if err != nil {
		log.Fatalln("abs:", err)
	}
	err = os.RemoveAll(fp)
	if err != nil {
		log.Fatal(err)
	}

Regarding RemoveAll, adding a Windows specific file is considered here but issue was closed after updating Unix-like builds.

[networkimprov]

We can create a Windows-specific os.RemoveAll. Maybe it can Chdir before removing directory entries, instead of using path+\+file .

The syscall package can be updated to fix bugs in the stdlib.

Can you elaborate on your code segment above, where would that live?

[iWdGo]

It does not seem that the syscall package has an issue as related syscall methods fail appropriately when using the example. RemoveAll hangs probably because of some recursion issue. Handling some of the possible errors requires probably to be tailored to Windows behavior.

package main

import (
	"log"
	"os"
	"path/filepath"
	"strings"
	"syscall"
)

func main() {
	// make a long path
	wd, err := os.Getwd()
	if err != nil {
		log.Fatal(err)
	}
	// relative creation fails on a path longer than MAX_PATH
	fp := filepath.Join(wd, "foo", "bar", strings.Repeat("a", 150), strings.Repeat("b", 150))
	err = os.MkdirAll(fp, 0755)
	if err != nil {
		log.Fatal(err)
	}

	// remove the root of the long path using syscall
	p, e := syscall.UTF16PtrFromString("foo")
	if e != nil {
		log.Fatalln(e)
	}
	err = syscall.RemoveDirectory(p)
	if err == syscall.ERROR_DIR_NOT_EMPTY {
		log.Println("relative RemoveDirectory failed as expected as directory is not empty")
	} else if err != nil {
		log.Println(err)
	}

	// Moving to full path
	fps, ef := syscall.FullPath("foo")
	if ef != nil {
		log.Fatalln(ef)
	}
	if fps != fp[0:len(fps)] {
		log.Fatalf("fullpath: got %s, want %s", fps, fp[0:len(fps)])
	}

	p, e = syscall.UTF16PtrFromString(fps)
	if e != nil {
		log.Fatalln(e)
	}
	err = syscall.RemoveDirectory(p)
	if err == syscall.ERROR_DIR_NOT_EMPTY {
		log.Println("absolute RemoveDirectory failed as expected as directory is not empty")
	} else if err != nil {
		log.Fatal(err)
	}
}

[networkimprov]

What happens for syscall.RemoveDirectory("foo\bar\a...\b...") ? I think that's where the hang must occur, since os.RemoveAll() tries that path for os.RemoveAll("foo")

EDIT: It could also hang in the syscall within os.Lstat(). And within os.Chdir() if we tried that instead of "path+\+file". So we need a solution re long relative paths for the whole os package. We should at least return an error if a long path cannot be made absolute doesn't start with \\?\. Maybe some help can be found here:
https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html

[networkimprov]

More here: microsoft/winfile#50 (see comments by @malxau)

EDIT: and https://stackoverflow.com/questions/38036943/getfullpathnamew-and-long-windows-file-paths