Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: RemoveAll hangs on a path longer than 260 characters on Windows #36375

Open
bsamek opened this issue Jan 3, 2020 · 14 comments
Open

os: RemoveAll hangs on a path longer than 260 characters on Windows #36375

bsamek opened this issue Jan 3, 2020 · 14 comments

Comments

@bsamek
Copy link

@bsamek bsamek commented Jan 3, 2020

On Go 1.13.5 on Windows 2008 R2, running os.RemoveAll on a directory that contains a path longer than 260 characters hangs.

In the following repro, the directory is created, and then the program hangs on os.RemoveAll.

package main

import (
	"log"
	"os"
	"path/filepath"
)

func main() {
	// make a long path
	a := ""
	b := ""
	for i := 0; i < 150; i++ {
		a += "a"
		b += "b"
	}
	wd, err := os.Getwd()
	if err != nil {
		log.Fatal(err)
	}
	err = os.MkdirAll(filepath.Join(wd, "foo", "bar", a, b), 0755)
	if err != nil {
		log.Fatal(err)
	}

	// remove the root of the long path
	err = os.RemoveAll("foo")
	if err != nil {
		log.Fatal(err)
	}
}
@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Jan 3, 2020

Previously: #3358

@ALTree ALTree changed the title os.RemoveAll hangs on a path longer than 260 characters on Windows os: RemoveAll hangs on a path longer than 260 characters on Windows Jan 3, 2020
@ALTree ALTree added this to the Go1.15 milestone Jan 3, 2020
@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Jan 4, 2020

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Jan 4, 2020

@bsamek I can reproduce it here, thank you very much.

The problem is that

os.RemoveAll("foo")

calls

os.Remove(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`)

which calls

syscall.RemoveDirectory(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`)

which fails, because syscalls don't allow for relative path to be longer than 256 chars. But syscall.RemoveDirectory should succeeds, because directory exists and can be deleted.

Path is converted with os.fixLongPath in os.Remove, but os.fixLongPath does not handle relative path, and returns them as is.

Alex

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Jan 4, 2020

Could os.RemoveAll() construct an absolute path and pass that to os.Remove() ?

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jan 12, 2020

Change https://golang.org/cl/214437 mentions this issue: os: handle long path in RemoveAll for windows

@gopherbot gopherbot closed this in 5c44cc4 Jan 13, 2020
@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Jan 13, 2020

I don't think that fix is correct...

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jan 13, 2020

Change https://golang.org/cl/214598 mentions this issue: os: actually remove long path in TestRemoveAll

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jan 13, 2020

Change https://golang.org/cl/214601 mentions this issue: Revert "os: handle long path in RemoveAll for windows"

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jan 13, 2020

Patch was reverted, so reopening issue.

gopherbot pushed a commit that referenced this issue Jan 13, 2020
This reverts CL 214437.

Does not fix the issue, and the test was wrong so it did not detect that it did not fix the issue.

Updates #36375

Change-Id: I6a4112035a1e90f4fdafed6fdf4ec9dfc718b571
Reviewed-on: https://go-review.googlesource.com/c/go/+/214601
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@iWdGo

This comment has been minimized.

Copy link
Contributor

@iWdGo iWdGo commented Jan 18, 2020

Converting a relative path to absolute path on Windows is also discussed here.
AFAIK, syscall package is frozen for Windows and only existing calls are available.

In this peculiar case, the end of the code can become until RemoveAll is improved:

// remove the root of the long path
	fp, err := filepath.Abs("foo")
	if err != nil {
		log.Fatalln("abs:", err)
	}
	err = os.RemoveAll(fp)
	if err != nil {
		log.Fatal(err)
	}

Regarding RemoveAll, adding a Windows specific file is considered here but issue was closed after updating Unix-like builds.

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Jan 18, 2020

We can create a Windows-specific os.RemoveAll. Maybe it can Chdir before removing directory entries, instead of using path+\+file .

The syscall package can be updated to fix bugs in the stdlib.

Can you elaborate on your code segment above, where would that live?

@iWdGo

This comment has been minimized.

Copy link
Contributor

@iWdGo iWdGo commented Jan 26, 2020

It does not seem that the syscall package has an issue as related syscall methods fail appropriately when using the example. RemoveAll hangs probably because of some recursion issue. Handling some of the possible errors requires probably to be tailored to Windows behavior.

package main

import (
	"log"
	"os"
	"path/filepath"
	"strings"
	"syscall"
)

func main() {
	// make a long path
	wd, err := os.Getwd()
	if err != nil {
		log.Fatal(err)
	}
	// relative creation fails on a path longer than MAX_PATH
	fp := filepath.Join(wd, "foo", "bar", strings.Repeat("a", 150), strings.Repeat("b", 150))
	err = os.MkdirAll(fp, 0755)
	if err != nil {
		log.Fatal(err)
	}

	// remove the root of the long path using syscall
	p, e := syscall.UTF16PtrFromString("foo")
	if e != nil {
		log.Fatalln(e)
	}
	err = syscall.RemoveDirectory(p)
	if err == syscall.ERROR_DIR_NOT_EMPTY {
		log.Println("relative RemoveDirectory failed as expected as directory is not empty")
	} else if err != nil {
		log.Println(err)
	}

	// Moving to full path
	fps, ef := syscall.FullPath("foo")
	if ef != nil {
		log.Fatalln(ef)
	}
	if fps != fp[0:len(fps)] {
		log.Fatalf("fullpath: got %s, want %s", fps, fp[0:len(fps)])
	}

	p, e = syscall.UTF16PtrFromString(fps)
	if e != nil {
		log.Fatalln(e)
	}
	err = syscall.RemoveDirectory(p)
	if err == syscall.ERROR_DIR_NOT_EMPTY {
		log.Println("absolute RemoveDirectory failed as expected as directory is not empty")
	} else if err != nil {
		log.Fatal(err)
	}
}
@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Jan 27, 2020

What happens for syscall.RemoveDirectory("foo\bar\a...\b...") ? I think that's where the hang must occur, since os.RemoveAll() tries that path for os.RemoveAll("foo")

EDIT: It could also hang in the syscall within os.Lstat(). And within os.Chdir() if we tried that instead of "path+\+file". So we need a solution re long relative paths for the whole os package. We should at least return an error if a long path cannot be made absolute doesn't start with \\?\. Maybe some help can be found here:
https://googleprojectzero.blogspot.com/2016/02/the-definitive-guide-on-win32-to-nt.html

@networkimprov

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.