New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
os: RemoveAll hangs on a path longer than 260 characters on Windows versions before Windows 10 1607 #36375
Comments
Previously: #3358 |
@bsamek I can reproduce it here, thank you very much. The problem is that os.RemoveAll("foo") calls os.Remove(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`) which calls syscall.RemoveDirectory(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`) which fails, because syscalls don't allow for relative path to be longer than 256 chars. But syscall.RemoveDirectory should succeeds, because directory exists and can be deleted. Path is converted with os.fixLongPath in os.Remove, but os.fixLongPath does not handle relative path, and returns them as is. Alex |
Could os.RemoveAll() construct an absolute path and pass that to os.Remove() ? |
Change https://golang.org/cl/214437 mentions this issue: |
I don't think that fix is correct... |
Change https://golang.org/cl/214598 mentions this issue: |
Change https://golang.org/cl/214601 mentions this issue: |
Patch was reverted, so reopening issue. |
This reverts CL 214437. Does not fix the issue, and the test was wrong so it did not detect that it did not fix the issue. Updates #36375 Change-Id: I6a4112035a1e90f4fdafed6fdf4ec9dfc718b571 Reviewed-on: https://go-review.googlesource.com/c/go/+/214601 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Converting a relative path to absolute path on Windows is also discussed here. In this peculiar case, the end of the code can become until RemoveAll is improved:
Regarding RemoveAll, adding a Windows specific file is considered here but issue was closed after updating Unix-like builds. |
We can create a Windows-specific os.RemoveAll. Maybe it can Chdir before removing directory entries, instead of using path+\+file . The syscall package can be updated to fix bugs in the stdlib. Can you elaborate on your code segment above, where would that live? |
It does not seem that the
|
What happens for EDIT: It could also hang in the syscall within os.Lstat(). And within os.Chdir() if we tried that instead of "path+\+file". So we need a solution re long relative paths for the whole os package. We should at least return an error if a long path |
More here: microsoft/winfile#50 (see comments by @malxau) EDIT: and https://stackoverflow.com/questions/38036943/getfullpathnamew-and-long-windows-file-paths |
Indeed, full path names are required. The code below shows which error messages current RemoveAll receives. output is amended for readability
|
AFAIK, using full path is not enough to fix hanging. Lstat is required to know if the path is a directory. A file handle is also required to list directories using Readdirnames(). All add concurrency on the file structure. From Microsoft documentation of RemoveDirectory, recursion requires to use shfileoperation which should probably be added to |
Have you tested os.Remove with a long absolute path, both empty and non-empty? I thought that was verified to work. We don't need os.Remove to do recursion, we expect it to return an error if it's a non-empty directory. I suggested two possible solutions here: #21782 (comment) |
RemoveDirectory is using windows function RemoveDirectoryW. It says:
Prepending "\\?\" shouldn't be enough to fix this? |
Yes, I suggested that in the issue referenced above, which we should fix at the same time. Feel free to submit a patch. (Note that you have to sign a CLA first.) @gopherbot add "help wanted" |
Note that prepending
In the past when I've done this it becomes important to define some form of consistency. It'd be bad if "foo." evaluates to different files depending on the API being invoked. |
Good point. Maybe the fix is to prepend I think the same logic applies to filepath.Walk(). cc @bcmills |
As mentioned in the comment re filepath.Walk() that I linked above, we can prepend That needs to be done for the directory entries passed to RemoveAll(entry), as described in my previous post. |
Yes, sure. I've introduced a syscall.FullPath into fixLongPath, and now it works for most cases presented, but it is still failing in Walk, because the path that's failing is 227 characters long (<248), so it doesn't get processed by this. (This was commented in one of the first issues in filepath.Walk ticket, mentioned before). I have to investigate why this is failing, and fix this. Cheers ;) |
@quentinmit describes here why he didn't do that already: #21782 (comment) I don't think we can touch fixLongPath(), because everything calls it, and we could easily break some case for which there isn't a test. We only need to fix filepath.Walk() and os.RemoveAll(), and we already know how to do so. |
mmm.. But Is backwards logic I think. fixLongPath is called in places were a fix for MAX_PATH is needed, but it's not working for relative paths. But I think that this list is way more extensive. If we are not going to fix fixLongPath, we should be clear about this not working for relative paths, and functions like lstat, open or so, should be clear about not working for relative paths longer than 248 chars on windows. |
See #20829 re long (i.e. invalid) relative path arguments. Users can adjust arguments they pass. But a valid relative path can become invalid by concatenation during a tree walk. So we just need to fix what os.RemoveAll() passes to RemoveAll() (and likewise for filepath.Walk()) by defining a parent from GetFullPathNameW() with prepended |
But ticket #20829 was fixed adding fixLongPath(name): Also, if we are not going to support for users to give us a relative path longer than MAX_PATH, then we should be returning a special error for that, instead of "not existent file/folder", that's not necesarily true |
Read the whole discussion therein. Changing fixLongPath() to return an error for long relative paths is worth considering, but out of scope for this issue. |
We are already weeks into the code freeze, moving to Go1.16. |
Change https://golang.org/cl/263538 mentions this issue: |
There is a CL for this change, but it didn’t make it in early. Punting to Go1.17. |
Change https://golang.org/cl/291291 mentions this issue: |
Windows 10 >= 1607 allows CreateFile and friends to use long paths if bit 0x80 of the PEB's BitField member is set. In time this means we'll be able to entirely drop our long path hacks, which have never really worked right (see bugs below). Until that point, we'll simply have things working well on recent Windows. Updates #41734. Updates #21782. Updates #36375. Change-Id: I765de6ea4859dd4e4b8ca80af7f337994734118e Reviewed-on: https://go-review.googlesource.com/c/go/+/291291 Trust: Jason A. Donenfeld <Jason@zx2c4.com> Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 291291 was merged. What else is there to do for this issue? Thanks. |
We could probably go in circles on this for a long time trying to handle what is now "old Windows", but given that it's no longer a problem into the future with Windows 10 >= 1607, maybe we can close this and not worry about the old stuff so much? That's a little less conservative than the Go project usually likes to be, but maybe it's not an all together bad approach for this? |
I suggest leaving this open as long as earlier Windows versions are supported. |
I just checked the program above #36375 (comment) now runs without any failures. So this issue is fixed. Probably by CL 291291. This is still broken for older versions of Windows. But, I agree with Jason. I don't see anyone working to fix this issue on older Windows. So I would close the issue. But I would let Ian to decide. Alex |
The fix is quite simple, and described here: #36375 (comment) cc @rasky |
CC @bufflig |
Based on the conversation, I understand this is fixed for newer Windows, but not older. It seems this might not make it to 1.17, so I'll keep the issue open but move it to Backlog for now. In a comment above, @networkimprov suggests there's a simple fix that can be evaluated. @bufflig, if you have a chance to look at this before beta 1, please feel free to move it back to Go 1.17 milestone. |
On Go 1.13.5 on Windows 2008 R2, running os.RemoveAll on a directory that contains a path longer than 260 characters hangs.
In the following repro, the directory is created, and then the program hangs on os.RemoveAll.
The text was updated successfully, but these errors were encountered: