os: RemoveAll hangs on a path longer than 260 characters on Windows #36375
Comments
Previously: #3358 |
@bsamek I can reproduce it here, thank you very much. The problem is that os.RemoveAll("foo") calls os.Remove(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`) which calls syscall.RemoveDirectory(`foo\bar\aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb`) which fails, because syscalls don't allow for relative path to be longer than 256 chars. But syscall.RemoveDirectory should succeeds, because directory exists and can be deleted. Path is converted with os.fixLongPath in os.Remove, but os.fixLongPath does not handle relative path, and returns them as is. Alex |
Could os.RemoveAll() construct an absolute path and pass that to os.Remove() ? |
Change https://golang.org/cl/214437 mentions this issue: |
I don't think that fix is correct... |
Change https://golang.org/cl/214598 mentions this issue: |
Change https://golang.org/cl/214601 mentions this issue: |
Patch was reverted, so reopening issue. |
This reverts CL 214437. Does not fix the issue, and the test was wrong so it did not detect that it did not fix the issue. Updates #36375 Change-Id: I6a4112035a1e90f4fdafed6fdf4ec9dfc718b571 Reviewed-on: https://go-review.googlesource.com/c/go/+/214601 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Converting a relative path to absolute path on Windows is also discussed here. In this peculiar case, the end of the code can become until RemoveAll is improved:
Regarding RemoveAll, adding a Windows specific file is considered here but issue was closed after updating Unix-like builds. |
We can create a Windows-specific os.RemoveAll. Maybe it can Chdir before removing directory entries, instead of using path+\+file . The syscall package can be updated to fix bugs in the stdlib. Can you elaborate on your code segment above, where would that live? |
It does not seem that the
|
What happens for EDIT: It could also hang in the syscall within os.Lstat(). And within os.Chdir() if we tried that instead of "path+\+file". So we need a solution re long relative paths for the whole os package. We should at least return an error if a long path |
More here: microsoft/winfile#50 (see comments by @malxau) EDIT: and https://stackoverflow.com/questions/38036943/getfullpathnamew-and-long-windows-file-paths |
Indeed, full path names are required. The code below shows which error messages current RemoveAll receives. output is amended for readability
|
AFAIK, using full path is not enough to fix hanging. Lstat is required to know if the path is a directory. A file handle is also required to list directories using Readdirnames(). All add concurrency on the file structure. From Microsoft documentation of RemoveDirectory, recursion requires to use shfileoperation which should probably be added to |
Have you tested os.Remove with a long absolute path, both empty and non-empty? I thought that was verified to work. We don't need os.Remove to do recursion, we expect it to return an error if it's a non-empty directory. I suggested two possible solutions here: #21782 (comment) |
RemoveDirectory is using windows function RemoveDirectoryW. It says:
Prepending "\\?\" shouldn't be enough to fix this? |
Yes, I suggested that in the issue referenced above, which we should fix at the same time. Feel free to submit a patch. (Note that you have to sign a CLA first.) @gopherbot add "help wanted" |
Note that prepending
In the past when I've done this it becomes important to define some form of consistency. It'd be bad if "foo." evaluates to different files depending on the API being invoked. |
Good point. Maybe the fix is to prepend I think the same logic applies to filepath.Walk(). cc @bcmills |
Let me set the environment and read some more about the process for submitting code, and probably will make this my first collaboration to go code |
Appending \\?\ is not a solution. Golang already does that for absolute paths, but explicity avoid doing this into relative path. I've readed a little more about this, and it looks like appending that only works for absolute paths. Relative paths always have to be len MAX_LENGHT
The only solution would be to convert relative paths into absolute paths, if possible. I will investigate about this. |
As mentioned in the comment re filepath.Walk() that I linked above, we can prepend That needs to be done for directory entries, as described in my previous post. |
Yes, sure. I've introduced a syscall.FullPath into fixLongPath, and now it works for most cases presented, but it is still failing in Walk, because the path that's failing is 227 characters long (<248), so it doesn't get processed by this. (This was commented in one of the first issues in filepath.Walk ticket, mentioned before). I have to investigate why this is failing, and fix this. Cheers ;) |
@quentinmit describes here why he didn't do that already: #21782 (comment) I don't think we can touch fixLongPath(), because everything calls it, and we could easily break some case for which there isn't a test. We only need to fix filepath.Walk() and os.RemoveAll(), and we already know how to do so. |
mmm.. But Is backwards logic I think. fixLongPath is called in places were a fix for MAX_PATH is needed, but it's not working for relative paths. But I think that this list is way more extensive. If we are not going to fix fixLongPath, we should be clear about this not working for relative paths, and functions like lstat, open or so, should be clear about not working for relative paths longer than 248 chars on windows. |
See #20829 re long (i.e. invalid) relative path arguments. Users can adjust arguments they pass. But a valid relative path can become invalid by concatenation during a tree walk. So we just need to fix what os.RemoveAll() passes to RemoveAll() (and likewise for filepath.Walk()) by defining a parent from GetFullPathNameW() with prepended |
But ticket #20829 was fixed adding fixLongPath(name): Also, if we are not going to support for users to give us a relative path longer than MAX_PATH, then we should be returning a special error for that, instead of "not existent file/folder", that's not necesarily true |
Read the whole discussion therein. Changing fixLongPath() to return an error for long relative paths is worth considering, but out of scope for this issue. |
We are already weeks into the code freeze, moving to Go1.16. |
Change https://golang.org/cl/263538 mentions this issue: |
There is a CL for this change, but it didn’t make it in early. Punting to Go1.17. |
Change https://golang.org/cl/291291 mentions this issue: |
On Go 1.13.5 on Windows 2008 R2, running os.RemoveAll on a directory that contains a path longer than 260 characters hangs.
In the following repro, the directory is created, and then the program hangs on os.RemoveAll.
The text was updated successfully, but these errors were encountered: