os: IsDir doesn't work with Windows OneDrive #22579
Comments
bradfitz
changed the title
bradfitz
added
help wanted
NeedsInvestigation
|
What does |
|
/cc @alexbrainman |
|
Thanks for the quick answers. It seems that the issue is here: Line 41 in 936b977 I'm not sure why OneDrive folders became symlinks so maybe that's not a bug. But this whole behavior breaks assumptions in other programs so that's kind of an issue. |
|
Or this condition shouldn't be a But in that case that contradict this old fix (1989921#diff-befffd222e4e39277f81cd4cb7116a0b). The fact that OneDrive transform stuff in symlinks will probably cause lots of weird issues. |
|
IMHO the ideal situation is that the standard library treats symlinks as transparently as possible, because otherwise many apps doing IO will have to write special code in order to work correctly with symlinks (but since this is subtle, it would probably mean that in practice apps written in Go wouldn't work correctly with symlinks). While the change introducing this issue was in response to some valid bugs, it would seem preferable that the specific cases where symlinks should not be handled transparently should explicitly check for it (e.g. when doing recursive traversal). |
|
With the same configuration, go version go1.9.2 windows/amd64, os.Stat() IMO, this is appropriate behavior because certain considerations need to be addressed and it provides the opportunity to specify, among other things a FILE_FLAG_OPEN_REPARSE_POINT flag. Symbolic links, as reparse points, have certain programming considerations specific to them.
The OpenFileById function will either open the file or the reparse point, depending on the use of the FILE_FLAG_OPEN_REPARSE_POINT flag. Backup applications that use file streams should specify BACKUP_REPARSE_DATA in the WIN32_STREAM_ID structure when backing up files with reparse points. Also, consider the following information regarding FILE_FLAG_OPEN_REPARSE_POINT: |
|
Users of the Arduino IDE are also reporting problems with OneDrive files not being read by the
I believe the OneDrive files/folders are not symlinks, but they are "reparse points" (and the most common types of reparse points seem to be symlinks and junctions, both kinds of links that ReadLink now supports explicitely). This post suggests they are "Cloud reparse points", though I'm not sure what that means exactly. Perhaps this provides some starting points for fixing this (and perhaps also ReadLink)? |
|
Interesting.. If they are not true symlinks, then it would seem that would strengthen the case that the IO library should treat them as files transparently. Especially if there are guarantees that "cloud reparse points" will always behave just as files do (in the sense that they're not going to cause loops while listing directory contents, etc.). |
Looking at your bug arduino/arduino-builder#254
I would agree that it, probably, is different problem. This current issue is about os IsDir. While yours is os.Readlink. They might be related, but they different problems. I suggest you file a new bug with small reproducible example. Mind you, os.Readlink is broken on Windows - there is no way to implement it properly on Windows. I suggest you use filepath.EvalSymlinks instead. Would that work for you? Alex PS: I still do not have Windows with "Fall Creator Update" to debug this. |
Yeah, I was hoping to do that, but no time at all right now. I was mostly thinking that the info would be useful for this bug as well.
IIRC we're using
Me neither, I'm just basing on reports by our users. |
I don't see that filepath.Walk uses os.Readlink. But I have been wrong before. That is why I suggest you create a new issue with small reproducible example, if you have one. Alex |
I've also looked, and couldn't find this either (perhaps it was a wrong conclusion before). I couldn't actually find any path to |
|
@ttrunck I finally managed to update my Windows 10 to correct version, and I can reproduce something similar to your bug. I suspect the problem is that we don't follow advise from https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/ to the letter. In particular, it says: "... If you use the FindFirstFile function, you can tell that you have a symbolic link because the file attributes will have the FILE_ATTRIBUTES_REPARSE_POINT flag set, and the dwReserved0 member will contain the special value IO_REPARSE_TAG_SYMLINK. ...". We do check the FILE_ATTRIBUTES_REPARSE_POINT flag set, but we do not check dwReserved0. We should rectify this problem, but only once go1.10 is released, because I suspect the change might be involved. We need to fix everything: os.Stat and os.File.Stat and os.Lstat and os.Readdir. Alex |
|
Thanks for looking at it! So this should be fixed in go1.11 and the timeline is mid 2018 right? For now my workaround is to not use that OneDrive feature or move my projects outside of OneDrive. I also think Docker is hitting this issue but I don't know how I can easily confirm that it's indeed the same issue. |
|
I punted this to Go 1.11 because old bugs (ones that aren't recent regressions) always get punted. But if this is suddenly more prominent because of OneDrive, we could consider a fix for Go 1.10, if somebody could prepare one soon that checks dwReserved0. |
|
Change https://golang.org/cl/86556 mentions this issue: |
Yes, the fix is bit convoluted, so it might have to wait for go1.11. @ttrunck can you, please, try the fix here https://golang.org/cl/86556 ? Does it fixes your problem? Thank you. Alex |
|
Hi Alex, Thanks for your work. I will find some time this week to check if this fixes my problem. To be sure I have to build both Go and Hugo (the original program where I noticed the issue) right? The library isn't dynamically linked? As I mentioned I'm just a user of a program written in Go and have zero experience with this language, so my check shouldn't worth more than "I found a case not working and with your patch it's now fixed" Thanks |
Go code is statically linked into the binary if you use the default Go compiler, which includes the standard library such as in this case. Pulling in the potential fix to the standard library might be a bit more confusing if you're not used to building Go from source. I would suggest looking at https://golang.org/doc/install/source. You could do something like:
|
|
Perfect I should be good to go. Thanks for your help. |
|
I am expanding on some @mvdan instructions ...
From https://golang.org/doc/install/source#fetch
Go to https://golang.org/cl/86556 click on Download in the right top corner and copy Checkout instructions into clipboard and run them
if you are on Windows. Alex |
|
Hi Alex, It took me a bit longer that expected to build/check your fix. CGO_ENABLED need to be kept to 1 but since minGW only give a 32 bits gcc, the easiest workaround is to set GOHOSTARCH to 386. From what I understand it shouldn't be an issue. So go version gave me As you probably know my exemple from the first post is now working as expected. But Hugo is still not working. If I use the integrated server, (hugo.exe server) some files aren't served. I also built Hugo from a not patched Go and if I build I still got my previous error: Finally if I use the Hugo.exe build from the patched Go in a directory outside of Onedrive it's working as expected. I haven't dig into the issue yet and I'm not sure that the stack trace will be helpful. I will try to debug that issue next week-end and provide a good minimal exemple. Thanks |
You should not need minGW to test my changes. If you struggle with minGW, just set CGO_ENABLED=0.
386 should work too, but with CGO_ENABLED=0 you could build things without fiddling with GOHOSTARCH.
545ea9ddb7 commit looks good - I have the same.
Well I need instructions on how to reproduce your problem, because my CL 86556 fixes problem you described here #22579 (comment) Maybe your Hugo problem is different problem. It could be even bug in Hugo.
That is good to know - my CL 86556 did not introduce any new problems for your non-OneDrive environment.
Yes, I would need an instructions on how reproduce that (at the very least). And I do not have much free time this days, so the simpler your reproduction the better for me.
That would be good. Thank you. Alex |
|
Change https://golang.org/cl/108755 mentions this issue: |
|
Change https://golang.org/cl/108776 mentions this issue: |
@alexbrainman So, basically, something like this: diff --git a/src/os/types_windows.go b/src/os/types_windows.go
index f3297c0338..6668d911fc 100644
--- a/src/os/types_windows.go
+++ b/src/os/types_windows.go
@@ -140,14 +140,14 @@ func (fs *fileStat) updatePathAndName(name string) error {
}
func (fs *fileStat) isSymlink() bool {
+ const reparsePointNameSurrogate = 0x20000000
// Use instructions described at
// https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/
// to recognize whether it's a symlink.
if fs.FileAttributes&syscall.FILE_ATTRIBUTE_REPARSE_POINT == 0 {
return false
}
- return fs.Reserved0 == syscall.IO_REPARSE_TAG_SYMLINK ||
- fs.Reserved0 == windows.IO_REPARSE_TAG_MOUNT_POINT
+ return fs.Reserved0&reparsePointNameSurrogate != 0
}
func (fs *fileStat) Size() int64 {would be good to have. Let me know if you want me to open a CI. |
Yes. Perhaps more comments are needed to explain the 0x20000000.
I don't see anything wrong with existing code - the code has comment explaining why it does what it does. What do you think the code needs updating? Do you have an issue that is affected by this code? Alex |
@alexbrainman per your own earlier comment: #22579 (comment) (also see comments above that one).
We had an issue with Docker on Windows with deduplication feature turned on (moby/moby#37026) which was solved (in moby/moby#37769) in a way very similar to your commit e83601b, but with the final check for is Perhaps both ways to check are equally correct for now, but it seems that the check for |
Sounds reasonable. But, please, create new issue first at https://golang.org/issues/new with simple repro so we have some history about why we made the change. Please, also describe the environment needed to be able to reproduce this issue. Thank you. Alex |
|
@kolyshkin also I recently fixed os.SameFile - it was completely broken when os.SameFile parameters were symlinks - 1c95d97 . Maybe it will fix your issue. Alex |
@kolyshkin, I decided to get rid of that code altogether. Here is my proposal https://go-review.googlesource.com/c/go/+/143578 Alex |
|
@alexbrainman it looks like you oversimplified things too much (maybe because you assumed that
is not sufficient, you'll get false positives ( The correct check should look something like func (fs *fileStat) isSymlink() bool {
const reparsePointNameSurrogate = 0x20000000 // TODO document this
return fs.FileAttributes&syscall.FILE_ATTRIBUTE_REPARSE_POINT != 0 &&
fs.Reserved0&reparsePointNameSurrogate != 0
}Of course, PS all this is theorizing based on the code I've seen; I don't even have a Windows system to try things out. |
Both moby/moby#37769 and moby/moby#37026 describe repro for a bug in Docker. I do not see how that bug is relevant to my change in https://go-review.googlesource.com/c/go/+/143578 . I have never looked at Docker code, and I do not have time to debug Docker. If you can distill that bug into small Go program, I will try and investigate it. As to your "theorizing", maybe you are correct, maybe you are wrong. I am not aware of any issues that https://go-review.googlesource.com/c/go/+/143578 does not cover. So I am going to proceed with my change, until I see something that breaks with my change. Alex |
@alexbrainman ... which is caused merely by incorrect identification of symlinks on Windows (which is fixed in Go 1.11 by your commit e83601b). I'm afraid it's going to be broken again. You don't have to dig into Docker source code, or to use Docker in order to reproduce. If you can take another look at moby/moby#37769 description, in particular "How to verify" section, you'll find the steps to enable deduplication. Once enabled, the test to perform (without using Docker) is to iterate through the files, doing something like this for each one: fi, err := os.Lstat(path)
if err != nil {
return err
}
if fi.Mode()&os.ModeSymlink != 0 {
link, err := os.Readlink(path)
if err != nil {
return err
}
}As you can see, the code assumes if a file is a symlink, Previously (before commit e83601b) the above code failed on deduplicated files with the error like this: From what I see, this bug might come back with the change in https://go-review.googlesource.com/c/go/+/143578. Ideally, of course, there should be a unit test added to check for this specific issue (treating dedup'ed files like symlinks). I apologize for not making things clear earlier, and hope this explanation makes sense. This is as close as I can get to presenting a simple reproducer without actual access to a Windows machine. |
Maybe yes, maybe not. We won't know that until we apply CL 143578 to your test case. If you give me your test case, I will try it, if I can. If you cannot provide me with a test case, I cannot even try. Feel free to try CL 143578 yourself, and report results back here. I will happy to help with instructions, if you need any.
This is to vague for a test. Please use https://github.com/golang/go/issues/new and answer all questions there.
os.Readlink is broken on Windows in many situations. You should not be using it in your code.
I agree, it might be difficult to have permanent test. But I would like to try and reproduce it myself at least somewhere.
Unfortunately this is not enough for me for a repro. Maybe you should try and explain what your are trying to do? Perhaps I might suggest the code that will work for you. Alex |
The code in question adds a file to a tar; the relevant part is creating a tar entry header. This is the code ( // addTarFile adds to the tar archive a file from `path` as `name`
func (ta *tarAppender) addTarFile(path, name string) error {
fi, err := os.Lstat(path)
if err != nil {
return err
}
var link string
if fi.Mode()&os.ModeSymlink != 0 {
var err error
link, err = os.Readlink(path)
if err != nil {
return err
}
}
hdr, err := FileInfoHeader(name, fi, link)
if err != nil {
return err
}On a system with deduplication turned on and before commit e83601b (and I suspect after CL 143578), this leads to As you can see, the code assumes that if You say that
Can you please elaborate (and/or propose a replacement for the above code)? |
|
@kolyshkin I played with files on deduped volume. I am happy to report that, you are correct - all files on deduped volume looks like files now, because Windows reports then with both FILE_ATTRIBUTE_REPARSE_POINT and IO_REPARSE_TAG_DEDUP set. So after my CL 143578 - which only checks for FILE_ATTRIBUTE_REPARSE_POINT - they will be interpreted as symlinks. I suppose you are saying they should be plain files, not symlinks, and I cannot go ahead with CL 143578 unless I find way to check for IO_REPARSE_TAG*. Fair enough. Alex |
|
@alexbrainman glad to be of help. Checking IO_REPARSE_TAG_* is not a good idea though, as it's not future-proof (say MS adds some other fancy feature and a flag, and then your code needs to be updated yet again to account for this new flag). It still seems to me the correct check is the one described in #22579 (comment), ie the check for reparsePointNameSurrogate (0x20000000) in Reserved0 field, populated from syscall.FindFirstFile by newFileStatFromWin32finddata. This is very similar to what the code in Go 1.11 does. I could create a CL but need to go through the horrors of setting up Windows development environment first |
One thing at a time. I am trying to fix #27225 and #27515 at this moment. Alex |
|
I adjusted CL 143578 for my yesterday's findings. I hope after CL 143578 deduped volumes will work as good / bad as they do now. Alex |
|
@alexbrainman thank you Alex, it looks like CL 143578 (patch set 3) is good, I have left my review comments right in there (should have done it in the first place, sorry for abusing github). |
I think there is an issue with FileInfo.IsDir in Windows 10 using the new Files On-Demand feature of OneDrive. (This occurs only after the Fall Creator Update that introduce this feature).
If I create a folder in OneDrive (with the Files-On-Demand feature on) IsDir return false. Note that just after the creation it return true but once OneDrive finish its sync it then return false. Also if I deactivate the Files-On-Demand feature I no longer repro (for newly created folder, the old one still have the issue). Using the "Always keep on the device option" or the "Free up space" doesn't change the repro.
Also I may miss something obvious, I never used Go before but I found this issue while debugging and issue with Hugo (a site generator in Go).
What version of Go are you using (
go version)?go version go1.9.2 windows/amd64Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?My OneDrive version is:
Version 2017 (Build 17.3.7105.1024)What did you do?
What did you expect to see?
What did you see instead?
The text was updated successfully, but these errors were encountered: