When using software written in the Go language, directories reportedly fail to open #406
Comments
|
To fix the FUSE mirror, it might be enough to just do an lstat and S_ISDIR before calling open: https://github.com/dokan-dev/dokany/blob/master/samples/fuse_mirror/fusexmp.c#L246 Since open() will not complain if a directory is passed in (unless write permissions are requested): https://linux.die.net/man/3/open But I'm not sure if the problem goes deeper in Dokany or not. |
|
The more I think about this, the less certain I am that the fix should be on our end (encfs/dokany). Under Linux, when someone calls open() on a directory, an error is not given unless write permissions are requested (EISDIR error). Similarly, on Windows, CreateFile will not return an error as long as OPEN_EXISTING is used (directory already exists) or the FILE_FLAG_BACKUP_SEMANTICS flag is specified. It seems inapproproate for Go to be expecting an error when requesting to open an existing directory in read-only mode (since both Windows and Linux clearly do not see this as an error situation). @Liryna -- what do you think? |
|
btw We use Go successfully to access a filesystem using dokan1.sys. @jetwhiz, to make Go work you need to check flags carefully in CreateFile and return an error if it is a directory and the request does not have the necessary flags. |
|
Hi @taruti -- this is mainly regarding the fuse_mirror sample, as it does not perform the check you describe (and so software using Go will fail awkwardly on fuse_mirror). see https://github.com/dokan-dev/dokany/blob/master/samples/fuse_mirror/fusexmp.c#L246 I guess my main concern is that opening a directory as a file is not considered an error (in Linux and Windows) unless you request write permissions. Adding a check to fix the issue with Go's implementation seems to go against expected behavior for a filesystem, however. IMO, Go should be the one to make the appropriate request (there is an important distinction between opening a directory as a file or as a directory, both of which are valid and do not return errors). If Go wants to open a directory as a directory, then it should explicitly make that request (rather than relying on non-standard behavior). |
|
@jetwhiz iirc the filesystem should be checking for FILE_DIRECTORY_FILE and FILE_NON_DIRECTORY_FILE from ZwCreateFile but the fuse mapping layer makes that kind of hard... Perhaps the dokan fuse mapping layer should handle these transparently? Since here Windows and Linux semantics are different. |
|
Ah I understand, so Go is sending the FILE_NON_DIRECTORY_FILE flag on Windows when it attempts to open a file (to force a fail for directories)? It might be better if the Dokany FUSE API handles this then since that flag is not currently passed through the API from what I can see? Alternatively, the newest POSIX version has the O_DIRECTORY creation flag (it seems O_DIRECTORY is the opposite of FILE_NON_DIRECTORY_FILE, though, and only tangentially-related to FILE_DIRECTORY_FILE). |
|
Hi @jetwhiz , Thank you for the report ! dokan fuse call mkdir/opendir if CreateOptions has the flag FILE_DIRECTORY_FILE. dokany/dokan_fuse/src/dokanfuse.cpp Line 187 in 63655e3 Normally when FILE_NON_DIRECTORY_FILE is set, it means that FILE_DIRECTORY_FILE is not set. So here will be called open . After here, we don't check if FILE_FLAG_BACKUP_SEMANTICS is set (it is a flag the give the possibility to open a folder as a flag. It would be interesting to know if Go is using this to open folder. |
|
Hi @Liryna Log file: http://pastebin.com/5e5DgKhG Hopefully this is a bit shorter then before everything until Close 0016 or Close 0017 was before the go application tried to read the files. |
|
@Liryna @taruti -- I'm not very familiar with Go, so I can't say one way or another what is happening behind the scenes there. From what I found here: https://github.com/golang/go/blob/master/src/syscall/syscall_windows.go#L248 func Open(path string, mode int, perm uint32) (fd Handle, err error) {
if len(path) == 0 {
return InvalidHandle, ERROR_FILE_NOT_FOUND
}
pathp, err := UTF16PtrFromString(path)
if err != nil {
return InvalidHandle, err
}
var access uint32
switch mode & (O_RDONLY | O_WRONLY | O_RDWR) {
case O_RDONLY:
access = GENERIC_READ
case O_WRONLY:
access = GENERIC_WRITE
case O_RDWR:
access = GENERIC_READ | GENERIC_WRITE
}
if mode&O_CREAT != 0 {
access |= GENERIC_WRITE
}
if mode&O_APPEND != 0 {
access &^= GENERIC_WRITE
access |= FILE_APPEND_DATA
}
sharemode := uint32(FILE_SHARE_READ | FILE_SHARE_WRITE)
var sa *SecurityAttributes
if mode&O_CLOEXEC == 0 {
sa = makeInheritSa()
}
var createmode uint32
switch {
case mode&(O_CREAT|O_EXCL) == (O_CREAT | O_EXCL):
createmode = CREATE_NEW
case mode&(O_CREAT|O_TRUNC) == (O_CREAT | O_TRUNC):
createmode = CREATE_ALWAYS
case mode&O_CREAT == O_CREAT:
createmode = OPEN_ALWAYS
case mode&O_TRUNC == O_TRUNC:
createmode = TRUNCATE_EXISTING
default:
createmode = OPEN_EXISTING
}
h, e := CreateFile(pathp, access, sharemode, sa, createmode, FILE_ATTRIBUTE_NORMAL, 0)
return h, e
}It looks like they're using "OPEN_EXISTING" by default, but I don't see the FILE_FLAG_BACKUP_SEMANTICS flag being set. |
|
In Go syscall.Open is called from https://github.com/golang/go/blob/master/src/os/file_windows.go#L150 for normal file opens. |
|
@Liryna yes I get the error described above I have never programmed anything in Go, but I might be able to try and write something up tomorrow, if no one else is willing to install go and try and write a basic hello world load dir application. I also found this test program (https://gist.github.com/klauspost/5f87caf402a8abf369d5#file-test-go-L42) from rclone/rclone#261 (comment) that might be useful as it is or modified. Update: The output of the above test program "Det går inte att hitta sökvägen." is Swedish for roughly "The Path can not be found" Here is the dokan log: If I ran the above test program on a folder c:\test that contains a file test.txt and a folder named test folder I get the following output: |
|
@qnorsten -- can you replace this line: f, err := os.Open(dir)with: f, err := os.OpenFile(dir, os.O_RDWR | os.O_CREATE, 0666)And see what happens? |
|
@jetwhiz still get the same error message and the Dokan log looks the same to me Update: Seems like the part that gives an error is not f, err := os.Open(dir)but instead info, err := ioutil.ReadDir(dir)https://github.com/golang/go/blob/master/src/io/ioutil/ioutil.go#L93 |
|
I don't think that shouldn't be happening -- fuse_mirror should be failing with an EISDIR errno (https://github.com/dokan-dev/dokany/blob/master/samples/fuse_mirror/fusexmp.c#L246) in that situation if I understand this correctly. https://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html Unless building with Cygwin leads to a different (non-POSIX) open() syscall behavior. |
|
@jetwhiz Just an update on this, do you think there is an issue on dokan for now ? or it is open() that seems to not return the correct error ? |
|
Is there some .exe I could try to reproduce this without setting up a dev-env for a language I have never used? |
|
@Rondom perhaps you can use the test program mentioned earlier from rclone/rclone#261 (comment) https://github.com/ncw/rclone/files/79771/test.zip it comes as an exe |
|
I have created small Go program https://play.golang.org/p/DrtghDt-48 to demonstrate what Go does to open directory and file. You would have to create c:\dir directory and c:\dir\file.txt file before running the program. Running it on my computer, the program prints: So calling Windows CreateFile for my directory returns ERROR_ACCESS_DENIED. I think this is expected. According to https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx "You must set this flag to obtain a handle to a directory." (FILE_FLAG_BACKUP_SEMANTICS). And Go does not pass FILE_FLAG_BACKUP_SEMANTICS to Windows CreateFile. I hope my Go program is simple enough, so you can translate it into C or whatever. Alternatively, you can download Go and build it yourselves. It is quite easy. Let me know if I can help in any way. Alex |
|
Hi @qnorsten @alexbrainman , Sorry for the delay, I had the time to look on this. Maybe I come too late 😢 or maybe go has changed something ? (I use go 1.7.5) |
|
That is strange @Liryna. Did you try it with the dokany fuse mirror sample? Cause it did not work for me last I tried. |
|
ohhh sorry >< so it is with FUSE that the issue only happen |
|
Yes it only a problem with fuse both the fuse sample and encfs4win at least |
|
Fixed with 4860b95 Thank all of you for the report and sorry again. This was quick to find but had no time to do it 😢 |
|
Just tried the lastest snapshot from appveyor CI. Now everything works perfectly. Thanks for the fix Liryna :) |
|
Thanks for fixing this @Liryna ! And sorry for the long hiatus. I moved at the beginning of this year to another country and have been swamped. I'll get this fix pulled into encfs4win as soon as I can, though! |
Environment
Check List
Description
This is based on various bug reports that involve encfs4win:
From what I can tell, the Go programming language will attempt to open everything as a file first and then (only if that fails) does it try opening it as a directory (see https://github.com/golang/go/blob/master/src/os/file_windows.go#L150).
This is apparently causing an issue with Dokany FUSE, since it looks like Dokany does not fail when attempting to open a directory as a file.
For instance, the rclone program (written in Go), fails with the following error when trying to copy files out of the Dokany fuse mirror:
With UNC paths:
Without UNC paths:
Logs
@qnorsten
This example was pulled off of my bug report: http://pastebin.com/QjeLcXY9
The text was updated successfully, but these errors were encountered: