os: File.Seek on Windows errors on offsets like 4G-1, 8G-1, 16G-1

[gilbertchen]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8.1

Does this issue reproduce with the latest release?

I didn't retry, but by looking at the relevant code on the master branch it looks like the bug is still there.

What operating system and processor architecture are you using (go env)?

Windows 64 bit

What did you do?

Create a 4GB sparse file by moving the file pointer to before the last byte and then write a \x00

Code at:
https://play.golang.org/p/kzMJvpa_eJ

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

It should create the file and returns with success

What did you see instead?

Instead it returns an invalid argument error

This is because SetFilePointer did not handle the return value correctly. According to Microsoft's doc:

Note Because INVALID_SET_FILE_POINTER is a valid value for the low-order DWORD of the new file pointer, you must check both the return value of the function and the error code returned by GetLastError to determine whether or not an error has occurred. If an error has occurred, the return value of SetFilePointer is INVALID_SET_FILE_POINTER and GetLastError returns a value other than NO_ERROR. For a code example that demonstrates how to check for failure, see the Remarks section in this topic.

SetFilePointer in Go always returns the error if you pass 4G - 1, 8G - 1, 16G - 1, etc to the Seek function, even if the operation does return successfully.

[mvdan]

Seems like the fix plus a test would be rather simple - would you like to give it a try? See https://golang.org/doc/contribute.html

[gilbertchen]

@mvdan I can try, but it may take a few days.

[heinrich5991]

SetFilePointerEx seems to be the more appropriate function here, it's even suggested in the docs of SetFilePointer for 64 bit seeks.

This function stores the file pointer in two LONG values. To work with file pointers that are larger than a single LONG value, it is easier to use the SetFilePointerEx function.

[odeke-em]

Gentle ping, how's it going @gilbertchen and @mvdan?

[bradfitz]

I tried the trivial change, like:

diff --git a/src/syscall/syscall_windows.go b/src/syscall/syscall_windows.go
index 21d5ecf..e6ceb9c 100644
--- a/src/syscall/syscall_windows.go
+++ b/src/syscall/syscall_windows.go
@@ -9,6 +9,7 @@ package syscall
 import (
        errorspkg "errors"
        "internal/race"
+       "runtime"
        "sync"
        "unicode/utf16"
        "unsafe"
@@ -332,6 +333,16 @@ func Write(fd Handle, p []byte) (n int, err error) {
 
 var ioSync int64
 
+var procSetFilePointerEx = modkernel32.NewProc("SetFilePointerEx")
+
+func setFilePointerEx(handle Handle, distToMove int64, newFilePointer *int64, whence uint32) error {
+       _, _, e1 := Syscall6(procSetFilePointerEx.Addr(), 4, uintptr(distToMove), uintptr(unsafe.Pointer(newFilePointer)), uintptr(whence), 0, 0, 0)
+       if e1 != 0 {
+               return errnoErr(e1)
+       }
+       return nil
+}
+
 func Seek(fd Handle, offset int64, whence int) (newoffset int64, err error) {
        var w uint32
        switch whence {
@@ -342,13 +353,17 @@ func Seek(fd Handle, offset int64, whence int) (newoffset int64, err error) {
        case 2:
                w = FILE_END
        }
-       hi := int32(offset >> 32)
-       lo := int32(offset)
        // use GetFileType to check pipe, pipe can't do seek
        ft, _ := GetFileType(fd)
        if ft == FILE_TYPE_PIPE {
                return 0, ESPIPE
        }
+       if runtime.GOARCH == "amd64" {
+               err = setFilePointerEx(fd, offset, &newoffset, w)
+               return
+       }
+       hi := int32(offset >> 32)
+       lo := int32(offset)
        rlo, e := SetFilePointer(fd, lo, &hi, w)
        if e != nil {
                return 0, e

But that doesn't work. Looks like SetFilePointerEx comes with a bunch of warnings about multi-threaded programs and overlapped I/O. Not sure whether those apply.

Also, that GOARCH == "amd64" check is bogus. This should also work on 386 but I don't know how to shove a LARGE_INTEGER through syscall.Syscall6 on Windows.

Maybe @alexbrainman is interested in looking into this?

[ianlancetaylor]

This is an existing issue. We can look at a CL, but otherwise kicking it down the road.

[heinrich5991]

This is the how SetFilePointerEx is called in Rust, maybe you can copy that (except for the actual calling of the syscall): https://github.com/rust-lang/rust/blob/833785b090c30d4a359d901fb41bfafbe1607ce9/src/libstd/sys/windows/fs.rs#L337-L352.

I think the remarks about overlapped IO and multithreading are irrelevant for the implementation of this function, it's clear that two threads trying to set the file pointer and read a bit from the file must synchronize somehow, that's no difference between SetFilePointer and SetFilePointerEx.

[bradfitz]

I think the remarks about overlapped IO and multithreading are irrelevant for the implementation

Yeah, true.

I just don't know the calling convention for GOARCH=386 passing a LARGE_INTEGER to Windows via Go's Syscall6.

But then again, my change also didn't work on amd64, so not sure what I screwed up.

[alexbrainman]

Maybe @alexbrainman is interested in looking into this?

I would like to fix this, but I have been busy lately. I will look into this when I have time, unless someone beats to me.

Alex

[as]

_, _, e1 := Syscall6(procSetFilePointerEx.Addr(), 4, uintptr(distToMove), uintptr(unsafe.Pointer(newFilePointer)), uintptr(whence), 0, 0, 0)

@bradfitz
The handle is passed in to setFilePointerEx, but it isn't given as an argument to the actuall syscall.Syscall6.

[bradfitz]

@as, hah. Whoops. Thanks!

[gopherbot]

Change https://golang.org/cl/82535 mentions this issue: syscall: use SetFilePointerEx on Windows, allowing large seek offsets

[bradfitz]

Ah, here's what LARGE_INTEGER means: https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx

[bradfitz]

And I can repro that my new test fails on 386 and passes with my new fix on amd64:

bradfitz@gdev:~/go/src/os$ gomote run user-bradfitz-windows-amd64-2016-0 go/bin/go test -short -v -run=Seek os
=== RUN   TestSeek
--- PASS: TestSeek (0.01s)
=== RUN   TestSeekError
--- PASS: TestSeekError (0.00s)
PASS
ok      os      0.045s
bradfitz@gdev:~/go/src/os$ gomote run -e GOARCH=386  user-bradfitz-windows-amd64-2016-0 go/bin/go test -short -v -run=Seek os
=== RUN   TestSeek
--- FAIL: TestSeek (0.01s)
        os_test.go:1382: #8: Seek(4294967295, 0) = 0, seek C:\Users\gopher\AppData\Local\Temp\1\_Go_TestSeek195906635: invalid argument want 4294967295, nil
        os_test.go:1382: #9: Seek(0, 1) = 0, seek C:\Users\gopher\AppData\Local\Temp\1\_Go_TestSeek195906635: invalid argument want 4294967295, nil
        os_test.go:1382: #10: Seek(8589934591, 0) = 0, seek C:\Users\gopher\AppData\Local\Temp\1\_Go_TestSeek195906635: invalid argument want 8589934591, nil
        os_test.go:1382: #11: Seek(0, 1) = 0, seek C:\Users\gopher\AppData\Local\Temp\1\_Go_TestSeek195906635: invalid argument want 8589934591, nil
=== RUN   TestSeekError
--- PASS: TestSeekError (0.00s)
FAIL
FAIL    os      0.137s
Error running run: exit status 1

Fixing on 386 now.