Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: ServeFile fails to serve files over 4GB on Windows #33193

Closed
NickTikhonov opened this issue Jul 19, 2019 · 11 comments
Closed

net/http: ServeFile fails to serve files over 4GB on Windows #33193

NickTikhonov opened this issue Jul 19, 2019 · 11 comments

Comments

@NickTikhonov
Copy link

@NickTikhonov NickTikhonov commented Jul 19, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\nicktikhonov\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=...
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\NICKTI~1\AppData\Local\Temp\go-build372822890=/tmp/go-build -gno-record-gcc-switches

What did you do?

Ran the following script. Before that, you can run

fsutil file createnew bigfile 8589934610

to generate a large file.

package main

import (
    "context"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

// Create a random file in the same directory
// fsutil file createnew bigfile 8589934610

type indexHandler struct {
}

func (h *indexHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "bigfile")
}

func main() {
    fmt.Println("Starting server....")
    httpServer := &http.Server{Addr: fmt.Sprintf("%v:%v", "localhost", "8080"), Handler: &indexHandler{}}
    go func() {
        err := httpServer.ListenAndServe()
        if err != nil {
            fmt.Println("Err from server")
            fmt.Println(err)
        }
    }()

    time.Sleep(5 * time.Second)

    fmt.Println("Starting client....")
    client := &http.Client{}
    req, err := http.NewRequest("GET", "http://localhost:8080/", nil)
    req.Header.Add("Connection", "Keep-Alive")
    req.Header.Add("Keep-Alive", "timeout=1000")
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }

    fmt.Println(req)

    totalLen := uint64(res.ContentLength)
    bodyReader := res.Body
    fmt.Printf("totalLen: %v\n", totalLen)
    written, err := io.Copy(ioutil.Discard, bodyReader)
    fmt.Printf("downloaded: %v\n", written)
    fmt.Printf("err: %v\n", err)
    httpServer.Shutdown(context.Background())
}

What did you expect to see?

I expected the download to have finished,

What did you see instead?

Instead, only 18 bytes are downloaded, and the client receives an unexpected EOF

totalLen: 8589934610
downloaded: 18
err: unexpected EOF
@dmitshur dmitshur added this to the Go1.14 milestone Jul 19, 2019
@dmitshur dmitshur changed the title http.ServeFile fails to serve files over 4GB on Windows net/http: ServeFile fails to serve files over 4GB on Windows Jul 19, 2019
@dmitshur dmitshur added the OS-Windows label Jul 19, 2019
@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Jul 20, 2019

What do you get when you try to fetch the file using curl ?

What is the cutoff size beyond which you are seeing failures ? Are 1GB files working fine ?

Are you using any virtualization layer like WSL (#27128) or VM (#30082) by any chance ?

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Jul 21, 2019

@NickTikhonov I could reproduce your problem on current tip. Thank you very much.

I did not have enough time to understand where the problem is.

Alex

@NickTikhonov

This comment has been minimized.

Copy link
Author

@NickTikhonov NickTikhonov commented Jul 22, 2019

@alexbrainman I believe the problem could be some sort of uint overflow. The number of bytes downloaded seems close to the total size modulo 2 ^ 32 - 1

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jul 22, 2019

Change https://golang.org/cl/187037 mentions this issue: net: do not call Windows TransmitFile for large files

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Jul 22, 2019

@NickTikhonov I think I found the problem. Can you, please, try https://go-review.googlesource.com/c/go/+/187037 and see if it fixes the problem for you. Thank you.

Alex

@mattn

This comment has been minimized.

Copy link
Member

@mattn mattn commented Jul 24, 2019

@alexbrainman Thank your fixing this. (I implemented sendfile on Windows)

@alexbrainman

This comment has been minimized.

Copy link
Member

@alexbrainman alexbrainman commented Jul 28, 2019

@NickTikhonov

You did not say, if https://go-review.googlesource.com/c/go/+/187037 fixes your problem. Please confirm.

Thank you.

Alex

@gopherbot gopherbot closed this in b963149 Aug 27, 2019
@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Aug 27, 2019

I'll reopen this issue as @bradfitz raised a comment on the CL https://go-review.googlesource.com/c/go/+/187037 about perhaps trying to send the file in 2GB chunks instead of failing the first time.

@odeke-em odeke-em reopened this Aug 27, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 28, 2019

Change https://golang.org/cl/192143 mentions this issue: Revert "net: do not call Windows TransmitFile for large files"

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Aug 30, 2019

Change https://golang.org/cl/192518 mentions this issue: net: handle >=2GiB files with sendfile on Windows

tomocy added a commit to tomocy/go that referenced this issue Sep 1, 2019
TransmitFile does not allow for number of bytes that can be
transmitted to be larger than 2147483646. See

https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nf-mswsock-transmitfile

for details. So adjust sendFile accordingly.

No test added, because this would require creating large file
(more than 2GB file).

Fixes golang#33193.

Change-Id: I82e0cb104d112264e4ea363bb20b6d02ac30b38e
Reviewed-on: https://go-review.googlesource.com/c/go/+/187037
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
t4n6a1ka added a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
TransmitFile does not allow for number of bytes that can be
transmitted to be larger than 2147483646. See

https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nf-mswsock-transmitfile

for details. So adjust sendFile accordingly.

No test added, because this would require creating large file
(more than 2GB file).

Fixes golang#33193.

Change-Id: I82e0cb104d112264e4ea363bb20b6d02ac30b38e
Reviewed-on: https://go-review.googlesource.com/c/go/+/187037
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
@gopherbot gopherbot closed this in 3a067f7 Sep 7, 2019
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Sep 8, 2019

Change https://golang.org/cl/194218 mentions this issue: net: permit sendFile on more than *io.LimitedReader on Windows

gopherbot pushed a commit that referenced this issue Sep 26, 2019
CL 192518 was a minimal simplification to get sendfile
on Windows to work with chunked files, but as I had mentioned,
I would add even more improvements.

This CL improves it by:
* If the reader is not an *io.LimitedReader, since the underlying
reader is anyways an *os.File, we fallback and stat that
file to determine the file size and then also invoke the chunked
sendFile on the underlying reader. This issue existed even
before the prior CL.
* Extracting the chunked TransmitFile logic and moving it directly
into internal/poll.SendFile.

Thus if the callers of net.sendFile don't use *io.LimitedReader,
but have a huge file (>2GiB), we can still invoke the chunked
internal/poll.SendFile on it directly.

The test case is not included in this patch as it requires
creating a 3GiB file, but that if anyone wants to view it, they
can find it at
    https://go-review.googlesource.com/c/go/+/194218/13/src/net/sendfile_windows_test.go

Updates #33193.

Change-Id: I97a67c712d558c84ced716d8df98b040cd7ed7f7
Reviewed-on: https://go-review.googlesource.com/c/go/+/194218
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.