net/http: ServeFile fails to serve files over 4GB on Windows

[NickTikhonov]

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\nicktikhonov\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=...
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\NICKTI~1\AppData\Local\Temp\go-build372822890=/tmp/go-build -gno-record-gcc-switches

What did you do?

Ran the following script. Before that, you can run

fsutil file createnew bigfile 8589934610

to generate a large file.

package main

import (
    "context"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

// Create a random file in the same directory
// fsutil file createnew bigfile 8589934610

type indexHandler struct {
}

func (h *indexHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "bigfile")
}

func main() {
    fmt.Println("Starting server....")
    httpServer := &http.Server{Addr: fmt.Sprintf("%v:%v", "localhost", "8080"), Handler: &indexHandler{}}
    go func() {
        err := httpServer.ListenAndServe()
        if err != nil {
            fmt.Println("Err from server")
            fmt.Println(err)
        }
    }()

    time.Sleep(5 * time.Second)

    fmt.Println("Starting client....")
    client := &http.Client{}
    req, err := http.NewRequest("GET", "http://localhost:8080/", nil)
    req.Header.Add("Connection", "Keep-Alive")
    req.Header.Add("Keep-Alive", "timeout=1000")
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }

    fmt.Println(req)

    totalLen := uint64(res.ContentLength)
    bodyReader := res.Body
    fmt.Printf("totalLen: %v\n", totalLen)
    written, err := io.Copy(ioutil.Discard, bodyReader)
    fmt.Printf("downloaded: %v\n", written)
    fmt.Printf("err: %v\n", err)
    httpServer.Shutdown(context.Background())
}

What did you expect to see?

I expected the download to have finished,

What did you see instead?

Instead, only 18 bytes are downloaded, and the client receives an unexpected EOF

totalLen: 8589934610
downloaded: 18
err: unexpected EOF

[agnivade]

What do you get when you try to fetch the file using curl ?

What is the cutoff size beyond which you are seeing failures ? Are 1GB files working fine ?

Are you using any virtualization layer like WSL (#27128) or VM (#30082) by any chance ?

[alexbrainman]

@NickTikhonov I could reproduce your problem on current tip. Thank you very much.

I did not have enough time to understand where the problem is.

Alex

[NickTikhonov]

@alexbrainman I believe the problem could be some sort of uint overflow. The number of bytes downloaded seems close to the total size modulo 2 ^ 32 - 1

[gopherbot]

Change https://golang.org/cl/187037 mentions this issue: net: do not call Windows TransmitFile for large files

[alexbrainman]

@NickTikhonov I think I found the problem. Can you, please, try https://go-review.googlesource.com/c/go/+/187037 and see if it fixes the problem for you. Thank you.

Alex

[mattn]

@alexbrainman Thank your fixing this. (I implemented sendfile on Windows)

[alexbrainman]

@NickTikhonov

You did not say, if https://go-review.googlesource.com/c/go/+/187037 fixes your problem. Please confirm.

Thank you.

Alex

[odeke-em]

I'll reopen this issue as @bradfitz raised a comment on the CL https://go-review.googlesource.com/c/go/+/187037 about perhaps trying to send the file in 2GB chunks instead of failing the first time.

[gopherbot]

Change https://golang.org/cl/192143 mentions this issue: Revert "net: do not call Windows TransmitFile for large files"

[gopherbot]

Change https://golang.org/cl/192518 mentions this issue: net: handle >=2GiB files with sendfile on Windows

[gopherbot]

Change https://golang.org/cl/194218 mentions this issue: net: permit sendFile on more than *io.LimitedReader on Windows