Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: ServeFile fails to serve files over 4GB on Windows #33193

Open
NickTikhonov opened this issue Jul 19, 2019 · 7 comments

Comments

@NickTikhonov
Copy link

commented Jul 19, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.7 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\nicktikhonov\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=...
set GOPROXY=
set GORACE=
set GOROOT=c:\go
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\NICKTI~1\AppData\Local\Temp\go-build372822890=/tmp/go-build -gno-record-gcc-switches

What did you do?

Ran the following script. Before that, you can run

fsutil file createnew bigfile 8589934610

to generate a large file.

package main

import (
    "context"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "time"
)

// Create a random file in the same directory
// fsutil file createnew bigfile 8589934610

type indexHandler struct {
}

func (h *indexHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "bigfile")
}

func main() {
    fmt.Println("Starting server....")
    httpServer := &http.Server{Addr: fmt.Sprintf("%v:%v", "localhost", "8080"), Handler: &indexHandler{}}
    go func() {
        err := httpServer.ListenAndServe()
        if err != nil {
            fmt.Println("Err from server")
            fmt.Println(err)
        }
    }()

    time.Sleep(5 * time.Second)

    fmt.Println("Starting client....")
    client := &http.Client{}
    req, err := http.NewRequest("GET", "http://localhost:8080/", nil)
    req.Header.Add("Connection", "Keep-Alive")
    req.Header.Add("Keep-Alive", "timeout=1000")
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }

    fmt.Println(req)

    totalLen := uint64(res.ContentLength)
    bodyReader := res.Body
    fmt.Printf("totalLen: %v\n", totalLen)
    written, err := io.Copy(ioutil.Discard, bodyReader)
    fmt.Printf("downloaded: %v\n", written)
    fmt.Printf("err: %v\n", err)
    httpServer.Shutdown(context.Background())
}

What did you expect to see?

I expected the download to have finished,

What did you see instead?

Instead, only 18 bytes are downloaded, and the client receives an unexpected EOF

totalLen: 8589934610
downloaded: 18
err: unexpected EOF

@dmitshur dmitshur added this to the Go1.14 milestone Jul 19, 2019

@dmitshur dmitshur changed the title http.ServeFile fails to serve files over 4GB on Windows net/http: ServeFile fails to serve files over 4GB on Windows Jul 19, 2019

@dmitshur dmitshur added the OS-Windows label Jul 19, 2019

@agnivade

This comment has been minimized.

Copy link
Member

commented Jul 20, 2019

What do you get when you try to fetch the file using curl ?

What is the cutoff size beyond which you are seeing failures ? Are 1GB files working fine ?

Are you using any virtualization layer like WSL (#27128) or VM (#30082) by any chance ?

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Jul 21, 2019

@NickTikhonov I could reproduce your problem on current tip. Thank you very much.

I did not have enough time to understand where the problem is.

Alex

@NickTikhonov

This comment has been minimized.

Copy link
Author

commented Jul 22, 2019

@alexbrainman I believe the problem could be some sort of uint overflow. The number of bytes downloaded seems close to the total size modulo 2 ^ 32 - 1

@gopherbot

This comment has been minimized.

Copy link

commented Jul 22, 2019

Change https://golang.org/cl/187037 mentions this issue: net: do not call Windows TransmitFile for large files

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Jul 22, 2019

@NickTikhonov I think I found the problem. Can you, please, try https://go-review.googlesource.com/c/go/+/187037 and see if it fixes the problem for you. Thank you.

Alex

@mattn

This comment has been minimized.

Copy link
Member

commented Jul 24, 2019

@alexbrainman Thank your fixing this. (I implemented sendfile on Windows)

@alexbrainman

This comment has been minimized.

Copy link
Member

commented Jul 28, 2019

@NickTikhonov

You did not say, if https://go-review.googlesource.com/c/go/+/187037 fixes your problem. Please confirm.

Thank you.

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.