Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: read/write messages >2GB on sockets #16266

Closed
sv opened this issue Jul 5, 2016 · 9 comments

Comments

Projects
None yet
6 participants
@sv
Copy link

commented Jul 5, 2016

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go version go1.6 darwin/amd64
  2. What operating system and processor architecture are you using (go env)?
    GOARCH="amd64"
    GOBIN=""
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="darwin"
    GOOS="darwin"
    GOPATH="/Users/sergeyvidyuk/go"
    GORACE=""
    GOROOT="/usr/local/go"
    GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
    GO15VENDOREXPERIMENT="1"
    CC="clang"
    GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
    CXX="clang++"
    CGO_ENABLED="1"
  3. What did you do?
    https://play.golang.org/p/wBCOUuRZeT
    Doesn't work in playground at all. Please run locally.
  4. What did you expect to see?
    500M elements in int64 slice
  5. What did you see instead?
    write tcp 127.0.0.1:50186->127.0.0.1:50187: write: invalid argument
    read tcp 127.0.0.1:50187->127.0.0.1:50186: read: invalid argument

@sv sv changed the title Read/Write messages >2GB net,encoding/binary: read/write messages >2GB Jul 5, 2016

@mikioh mikioh changed the title net,encoding/binary: read/write messages >2GB kernel: read/write messages >2GB Jul 5, 2016

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2016

I don't understand what you are trying to do, but I guess, if you want to read and write a bit large chunk per socket IO system call, you need to modify the kernel implementation. FWIW, there are sort of soft and hard limits; see kern.ipc.sorecvmincopy and friends.

@mikioh mikioh changed the title kernel: read/write messages >2GB kernel: read/write messages >2GB on sockets Jul 5, 2016

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2016

See https://github.com/opensource-apple/xnu/blob/10.11/bsd/kern/uipc_socket.c#L2957. Closing, there is nothing we can do about it.

@mikioh mikioh closed this Jul 5, 2016

@sv

This comment has been minimized.

Copy link
Author

commented Jul 5, 2016

I want to encode large data chunks using binary encoding and send/receive them via network. This seems to be not possible at the moment. Would it be possible for Go(net or encoding/binary) to break message down to 2GB chunks during reads/writes. Maybe ReadFull can be relaxed and made incremental ?

@robpike

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2016

You could do that yourself. The data sizes you are dealing with are unreasonably large for a single operation, and if you expect to handle items like that you should design your software not to lock down so much memory in a single call. Use a streaming design instead.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2016

In the os package we cap reads and writes at 1GB on Darwin and FreeBSD. See issue #7812 and https://golang.org/cl/89900044. This issue is essentially the same as #7812, only for the net package rather than the os package, and we could fix it the same way. Reopening.

@ianlancetaylor ianlancetaylor reopened this Jul 5, 2016

@ianlancetaylor ianlancetaylor added this to the Go1.8Maybe milestone Jul 5, 2016

@ianlancetaylor ianlancetaylor changed the title kernel: read/write messages >2GB on sockets net: read/write messages >2GB on sockets Jul 5, 2016

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2016

@ianlancetaylor,

Just for clarification, I'm still unclear what you mean by "the same way", because unlike IO operation to filesystem, there are various characteristics of the underlying protocols for socket IO operation. Are you suggesting to introduce sort of chunk split layer not only into read/write on byte stream protocols but into datagram protocols? Or just for byte stream protocols?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2016

That's true, it seems fairly clear that we should only do it for TCP, not UDP. Fortunately, I think we have that information in the Read and Write methods in net/fd_unix.go.

@robpike

This comment has been minimized.

Copy link
Contributor

commented Jul 7, 2016

I remain unconvinced of the value of lifting the limit.

@quentinmit quentinmit added the NeedsFix label Oct 10, 2016

@gopherbot

This comment has been minimized.

Copy link

commented Oct 24, 2016

CL https://golang.org/cl/31584 mentions this issue.

@gopherbot gopherbot closed this in 03d641b Oct 29, 2016

@golang golang locked and limited conversation to collaborators Oct 29, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.