bytes: it is impossible to Seek() in slice with len more than 2^32 bytes

[gopherbot]

by marko@kevac.org:

This bug is in all go versions (including tip).

It is impossible to Seek() in bytes.Reader to offset higher than 2^32 because of the
check inside Seek() implementation.

This check is not needed IMHO. Latest Go versions allow slices with length larger than
2^32.

Patch would be very simple. Just remove check from Seek() implementation.

[cznic]

Comment 1:

The specs allow slices with indices/len/cap of only int sizes. Note that 'int' is
architecture specific. Allowing bytes.Reader to accommodate buffers greater than or
equal to 1<<31 (not 2^32, btw) will enable writing programs which would run on 8
byte int archs but would not run on architectures having only 4 byte ints. Also, for
example Reader's Len would have no way how to return a correct value in some cases on
such archs. IOW, this cannot be changed, AFAICT.
#WAI

[gopherbot]

Comment 2 by marko@kevac.org:

Right now bytes does not work for 64bit archs.
Would it be correct if we change the check to use unsafe.Sizeof instead of using
1<<31?

[bradfitz]

Comment 3:

Yeah, this is surprising. strings.Reader, too.
I think we could change this and make it bounded by the size of int on the local
machine, not 1<<31.
No point giving you 64-bit ints and slice/string lengths and be unable to use it all
here.  There are already Go programs that only run on 64-bit machines (programs needing
lots of memory), so I'm not concerned about that.

Labels changed: added release-go1.3maybe.

Status changed to Accepted.

[cznic]

Comment 4:

Seek(1<<33) would then compile only on 8 byte int archs. I know that the same
problem exists for _ = slice[1<<33], but I think such code should discouraged by
the stdlib. Also a 2GB buffer is big enough for anything I can imagine. The purpose of
buffer is not to accommodate _all_ data but to make processing the data faster when
interleaved I/O and processing is the bottleneck. The improvement often happens with
buffers as small as only few kB. Then the law of diminishing results inevitably steps in
and huge buffers can even make things slower wrt to latency of some initial result data
being produced.

[gopherbot]

Comment 5 by marko@kevac.org:

I have stumbled upon this issue when trying to use bytes.Reader on []byte returned from
syscall.Mmap. Are you saying that having files larger than 2GB is impractical? :-)

[bradfitz]

Comment 6:

Sent https://golang.org/cl/81530043

Status changed to Started.

[bradfitz]

Comment 7:

This issue was closed by revision f074565.

Status changed to Fixed.