io: add an Err field to LimitedReader

[DeedleFake]

New Proposal

As per @ianlancetaylor's suggestion, this proposal is now to add an Err field to io.LimitedReader. If not nil, this field's value will be returned when trying to read past the limit imposed by the reader instead of the default io.EOF.

Original Proposal

For various reasons, I had a need to use MaxBytesReader() in a situation where a ResponseWriter wasn't easily available, so I looked through the code to see if it was safe to pass nil and found that all it does is check a type assertion and then call an unexported method on it:

type requestTooLarger interface {
	requestTooLarge()
}
if res, ok := l.w.(requestTooLarger); ok {
	res.requestTooLarge()
}

While it's true that because of this it's safe to pass it a nil ResponseWriter, it feels quite odd to rely on undocumentated behavior of this kind.

Proposal

Several ways to clean this situation up come to mind:

1. Add an alternative in the http package that doesn't need a ResponseWriter. Because of how it works, it could actually just return the same implementation, but the function will be more obvious in its usage.
2. Document how the implementation uses the ResponseWriter and explicitly state that a nil ResponseWriter is valid.
3. Add a reimplementation of MaxBytesReader() in io that doesn't use a ResponseWriter at all and isn't tied to http behavior, but includes the other differences. This could actually be done, for the most part, by just adding an Err error field to io.LimitedReader that, if non-nil, is the error returned when the limit is reached instead of io.EOF.

In whichever case, the documentation for MaxBytesReader() should probably be filled in a bit. It is quite odd that something called Reader requires a type that is primarily an io.Writer implementation for non-obvious reasons and doesn't mention it anywhere in the documentation.

[jimmyfrasche]

I have forked io.LimitedReader to return a custom error at least three times.

[ianlancetaylor]

If the proposed function doesn't use http.ResponseWriter, then I can't see any reason that it should live in the net/http package. That seems to reduce this to

type MyLimitedReader struct {
    r   io.LimitedReader
    err error
}

func (mlr *MyLimitedReader) Read(p []byte) (int, error) {
    n, err := mlr.Read(p)
    if err == io.EOF {
        err = mlr.err
    }
    return n, er
}

Is that correct?

If so, perhaps this proposal should be rewritten to add an Err field io.LimitedReader.

[carlmjohnson]

I like the new proposal. Returning io.EOF makes io.LimitedReader pretty unhelpful. But I think it would also be good to document that http.MaxBytesReader accepts nil ResponseWriter.

[bcmills]

I have forked io.LimitedReader to return a custom error at least three times.

FWIW, my moreio.LimitedWriter has an Err field too:

[carlmjohnson]

Can someone tag this proposal as under review?

[Xeoncross]

As an alternative to defining your own error, I just created a package and export the error so the caller can check against it.

// Throws an error if the reader is bigger than limit.
var ErrSizeExceeded = errors.New("stream size exceeded")

type MaxBytesReader struct {
	io.ReadCloser       // reader object
	N             int64 // max bytes remaining.
}

func NewMaxBytesReader(r io.ReadCloser, limit int64) *MaxBytesReader {
	return &MaxBytesReader{r, limit}
}

func (b *MaxBytesReader) Read(p []byte) (n int, err error) {
	if b.N <= 0 {
		return 0, ErrSizeExceeded
	}

	if int64(len(p)) > b.N {
		p = p[0:b.N]
	}

	n, err = b.ReadCloser.Read(p)
	b.N -= int64(n)
	return
}

[carlmjohnson]

That would break current users of io.LimitReader/LimitedReader.

I misunderstood the proposal.

[ianlancetaylor]

@carlmjohnson It is already so tagged. We'll get to it soon, I think.

[gopherbot]

Change https://go.dev/cl/396215 mentions this issue: io: add an Err field to LimitedReader

[rsc]

This seems reasonable. Thanks for the proposal.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

Does anyone object to adding this?

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[rsc]

Sounds like we agree to arrange to read one extra byte (whether in the final normal read or in an extra one, depending on room) and use that to decide EOF or not.

[dmitshur]

This is currently labeled to block beta 1 which we're aiming to release next week if possible. Do you think the change above can happen before then, or otherwise is it safe to land it after beta 1 (possibly in time for beta 2 or so)? Thanks.

[gopherbot]

Change https://go.dev/cl/410035 mentions this issue: io: return EOF from LimitedReader.Read if at EOF

[ianlancetaylor]

@rogpeppe Does https://go.dev/cl/410035 act as you expected? Thanks.

[rogpeppe]

@ianlancetaylor Technically yes, but I think it would be preferable if we could avoid the extra Read call in most cases by adding an extra byte to the previous call.

For example, with a limit of 8 bytes, if the first Read call asked for 20 bytes, we could ask for 9 bytes from the underlying reader rather than 8 bytes and then another single byte.

I have an implementation that does that, but I haven't yet been able to make it as simple as I'd like.

Then again, maybe that could be considered to be premature optimisation. What do you think?

[ianlancetaylor]

@rogpeppe I changed the CL as you suggest. It's a fair bit more complicated.

[ianlancetaylor]

Note that with the new implementation in CL 410035 confusing things can happen if users change the R or N fields of the LimitedReader. That seems hard to avoid.

[carlmjohnson]

I find the changes sort of confusing and have a slight preference that LimitedReader is reverted to the 1.18 status quo ante. With so many caveats, it’s hard to know when I would use LimitedReader.Err.

[bcmills]

Yeah, I think my main takeaway is that a LimitedWriter seems like an ultimately cleaner approach than trying to tune the LimitedReader semantics to always do something intuitive.

[rogpeppe]

@bcmills In my use case at least, using a LimitedWriter was not an option - we needed to limit the amount of data incoming to a streaming parser. There was no Writer involved.

FWIW, this was as far as I got with my version of the code (I don't think it's quite there yet):

func (l *LimitedReader) Read(p []byte) (int, error) {
	if int64(len(p)) < l.N {
		// Common case: not near the limit.
		n, err := l.R.Read(p)
		if n <= 0 {
			return n, err
		}
		l.N -= int64(n)
		return n, err
	}
	if l.Err == nil || l.Err == EOF {
		// No limit-exceeded special case.
		if l.N <= 0 {
			return 0, EOF
		}
		n, err := l.R.Read(p[0:l.N])
		l.N -= int64(n)
		return n, err
	}
	// Limit-exceeded special case: need to read one more byte to
	// determine if limit was actually exceeded. We only know that
	// if we've read more than the limit (i.e. l.N is negative)
	if l.N < 0 {
		return 0, l.Err
	}
	if int64(len(p)) > l.N+1 {
		p = p[:l.N+1]
	}
	n, err := l.R.Read(p)
	l.N -= int64(n)
	if l.N < 0 {
		// Don't return the extra byte that was requested.
		n--
		err = l.Err
	}
	return n, err
}

I went with not adding any new fields to LimitedReader and using a negative N as a signifier that we have actually read past the end of input (which is arguably somewhat intuitive - we will actually have read one more than the total requested number of bytes).

[rsc]

Why are we worrying so much about the extra Read?
Isn't the point of using a limited reader that you don't expect to reach the limit?
Does the "actually hit the limit" case really need to optimize away the final read?

[rsc]

In any event, the beta is upon us, @ianlancetaylor is away, and this isn't critical -
anyone who needs a custom err from LimitedReader can easily use a copy of the code.
I'm going to send a revert CL, and we can figure out the right implementation for next cycle.

[gopherbot]

Change https://go.dev/cl/410133 mentions this issue: io: revert: add an Err field to LimitedReader

[rogpeppe]

Does the "actually hit the limit" case really need to optimize away the final read?

I tend to agree that this indeed premature optimisation. I thought there might be an elegant way of doing it without significantly increasing code complexity, in which case it might be worth doing, but that doesn't appear to be the case.

[carlmjohnson]

Isn't the point of using a limited reader that you don't expect to reach the limit?

There are two cases for LimitedReader. One is that you don't want to process more than X amount of data, and then hitting the limit is unusual, and it's pretty much fine to have a small stray read. But another case is you're parsing some data directly out of a reader but you want it chunked into fixed sized units, and then overreading make you lose your place in the stream. For the second case, I think you'd mostly want to use io.EOF anyway, but it's weird if you get different behavior by setting Err.

[gopherbot]

Change https://go.dev/cl/410357 mentions this issue: doc/go1.19: remove TODO about LimitedReader

[rsc]

Rolled back the doc TODO as well. Moving to Go 1.20.