New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
io: add an Err
field to LimitedReader
#51115
Comments
I have forked |
If the proposed function doesn't use type MyLimitedReader struct {
r io.LimitedReader
err error
}
func (mlr *MyLimitedReader) Read(p []byte) (int, error) {
n, err := mlr.Read(p)
if err == io.EOF {
err = mlr.err
}
return n, er
} Is that correct? If so, perhaps this proposal should be rewritten to add an |
MaxBytesReader
that doesn't require a ResponseWriter
Err
field to LimitedReader
I like the new proposal. Returning io.EOF makes io.LimitedReader pretty unhelpful. But I think it would also be good to document that http.MaxBytesReader accepts nil ResponseWriter. |
FWIW, my |
Can someone tag this proposal as under review? |
As an alternative to defining your own error, I just created a package and export the error so the caller can check against it.
|
I misunderstood the proposal. |
@carlmjohnson It is already so tagged. We'll get to it soon, I think. |
Change https://go.dev/cl/396215 mentions this issue: |
This seems reasonable. Thanks for the proposal. |
This proposal has been added to the active column of the proposals project |
Does anyone object to adding this? |
Based on the discussion above, this proposal seems like a likely accept. |
No change in consensus, so accepted. |
Err
field to LimitedReader
Err
field to LimitedReader
Thanks. Working on it. |
Seemed like a natural job for fuzzing, but I couldn't figure out how to make fuzzing work with the standard library, so I did it in a separate module and just copied some interesting cases back. |
Change https://go.dev/cl/405854 mentions this issue: |
@rogpeppe, I'm looking at your example again, but I don't really understand why it's a problem with
I think if the caller really wants to get EOF instead of the |
@bcmills the point I'm trying to make is that when the limit isn't exceeded, we shouldn't see an error from In my eyes, it makes most sense for the error to be returned when the limit is exceeded, not when it's reached. Of course, with the old LimitReader, there is no distinction between the two cases when the number of bytes exactly equals the limit, but with the new error there is. |
Options:
Any other ideas? |
The more I think about it, the more I think that the original goal isn't nicely attainable just by adding an
One possibility might be to document that an extra byte will be read when Err is non-nil, precisely to distinguish the two cases. Another might be to add another entry point rather than modifying |
@rogpeppe, when the limit isn't exceeded, we don't see an error from There is no general way to force a reader to tell you that it is at EOF other than trying to read a nonzero number of bytes. (A 0-length read could legitimately return |
That's fine AFAICS. In that case, the limit hasn't been exceeded so (to use the original use case) returning a "request too large" error would not be appropriate. |
I think for that use-case, if you want to accept a request of up to N bytes then you'd want to set the LimitedReader's initial limit to N+1. Then you check the (But I guess at that point it doesn't really matter what you set the |
But that means that you can exceed the limit when copying, which surely isn't great? |
FWIW, I find the naming around io.LimitReader vs io.LimitedReader confusing (it makes sense if I think about it, verb vs noun, but I have to think), so I would be happy with a new name. |
...and that is exactly why I have a |
|
I don't see how it makes sense to use |
I suppose one scenario is that for example, I'm copying a http.Response to disk, but I want to bail out after some size to prevent abuse, and in that case I want to know that it was a short write and delete what's been written so far and possibly signal to some operator or user that the file couldn't be saved. However, another way to tackle that, which might make more sense is to use a LimitWriter. We could drop the changes to LimitedReader and create that instead. |
like @jimmyfrasche, i've also forked Given all these new questions it seems prudent to put this off until after Go 1.19 rather than get stuck with this less-than-great behavior forever. |
This should be tagged with needs-decision / release blocker. |
I don't understand this assertion. The implementation that I've seen will return the specified error only when the limit is reached. In all other circumstances (and importantly when EOF is reached before the limit) the error from the underlying reader is returned. AIUI the question that we're debating here is what error to return when the size of the data available from the underlying reader is exactly the limit. It is my contention that in all the use cases I've seen so far, we want the special error to apply when the limit has been exceeded, not just reached, because otherwise the contract isn't clear: we either allow too many bytes to be transferred or we return a "limit exceeded" error when the limit hasn't actually been exceeded. It's not possible to know whether the limit has been reached until at least the LimitedReader has attempted to read at least one more byte than the limit, but that breaks the current contract. For the record, the specific situation I used this last was for stream-parsing the body of an HTTP request. The parser itself isn't aware of the request size limit (and neither should it be). The limit is documented, so it's reasonable for clients to send a request with exactly the limit number bytes and it's reasonable for the reader to assume that the number of bytes won't be more than that |
Close. The problem is the case of where the underlying data is exactly the limit, but the underlying reader does not return For example, here's the scenario without io.LimitedReader: r := someReaderThatReads100Bytes()
var buf [100]byte
n, err := r.Read(buf[:])
// Right now, n == 100 and err == nil.
n, err = r.Read(buf[:])
// And now n == 0 and err == io.EOF. And here's the same scenario with it: r := &io.LimitedReader{
R: someReaderThatReads100Bytes(),
N: 100,
Err: ErrCustom,
}
var buf [100]byte
n, err := r.Read(buf[:])
// Right now, n == 100 and err == nil.
n, err = r.Read(buf[:])
// And now n == 0 and err == ErrCustom, despite the fact that
// EOF was actually reached with the first call to Read(). The only solution is to do an extra read after the limit has been reached, but the whole point of |
It's true that if you know for sure that the That aside, clearly |
I'm not sure what you mean by "get confused" here. Returning a non-EOF error when the underlying reader returns one is standard and documented behaviour for |
If we have read N bytes and get called for Read again, and Err != io.EOF and Err != nil, is it wrong to read 1 byte from the underlying reader to find out whether to use io.EOF or Err? In this case the LimitedReader would be limiting the amount returned by its own Read method to N bytes and limiting the amount read from below to N+1 bytes (but only if Err != io.EOF and Err != nil). Since this is entirely new behavior, reading N+1 bytes underneath seems OK. Does anyone have any reasons otherwise? Thanks. |
@rogpeppe Sorry for the wording. Let me try it this way: why would you want to use |
@rsc I think that sounds like a very reasonable way forward. FWIW I wouldn't suggest actually issuing an extra 1-byte read - instead just truncate the read on the underlying reader by one less byte (and don't return that extra byte). That way, in the usual case we'll issue exactly the same number of underlying read calls. @ianlancetaylor The issue applies to anything that's using a reader. |
IIUC, then the fix is just to change the doc to “If the underlying reader can read exactly N bytes, whether LimitedReader returns the sentinel or io.EOF is undefined.” |
That's not my understanding. The behaviour should be defined - if the exactly N bytes can be read from the underlying reader, the final error should be io.EOF. If more than N bytes can be read, exactly N bytes should be returned and the final error should be Err. |
New Proposal
As per @ianlancetaylor's suggestion, this proposal is now to add an
Err
field toio.LimitedReader
. If notnil
, this field's value will be returned when trying to read past the limit imposed by the reader instead of the defaultio.EOF
.Original Proposal
For various reasons, I had a need to use
MaxBytesReader()
in a situation where aResponseWriter
wasn't easily available, so I looked through the code to see if it was safe to passnil
and found that all it does is check a type assertion and then call an unexported method on it:While it's true that because of this it's safe to pass it a
nil
ResponseWriter
, it feels quite odd to rely on undocumentated behavior of this kind.Proposal
Several ways to clean this situation up come to mind:
http
package that doesn't need aResponseWriter
. Because of how it works, it could actually just return the same implementation, but the function will be more obvious in its usage.ResponseWriter
and explicitly state that anil
ResponseWriter
is valid.MaxBytesReader()
inio
that doesn't use aResponseWriter
at all and isn't tied tohttp
behavior, but includes the other differences. This could actually be done, for the most part, by just adding anErr error
field toio.LimitedReader
that, if non-nil, is the error returned when the limit is reached instead ofio.EOF
.In whichever case, the documentation for
MaxBytesReader()
should probably be filled in a bit. It is quite odd that something calledReader
requires a type that is primarily anio.Writer
implementation for non-obvious reasons and doesn't mention it anywhere in the documentation.The text was updated successfully, but these errors were encountered: