New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
io: add an Err
field to LimitedReader
#51115
Comments
I have forked |
If the proposed function doesn't use type MyLimitedReader struct {
r io.LimitedReader
err error
}
func (mlr *MyLimitedReader) Read(p []byte) (int, error) {
n, err := mlr.Read(p)
if err == io.EOF {
err = mlr.err
}
return n, er
} Is that correct? If so, perhaps this proposal should be rewritten to add an |
MaxBytesReader
that doesn't require a ResponseWriter
Err
field to LimitedReader
I like the new proposal. Returning io.EOF makes io.LimitedReader pretty unhelpful. But I think it would also be good to document that http.MaxBytesReader accepts nil ResponseWriter. |
FWIW, my |
Can someone tag this proposal as under review? |
As an alternative to defining your own error, I just created a package and export the error so the caller can check against it.
|
I misunderstood the proposal. |
@carlmjohnson It is already so tagged. We'll get to it soon, I think. |
Change https://go.dev/cl/396215 mentions this issue: |
This seems reasonable. Thanks for the proposal. |
This proposal has been added to the active column of the proposals project |
Does anyone object to adding this? |
Based on the discussion above, this proposal seems like a likely accept. |
No change in consensus, so accepted. |
Err
field to LimitedReader
Err
field to LimitedReader
Sounds like we agree to arrange to read one extra byte (whether in the final normal read or in an extra one, depending on room) and use that to decide EOF or not. |
This is currently labeled to block beta 1 which we're aiming to release next week if possible. Do you think the change above can happen before then, or otherwise is it safe to land it after beta 1 (possibly in time for beta 2 or so)? Thanks. |
Change https://go.dev/cl/410035 mentions this issue: |
@rogpeppe Does https://go.dev/cl/410035 act as you expected? Thanks. |
@ianlancetaylor Technically yes, but I think it would be preferable if we could avoid the extra Read call in most cases by adding an extra byte to the previous call. For example, with a limit of 8 bytes, if the first Read call asked for 20 bytes, we could ask for 9 bytes from the underlying reader rather than 8 bytes and then another single byte. I have an implementation that does that, but I haven't yet been able to make it as simple as I'd like. Then again, maybe that could be considered to be premature optimisation. What do you think? |
@rogpeppe I changed the CL as you suggest. It's a fair bit more complicated. |
Note that with the new implementation in CL 410035 confusing things can happen if users change the R or N fields of the LimitedReader. That seems hard to avoid. |
I find the changes sort of confusing and have a slight preference that LimitedReader is reverted to the 1.18 status quo ante. With so many caveats, it’s hard to know when I would use LimitedReader.Err. |
Yeah, I think my main takeaway is that a |
@bcmills In my use case at least, using a FWIW, this was as far as I got with my version of the code (I don't think it's quite there yet):
I went with not adding any new fields to |
Why are we worrying so much about the extra Read? |
In any event, the beta is upon us, @ianlancetaylor is away, and this isn't critical - |
Change https://go.dev/cl/410133 mentions this issue: |
I tend to agree that this indeed premature optimisation. I thought there might be an elegant way of doing it without significantly increasing code complexity, in which case it might be worth doing, but that doesn't appear to be the case. |
There are two cases for LimitedReader. One is that you don't want to process more than X amount of data, and then hitting the limit is unusual, and it's pretty much fine to have a small stray read. But another case is you're parsing some data directly out of a reader but you want it chunked into fixed sized units, and then overreading make you lose your place in the stream. For the second case, I think you'd mostly want to use io.EOF anyway, but it's weird if you get different behavior by setting Err. |
Change https://go.dev/cl/410357 mentions this issue: |
We are having a hard time deciding the exact semantics of the Err field, and we need to ship the beta. So revert the Err field change; it can wait for Go 1.20. For #51115. This reverts CL 396215. Change-Id: I7719386567d3da10a614058a11f19dbccf304b4d Reviewed-on: https://go-review.googlesource.com/c/go/+/410133 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com> Auto-Submit: Russ Cox <rsc@golang.org>
Rolled back the doc TODO as well. Moving to Go 1.20. |
Rolled back in CL 410133. For #51115. Change-Id: I009c557acf98a98a9e5648fa82d998d41974ae60 Reviewed-on: https://go-review.googlesource.com/c/go/+/410357 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
New Proposal
As per @ianlancetaylor's suggestion, this proposal is now to add an
Err
field toio.LimitedReader
. If notnil
, this field's value will be returned when trying to read past the limit imposed by the reader instead of the defaultio.EOF
.Original Proposal
For various reasons, I had a need to use
MaxBytesReader()
in a situation where aResponseWriter
wasn't easily available, so I looked through the code to see if it was safe to passnil
and found that all it does is check a type assertion and then call an unexported method on it:While it's true that because of this it's safe to pass it a
nil
ResponseWriter
, it feels quite odd to rely on undocumentated behavior of this kind.Proposal
Several ways to clean this situation up come to mind:
http
package that doesn't need aResponseWriter
. Because of how it works, it could actually just return the same implementation, but the function will be more obvious in its usage.ResponseWriter
and explicitly state that anil
ResponseWriter
is valid.MaxBytesReader()
inio
that doesn't use aResponseWriter
at all and isn't tied tohttp
behavior, but includes the other differences. This could actually be done, for the most part, by just adding anErr error
field toio.LimitedReader
that, if non-nil, is the error returned when the limit is reached instead ofio.EOF
.In whichever case, the documentation for
MaxBytesReader()
should probably be filled in a bit. It is quite odd that something calledReader
requires a type that is primarily anio.Writer
implementation for non-obvious reasons and doesn't mention it anywhere in the documentation.The text was updated successfully, but these errors were encountered: