Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fmt: support bounded buffers (Snprint, Snprintf, etc.) #17688

Open
therc opened this issue Oct 31, 2016 · 13 comments
Open

fmt: support bounded buffers (Snprint, Snprintf, etc.) #17688

therc opened this issue Oct 31, 2016 · 13 comments

Comments

@therc
Copy link

@therc therc commented Oct 31, 2016

Right now, printing large compound objects with %v or %#v can be very expensive, in terms of both CPU and RAM. It would be great to have a way for callers to tell formatting code to stop after N runes have been output. The current alternative is formatting everything anyway, only to discard most of the work after the call returns.

@josharian
Copy link
Contributor

@josharian josharian commented Oct 31, 2016

Haven't confirmed, but I'd guess that you can accomplish this now by using fmt.Fprintf into a byte slice with a Write method that rejects data after the buffer is full.

@martisch
Copy link
Contributor

@martisch martisch commented Oct 31, 2016

@josharian:
afaik that wont work since fmt only ever writes once https://github.com/golang/go/blob/master/src/fmt/print.go#L182 and that i think was an intentional design decision.

@josharian
Copy link
Contributor

@josharian josharian commented Oct 31, 2016

Ah. Bummer. This is not the first time I've bumped up against that particular design decision and been frustrated by it.

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Nov 1, 2016

Can't you specify a precision to limit the output? e.g. %.80v to limit the output to 80 characters? I don't know if this is actually smart about stopping early, though.

@quentinmit quentinmit added this to the Go1.9Maybe milestone Nov 1, 2016
@therc
Copy link
Author

@therc therc commented Nov 1, 2016

I think the precision gets applied to the individual fields...

@therc
Copy link
Author

@therc therc commented Nov 1, 2016

@jech
Copy link

@jech jech commented Nov 4, 2016

Couldn't this be solved by defining a LimitedWriter interface with a Remains() int64 method? (Perhaps LimitedWriter is not the best name for this, due to the asymmetry with LimitedReader.) I'd personally find it jarring if there were an Snprintf but no efficient way to do Fnprintf.

@martisch
Copy link
Contributor

@martisch martisch commented Nov 4, 2016

Regardless of the design of the writer the current design of fmt would first fill a buffer with the to be written output. Which does not seem to be a solution to having less ram and CPU use. For fmt to stop generating output after x runes directly the package would need to be redesigned to keep track of how much space is left and then stop in the respective format methods or calling them. Even then fmt has no influence of how e.g. string methods would consume cpu and ram resources and how much output they generate before trying to write it to the internal buffer.

It is currently possible to implement the https://golang.org/pkg/fmt/#GoStringer or https://golang.org/pkg/fmt/#Formatter interface for custom types that could keep track of generated runes and stop generating output themselves if a limit is reached. Formatter can use Precision() to see what precision was set.

To implement Snprintf or Fnprintf in a way that limits also cpu and ram resources used would seems to me to be a larger change and would also have a performance impact for non limited operations.

Just my observations. If there is a good design and consensus as well as proved need to have a limiting print function i am happy to work or help implementing them.

@jech
Copy link

@jech jech commented Nov 4, 2016

I was thinking of fmt doing something like

wl, ok := w.(LimitedWriter)
if ok {
    n := wl.Remains()
    // do something smart since we know that we'll only be outputting n bytes/runes/whatever
}
@martisch
Copy link
Contributor

@martisch martisch commented Nov 4, 2016

@jech i though that this would be the intended use of the LimitedWriter. And smart is likely to mean passing an extra parameter down to every format function, check it, and stop writing to the internal buffer if limit reached. fmt can not limit what handlemethods (Stringer,Formatter, ... ) will do and there is no way to communicate the limit to e.g. a String method. Also not sure what should happen if fmt than tries to write to the writer but someone else has taken up some of the remaining capacity. I guess Fnprintf would be fine without introducing LimitedWriter. The question i had is more about how the functionality would be designed to work with all the internal formatting functions.

@minux
Copy link
Member

@minux minux commented Nov 8, 2016

Actually truncating the output is not very useful either,
I'd suggest a go-gettable package for configurable formatting
that ignores or abbreviate a field if it's too large or nesting
too deep, to make the output look like this:

{s: "ABC ... (2343 chars omitted)", ptr: &D{ /* omitted */ }, ...}

I don't think such functionality belong in fmt though.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented May 24, 2017

@jech, I just came to propose the same thing as your io.LimitedWriter check. fmt already depends on io, so that's fine.

Want to send a change for Go 1.10?

@bradfitz bradfitz added NeedsFix and removed NeedsDecision labels May 24, 2017
@bradfitz bradfitz modified the milestones: Go1.10, Go1.9Maybe May 24, 2017
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented May 24, 2017

Oh, LimitedWRITER, which doesn't exist. We have io.LimitedReader. I suppose we need to have a conversation first about adding LimitedWriter, but doesn't seem super offensive.

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017
@gopherbot gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants
You can’t perform that action at this time.