Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msgp: allow for unbuffered writes to prevent allocations #291

Closed
lithdew opened this issue Jun 9, 2021 · 1 comment
Closed

msgp: allow for unbuffered writes to prevent allocations #291

lithdew opened this issue Jun 9, 2021 · 1 comment

Comments

@lithdew
Copy link

lithdew commented Jun 9, 2021

Hello,

This is more of a minor feature request - happy to also do a PR for it.

As of now, msgp.Writer assumes that all writes are buffered.

There are many cases where writes need not be buffered, which could save an extra allocation and remove the need for the application to make use of the global writer sync.Pool when calling msgp.Encode.

An example case where buffered writes aren't necessary and aren't worth an extra allocation would be when you would like to compute the checksum of a MessagePack struct by writing its contents into a hash.Hasher.

This could be done without requiring any breaking changes by adding in a condition to all (*msgp.Writer).Write* methods, where if (*msgp.Writer).buf is nil, directly write to (*msgp.Writer).w instead of buffering writes into (*msgp.Writer).buf.

A more general breaking change that I'd also like to propose as an alternative to the above is to refactor the entire API to not assume writes are buffered at all. To do this, msgp.Writer could be removed and replaced in favor of io.Writer, and users may then choose to wrap bufio.Writer over their provided io.Writer should they wish for writes to be buffered.

@lithdew lithdew changed the title msgp: allow for unbuffered writer to prevent allocations msgp: allow for unbuffered writes to prevent allocations Jun 9, 2021
@klauspost
Copy link
Collaborator

You can use msgp.NewWriterSize, though it we set it to minimum 18 so it has the buffer space it needs.

I don't see any benefit to completely unbuffered writes, and you end up spending an unreasonable amount of time calling the upstream "Write" function for every byte.

If the allocation is an issue just re-use the writer or use msg.Encode(...).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants