Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): amortize many ready messages into fewer, larger buffers #1423

Merged
merged 2 commits into from
Sep 1, 2023

Conversation

jgraettinger
Copy link
Contributor

Introduce an EncodedBytes combinator which encodes multiple ready messages - up to a yield threshold - before yielding the next bytes buffer.

Or, if the message stream polls to pending, yield the available bytes immediately.

These ammortized buffers exhibit far better throughput when streaming a high rate of small messages, because hyper and h2 avoid copying the yielded buffer and dispatch each as a separate, non-vectorized tcp send.

Motivation

We have observed that tonic exhibits poor CPU performance when sending a high data-rate of small messages. Each message is dispatched as a single tcp_sendmsg, and each incurs overhead for routing and other kernel activities in the network path. It's incumbent to increase the size of network sends without adding latency.

Solution

Introduce a combinator which repeatedly polls its delegate stream for ready messages, extending a current buffer (up to a yield threshold). This approach avoids any new allocation and adds just one "magic" constant: a yield threshold after which the combinator will immediately yield the next chunk of bytes.

@jgraettinger jgraettinger marked this pull request as ready for review June 27, 2023 04:59
Introduce an EncodedBytes combinator which encodes multiple ready
messages - up to a yield threshold - before yielding the next bytes
buffer.

Or, if the message stream polls to pending, yield the available bytes
immediately.

These ammortized buffers exhibit far better throughput when streaming
a high rate of small messages, because hyper and h2 avoid copying the
yielded buffer and dispatch each as a separate, non-vectorized tcp send.
Copy link
Member

@LucioFranco LucioFranco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

Copy link
Member

@LucioFranco LucioFranco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we can get the merge conflicts resolved and remove fuse/use tokio-stream then we can merge it.

tonic/src/codec/encode.rs Show resolved Hide resolved
@LucioFranco LucioFranco added this pull request to the merge queue Sep 1, 2023
Merged via the queue into hyperium:master with commit 76eedc1 Sep 1, 2023
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants