-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement capnproto replication #7659
base: main
Are you sure you want to change the base?
Conversation
7fdf16a
to
253f417
Compare
Our profiles from production show that a lot of CPU and memory in receivers is used for unmarshaling protobuf messages. Although it is not possible to change the remote-write format, we have the freedom to change the protocol used for replicating timeseries data. This commit introduces a new feature in receivers where replication can be done using Cap'n Proto instead of gRPC + Protobuf. The advantage of the former protocol is that deserialization is far cheaper and fields can be accessed directly from the received message (byte slice) without allocating intermediate objects. There is an additional cost for serialization because we have to convert from Protobuf to the Cap'n proto format, but in our setup this still results in a net reduction in resource usage. Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
253f417
to
7042bea
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't this include a whole new RPC framework? I'm very interested in this. Glad to finally see #7071 tackled. How do you think this compared to dRPC? And is it worth eschewing gRPC altogether?
if err != nil { | ||
return err | ||
} | ||
go func() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to use a pool of goroutines here (and with a limit)?
Really interested in this too! Our profiles show similar for unmarshaling |
Yes this comes with its own RPC framework which can be a double edge sword. It's one more thing to learn and troubleshoot if it goes wrong. I haven't looked into dRPC yet in practice, but maybe there is a way to abstract the RPC framework and not expose internals to end users, this way we can swap it out in future releases in a transparent manner. In this PR we expose the listen port as a parameter since we need separate connections for Cap'n Proto. |
DRPC allows serving both gRPC and DRPC on the same socket to allow for incrementally upgrading/migrating a fleet. We might consider doing something similar whether we go with capnproto or DRPC |
Mhm, we probably need to experiment more with the transparent approach before deciding what to do. For example, we could maybe deprecate My only suggestion would be:
Instead of having two addresses here, let's have |
Perhaps we can use something like https://github.com/soheilhy/cmux to multiplex multiple protocols on the same port. This way we don't need to pollute arguments with flags for each protocol. Let me see if this is easily doable, and also add some tests for the new RPC. |
a752425
to
729ad10
Compare
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
d22809b
to
5cfae4c
Compare
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
2e259aa
to
3df360a
Compare
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
3df360a
to
2fdeba7
Compare
Signed-off-by: Filip Petkovski <filip.petkovsky@gmail.com>
Our profiles from production show that a lot of CPU and memory in receivers is used for unmarshaling protobuf messages. Although it is not possible to change the remote-write format, we have the freedom to change the protocol used for replicating timeseries data.
This commit introduces a new feature in receivers where replication can be done using Cap'n Proto instead of gRPC + Protobuf. The advantage of the former protocol is that deserialization is far cheaper and fields can be accessed directly from the received message (byte slice) without allocating intermediate objects. There is an additional cost for serialization because we have to convert from Protobuf to the Cap'n proto format, but in our setup this still results in a net reduction in resource usage.
We have a split router-receiver setup and this is our resource usage in staging after enabling the new replication method:
Router usage did go up as well, but we still got an overall net reduction:
We can also experiment with other formats by having a more generic flag:
receive.replication-format=...
Changes
Verification