-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[YSQL] Use lightweight protobuf for writing to tserver #11720
Labels
area/ysql
Yugabyte SQL (YSQL)
Comments
spolitov
added a commit
that referenced
this issue
Mar 25, 2022
Summary: We use google protobuf for cummunicating between nodes or postgres and tserver. Native protobuf implementation allocates a lot of memory while generating requests. This diff switches Perform RPC to our lightweight protobuf implementation. What uses arena for allocation and tries to avoid allocation as much as possible. Test Plan: Jenkins Reviewers: lnguyen Reviewed By: lnguyen Subscribers: smishra, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D16088
This was referenced Mar 25, 2022
Initial performance test (on copying 10M rows):
(e.g. copy time reduced by 6.8% compared to baseline). |
spolitov
added a commit
that referenced
this issue
Mar 28, 2022
Summary: This diff adds arena to PgStatement and switches PgConstant to LWQLValuePB to avoid superfluous conversions. Test Plan: Jenkins Reviewers: lnguyen Reviewed By: lnguyen Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D16220
|
spolitov
added a commit
that referenced
this issue
Mar 29, 2022
…protobufs Summary: Currenly we could evaluate expression represented as google protobufs. This diff adds ability to evaluate expression represented as lightweight protobufs. Test Plan: Jenkins Reviewers: bogdan, dmitry Reviewed By: bogdan, dmitry Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D16221
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Currently when running bulk insert / copy command, in the postgres backend for, about 15 percent of CPU time is spent on memory allocation / deallocation.
Hence we could see good reduction of copy time if we optimize for this allocation/deallocation process. One of such idea is to use Protobuf's arena: https://developers.google.com/protocol-buffers/docs/reference/arenas. This will optimize for allocating & deallocation the protobufs itself.
We'd like to make this change for the copy/write path to the tserver and test how much time reduction it would take with this change enabled.
The text was updated successfully, but these errors were encountered: