Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: lazy initialize vector builder on write #3210

Merged
merged 6 commits into from Jan 22, 2024

Conversation

waynexia
Copy link
Member

@waynexia waynexia commented Jan 21, 2024

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Lazy create MutableVector for TimeSeries. This change can save lots of memory when the data's field columns are sparse. In my test scenario where ~90% fields are null, it can cut up to 80% memory consumption:

Snipaste_2024-01-21_15-20-39

I also run a tsbs-load benchmark to see if this change slow down the normal case. The result shows it doesn't have bad influence:

before this change:

loaded 86400000 rows in 141.686sec with 6 workers (mean rate 609798.27 rows/sec)

after this change:

loaded 86400000 rows in 140.448sec with 6 workers (mean rate 615174.51 rows/sec)

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@github-actions github-actions bot added Doc not needed This PR do not impact docs Size: S labels Jan 21, 2024
Copy link

codecov bot commented Jan 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (6320590) 85.81% compared to head (0c2b2e8) 85.37%.
Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3210      +/-   ##
==========================================
- Coverage   85.81%   85.37%   -0.44%     
==========================================
  Files         832      840       +8     
  Lines      136308   137598    +1290     
==========================================
+ Hits       116967   117479     +512     
- Misses      19341    20119     +778     

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
waynexia and others added 2 commits January 22, 2024 14:32
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Copy link
Contributor

@v0y4g3r v0y4g3r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@waynexia waynexia added this pull request to the merge queue Jan 22, 2024
Merged via the queue into GreptimeTeam:main with commit 278e4c8 Jan 22, 2024
15 checks passed
@waynexia waynexia deleted the lazy-vec-builder branch January 22, 2024 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Doc not needed This PR do not impact docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants