Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: read and write support for forest.car.zst files #3213

Merged
merged 111 commits into from
Jul 26, 2023

Conversation

aatifsyed
Copy link
Contributor

@aatifsyed aatifsyed commented Jul 17, 2023

Summary of changes

Changes introduced in this pull request:

  • Add snapshot compress command for generating .forest.car.zst files.
  • Change all other commands to accept either .car, .car.zst or .forest.car.zst.

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

@aatifsyed aatifsyed force-pushed the aatifsyed/index-compressed-cars branch from eae9728 to 213c80c Compare July 18, 2023 11:18
@aatifsyed aatifsyed mentioned this pull request Jul 18, 2023
11 tasks
@lemmih lemmih self-assigned this Jul 22, 2023
@lemmih lemmih marked this pull request as ready for review July 26, 2023 10:28
@lemmih lemmih requested a review from a team as a code owner July 26, 2023 10:28
@lemmih lemmih requested review from LesnyRumcajs and elmattic and removed request for a team July 26, 2023 10:28
Copy link
Contributor

@elmattic elmattic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but it's really big for a single PR!

benches/car-index.rs Outdated Show resolved Hide resolved
src/db/car/any.rs Show resolved Hide resolved
src/db/car/forest.rs Show resolved Hide resolved
src/db/car/forest.rs Show resolved Hide resolved
src/db/car/forest.rs Show resolved Hide resolved
Copy link
Member

@LesnyRumcajs LesnyRumcajs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd have to spend at least a few days to understand this PR, so accepting on the basis of "I didn't see any intentional malicious code here", faith and a green checkmark.

@@ -0,0 +1,66 @@
// Copyright 2019-2023 ChainSafe Systems
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this benchmark compiled in the CI?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not. :( We'll fix that in a follow-up PR.

impl CarIndexBuilder {
// Number of buckets given `len` number of elements
pub fn capacity_at(len: usize) -> usize {
len * 100 / 81
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not clear where are those numbers coming from to me.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's the load-factor. I'll add some more documentation.

@lemmih lemmih enabled auto-merge July 26, 2023 18:33
@lemmih lemmih added this pull request to the merge queue Jul 26, 2023
Merged via the queue into main with commit 2716545 Jul 26, 2023
20 checks passed
@lemmih lemmih deleted the aatifsyed/index-compressed-cars branch July 26, 2023 19:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants