Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Garbage collect blobs from file system #571

Merged
merged 14 commits into from
Oct 11, 2023
Merged

Conversation

sandreae
Copy link
Member

@sandreae sandreae commented Sep 22, 2023

We already garbage collect unused blob views from the database, this PR implements garbage collecting blob views from the file system as well. This occurs both when a blob view is not the current view or pinned from another document, and for the current view as well when the blob document itself is not related to by any other document.

During implementation I did a little general renaming/refactoring in the garbage collection task to help code comprehension.

📋 Checklist

  • Add tests that cover your changes
  • Add this PR to the Unreleased section in CHANGELOG.md
  • Link this PR to any issues it closes
  • New files contain a SPDX license header

@codecov
Copy link

codecov bot commented Sep 22, 2023

Codecov Report

Attention: 11 lines in your changes are missing coverage. Please review.

Comparison is base (3ac8b70) 92.43% compared to head (93ba503) 92.47%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #571      +/-   ##
==========================================
+ Coverage   92.43%   92.47%   +0.03%     
==========================================
  Files         106      106              
  Lines       18164    18396     +232     
==========================================
+ Hits        16790    17011     +221     
- Misses       1374     1385      +11     
Files Coverage Δ
aquadoggo/src/db/stores/blob.rs 98.18% <100.00%> (+<0.01%) ⬆️
aquadoggo/src/db/stores/document.rs 98.76% <100.00%> (-0.07%) ⬇️
aquadoggo/src/materializer/tasks/reduce.rs 94.92% <0.00%> (ø)
aquadoggo/src/test_utils/node.rs 95.00% <97.29%> (+0.35%) ⬆️
...doggo/src/materializer/tasks/garbage_collection.rs 97.80% <95.40%> (-0.83%) ⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sandreae
Copy link
Member Author

Very concise useful Postgres error in the CI: Error: The operation was canceled.

@adzialocha adzialocha marked this pull request as draft September 26, 2023 11:22
@sandreae sandreae linked an issue Oct 6, 2023 that may be closed by this pull request
@sandreae sandreae marked this pull request as ready for review October 7, 2023 21:39
@sandreae sandreae changed the title Purge unused/deleted blobs from filesystem Garbage collect blobs from file system Oct 7, 2023
Copy link
Member

@adzialocha adzialocha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super cool to have gc for files! ✊🏻 Only a minor change request, otherwise ready to me

aquadoggo/src/materializer/tasks/garbage_collection.rs Outdated Show resolved Hide resolved
@sandreae sandreae merged commit 94bc8e9 into main Oct 11, 2023
10 checks passed
@adzialocha adzialocha deleted the purge-blob-from-filesystem branch October 12, 2023 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Garbage collect blobs in the file system
2 participants