Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"prune" materializer task #496

Merged
merged 23 commits into from
Aug 11, 2023
Merged

"prune" materializer task #496

merged 23 commits into from
Aug 11, 2023

Conversation

sandreae
Copy link
Member

@sandreae sandreae commented Aug 9, 2023

Introduces a "prune" materialization task which:

  • is issued whenever a "reduce" task completes when the input was a DocumentId (this is a new current view)
  • the "prune" task looks at the updated document and deletes any rows from the document_views and document_view_fields table where: they are not the current view AND they are not targeted from a pinned relation of any other document

I also had to make some changes to prune_document_views method which was introduced onto the store in a previous PR cos it was not doing what it's supposed to....

📋 Checklist

  • Add tests that cover your changes
  • Add this PR to the Unreleased section in CHANGELOG.md
  • Link this PR to any issues it closes
  • New files contain a SPDX license header

@sandreae sandreae changed the title "prune" task "prune" materializer task Aug 9, 2023
@codecov
Copy link

codecov bot commented Aug 9, 2023

Codecov Report

Patch coverage: 97.55% and project coverage change: +0.99% 🎉

Comparison is base (81cae95) 91.20% compared to head (0a93f68) 92.20%.

❗ Current head 0a93f68 differs from pull request most recent head 8c7938f. Consider uploading reports for the commit 8c7938f to get more accurate results

Additional details and impacted files
@@               Coverage Diff               @@
##           development     #496      +/-   ##
===============================================
+ Coverage        91.20%   92.20%   +0.99%     
===============================================
  Files              101      102       +1     
  Lines            16393    16644     +251     
===============================================
+ Hits             14952    15347     +395     
+ Misses            1441     1297     -144     
Files Changed Coverage Δ
aquadoggo/src/materializer/tasks/reduce.rs 95.04% <86.20%> (-0.24%) ⬇️
aquadoggo/src/test_utils/node.rs 92.51% <96.61%> (-2.32%) ⬇️
aquadoggo/src/materializer/tasks/prune.rs 98.07% <98.07%> (ø)
aquadoggo/src/db/stores/document.rs 98.74% <99.00%> (+12.93%) ⬆️
aquadoggo/src/materializer/service.rs 94.76% <100.00%> (+0.01%) ⬆️
aquadoggo/src/materializer/tasks/dependency.rs 97.54% <100.00%> (+<0.01%) ⬆️

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sandreae sandreae linked an issue Aug 10, 2023 that may be closed by this pull request
@sandreae sandreae marked this pull request as ready for review August 10, 2023 08:52
@sandreae sandreae merged commit ffc1b74 into development Aug 11, 2023
8 checks passed
sandreae added a commit that referenced this pull request Aug 12, 2023
@sandreae
Copy link
Member Author

This work is being continued in a new branch here: #500

@sandreae sandreae deleted the prune-task branch August 12, 2023 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Garbage collection service/task for document views
1 participant