Task for automatic garbage collection of unused documents and views (#…

…500) * Return removed document view ids from `prune_document_views` * Introduce "prune" task * Issue "prune" tasks from "reduce" tasks * Only issue dependency tasks in test utils * Don't issue a dependency task when reduced document is deleted * Update all tests * Clippy * More comments in prune task * Correctly process effected child relations of deleted views * Improve comments in prune task * Correct item name * Test helper method for updating documents * Pruning SQL gets a little more epic * Test for recursive pruning * fmt * Tests for "prune" task * Update CHANGELOG * Remove unused method * Improve doc string * Use transaction in prune method * SQL re-use and more comments * Change comments again * Add cascading deletes to operation_fields_v1 and document_views * Add a trigger which purges a documents logs, entries and operations from the database * Add purge method to document store w/ tests * Re-use common sql strings * Refactor prune_document_views() into separate methods * Remove prune_documents_view() method from DocumentStore * Implement pruning logic in task * Rename "prune" task "garbage_collection" * Check if all views for a document have been removed in "garbage_collection" task * fmt + clippy * Add purge_blob method to BlobStore * Use purge_blob in "garbage_collection" task * Don't remove row from `logs` table on purge * Correct method naming * Some comment improvements * Remove unused import * Move add_blob to test utils * Move assert_query method to test utils * Tests for "garbage_collection" task * Change task name to "garbage_collection" * Clippy * Update CHANGELOG entry * Move SQL trigger into purge_document method on the store * Inline all SQL query constant strings * Update CHANGELOG * Add more comments in purge_blob method * Add test for purging updated blob * Clippy * fmt * Improve comments in garbage_collection task
p2panda · Aug 31, 2023 · 6c5d477 · 6c5d477
1 parent 71208cc
commit 6c5d477
Show file tree

Hide file tree

Showing 13 changed files with 1,465 additions and 117 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Serve static files from `blobs` directory [#480](https://github.com/p2panda/aquadoggo/pull/480)
 - Add method to store for pruning document views [#491](https://github.com/p2panda/aquadoggo/pull/491)
 - Introduce `BlobStore` [#484](https://github.com/p2panda/aquadoggo/pull/484)
+- Task for automatic garbage collection of unused documents and views [#500](https://github.com/p2panda/aquadoggo/pull/500)
 
 ## [0.5.0]
 

diff --git a/aquadoggo/migrations/20220509090252_create-operations.sql b/aquadoggo/migrations/20220509090252_create-operations.sql
@@ -16,7 +16,7 @@ CREATE TABLE IF NOT EXISTS operation_fields_v1 (
     field_type              TEXT             NOT NULL,
     value                   TEXT             NULL,
     list_index              INT              NOT NULL,
-    FOREIGN KEY(operation_id) REFERENCES operations_v1(operation_id)
+    FOREIGN KEY(operation_id) REFERENCES operations_v1(operation_id) ON DELETE CASCADE
 );
 
 CREATE INDEX idx_operation_fields_v1 ON operation_fields_v1 (operation_id, name);
diff --git a/aquadoggo/migrations/20230114140233_alter-documents.sql b/aquadoggo/migrations/20230114140233_alter-documents.sql
@@ -1,3 +1,3 @@
 -- SPDX-License-Identifier: AGPL-3.0-or-later
 
-ALTER TABLE document_views ADD COLUMN document_id TEXT NOT NULL REFERENCES documents(document_id);
+ALTER TABLE document_views ADD COLUMN document_id TEXT NOT NULL REFERENCES documents(document_id) ON DELETE CASCADE;
diff --git a/aquadoggo/src/db/errors.rs b/aquadoggo/src/db/errors.rs
@@ -17,6 +17,10 @@ pub enum SqlStoreError {
     /// Error returned from BlobStore.
     #[error(transparent)]
     BlobStoreError(#[from] BlobStoreError),
+
+    /// Error returned from `DocumentStore` methods.
+    #[error(transparent)]
+    DocumentStorage(#[from] DocumentStorageError),
 }
 
 /// `SchemaStore` errors.

diff --git a/aquadoggo/src/db/stores/blob.rs b/aquadoggo/src/db/stores/blob.rs