Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StorageProvider development #91

Closed
wants to merge 131 commits into from
Closed
Show file tree
Hide file tree
Changes from 127 commits
Commits
Show all changes
131 commits
Select commit Hold shift + click to select a range
f781ca8
Update p2panda-rs in Cargo.toml
sandreae Mar 11, 2022
54cc334
Update p2panda-rs again
sandreae Mar 11, 2022
18d7b3f
Handle new SchemaId type
sandreae Mar 11, 2022
e9b265b
Add basic storage traits for entry
sandreae Mar 14, 2022
a6de0a4
Request and response traits
sandreae Mar 14, 2022
8a23f18
Conversion traits for log
sandreae Mar 14, 2022
1c37070
Add storage traits for log
sandreae Mar 14, 2022
8f512a5
Implement EntryStore trait
sandreae Mar 14, 2022
38ef005
Implement StorageProvider trait for SqlStorage
sandreae Mar 14, 2022
4201a32
A lot of tidy up
sandreae Mar 14, 2022
91acc25
Implement high level API calls and update tests
sandreae Mar 14, 2022
5e9e935
Change p2panda-rs to git target
sandreae Mar 14, 2022
839150f
Fix typo
sandreae Mar 14, 2022
8ba3a63
Remove unused imports
sandreae Mar 14, 2022
54a111e
WIP: rework conversion traits
sandreae Mar 15, 2022
8f98e21
Update storage provider API
sandreae Mar 15, 2022
6692940
Update storage traits
sandreae Mar 23, 2022
8f03066
Cargo.lock update
sandreae Mar 23, 2022
c3c8e63
Merge branch 'main' into storage-provider
sandreae Mar 23, 2022
ff25528
Update cargo lock again
sandreae Mar 23, 2022
40dd09a
Fix after merge
sandreae Mar 23, 2022
81ba846
Import order
sandreae Mar 23, 2022
85eae06
Update for storage_provider API changes
sandreae Mar 24, 2022
7e705fd
Bump bamboo-yasmf to 0.1.1
sandreae Apr 13, 2022
5efb859
Update storage provider API
sandreae Apr 13, 2022
a0b68d0
Bump axum
sandreae Apr 13, 2022
45e97c7
Bump p2panda-rs
sandreae Apr 13, 2022
4c6c68e
Correctly return payload bytes
sandreae Apr 13, 2022
d83d9f8
Remove unused imports
sandreae Apr 13, 2022
8634eee
Update CHANGELOG
sandreae Apr 13, 2022
399d6f2
Remove aquadoggo Entry struct
sandreae Apr 15, 2022
0c79cd0
Update p2panda-rs storage traits API
sandreae Apr 17, 2022
0926554
Slightly restructure store methods
sandreae Apr 20, 2022
68c863d
Tidy imports
sandreae Apr 20, 2022
b1b2d54
Remove println
sandreae Apr 24, 2022
969c3fb
Update p2panda-rs
sandreae Apr 24, 2022
ba90ae7
Reinstate missing tests
sandreae Apr 24, 2022
4e9a22e
Add migration file for operations
sandreae Apr 24, 2022
6e9b832
Define operation structs
sandreae Apr 25, 2022
84f3396
Define operation structs and storage traits
sandreae Apr 25, 2022
97e93bf
Add get method
sandreae Apr 25, 2022
a73a653
Track p2panda main branch in Cargo.toml
sandreae Apr 26, 2022
9a578ad
SQL queries for inserting an operation
sandreae Apr 26, 2022
769cc78
Fix column name in operation_v1
sandreae Apr 26, 2022
c0e421c
Test for insert operation
sandreae Apr 26, 2022
da14659
Correct logic error when deriving document_view_id
sandreae Apr 26, 2022
b8f8799
WIP get_operation_by_id method
sandreae Apr 26, 2022
3ee882a
Add comment about storing operation value as string
sandreae Apr 26, 2022
cbf539c
Add comment about fields on DoggoOperation
sandreae Apr 26, 2022
884d73c
Small clippy make happy
sandreae Apr 27, 2022
1764fcc
Refactor DoggoOperation fields and constructor
sandreae Apr 27, 2022
0de1553
Comment fix
sandreae Apr 27, 2022
8bd99b3
Temp solution: store whole encoded operation in value column
sandreae Apr 27, 2022
2777ccb
Add comment
sandreae Apr 27, 2022
ee13dfc
Remove comment
sandreae Apr 27, 2022
59535e9
Handle relation operation types in a special way
sandreae Apr 27, 2022
333c087
Some comments
sandreae Apr 27, 2022
8106d0f
Add all fields to test operation
sandreae Apr 27, 2022
eb676e8
Remove unused document_id field from DoggoOperation
sandreae Apr 27, 2022
f1ac8db
Remove relation columns from field table
sandreae Apr 28, 2022
75d1eb1
Add list_index column to operation_field table
sandreae Apr 28, 2022
6a53b6d
Add comments for struct fields
sandreae Apr 28, 2022
6e5f269
Remove document_view_id_hash field
sandreae Apr 28, 2022
5582b65
Make index None for none list values
sandreae Apr 28, 2022
cf94665
More comments
sandreae Apr 28, 2022
0d41269
Merge branch 'development' into db-operation
sandreae Apr 28, 2022
df4823a
Merge branch 'development' into db-operation
sandreae Apr 29, 2022
030c287
Bring back futures
sandreae Apr 29, 2022
be0fc5b
Create index for operation_field_v1 in db
sandreae Apr 29, 2022
24820f5
Remove field_type from indexing SQL
sandreae Apr 29, 2022
99a0c89
Add method for getting operation fields by operation id
sandreae Apr 29, 2022
b1e71ce
Small refactor
sandreae Apr 29, 2022
184bd6a
Rename method
sandreae Apr 29, 2022
c331a4d
Correct doc string
sandreae May 1, 2022
e82d017
Add comment
sandreae May 1, 2022
1a2c001
Create migration for document and document_view
sandreae May 2, 2022
9b3e02f
Changes to document tables
sandreae May 2, 2022
4eff1ec
Method for inserting document_views
sandreae May 2, 2022
0f1be52
Remove get_fields_by_operation_id method
sandreae May 2, 2022
7341a35
WIP: Method for getting document views
sandreae May 2, 2022
3ad8311
WIP: Resolve values form multiple operations when getting document view
sandreae May 2, 2022
29e7afc
Small tidy up
sandreae May 2, 2022
fdd26f8
Formatting
sandreae May 2, 2022
6e25f28
Comments and clean-up
sandreae May 2, 2022
02d6030
Comment in migration
sandreae May 2, 2022
91f692b
Restructure module
sandreae May 2, 2022
97ee1d7
Remove question comments now raised as issues
sandreae May 2, 2022
955748e
Use full schema id string as identifier
sandreae May 3, 2022
b46c0ce
Use full schema_id string also in DocumentView
sandreae May 3, 2022
b4c52af
Make schema_id TEXT type in db
sandreae May 3, 2022
ca5756e
Include document_id and previous_operations in operations_v1 table
sandreae May 3, 2022
4056d0f
Add get_document_by_operation_id() to OperationStore
sandreae May 3, 2022
a162659
Add get_operation_fields_by_id() to OperationStore
sandreae May 3, 2022
a4568df
WIP: More work in OperationStore
sandreae May 3, 2022
3459ec3
WIP: Sketch trait API a little more
sandreae May 3, 2022
10a9a63
A little clippy happiness
sandreae May 3, 2022
fd85df3
Add some more doc strings and comments
sandreae May 3, 2022
b865f4b
Create utils method for parsing OperationFieldRow's into OperationFields
sandreae May 3, 2022
1896f20
Change order of hashes in relations
sandreae May 3, 2022
501c1eb
Temp fix for failing Eq comp of unsorted relation lists
sandreae May 4, 2022
358fd41
Add foreign keys in operation tables
sandreae May 4, 2022
732a44c
Update p2panda-rs
sandreae May 5, 2022
f220e01
Doc strings in DocumentStore traits
sandreae May 5, 2022
85d5254
Implement get_entry_by_hash() required by new `EntryStore` trait method
sandreae May 5, 2022
f1852ed
Add rstest to dev dependencies
sandreae May 5, 2022
8513446
Add rstest to dev dependencies
sandreae May 5, 2022
744c016
Rename EntryRow to DoggoEntry
sandreae May 5, 2022
dd5dc9e
Cast seq_num to INTEGER in SQL query when retrieving latest
sandreae May 5, 2022
3f1995a
Use typed values in response structs
sandreae May 5, 2022
4717d25
WIP: Populated test db
sandreae May 5, 2022
9c6c439
Include DESC filter in sql for latest seq num
sandreae May 5, 2022
d00fbae
Tests for `EntryStore`
sandreae May 5, 2022
27eaa50
Add test cfg flag to missed test
sandreae May 5, 2022
875a30e
Remove unused imports
sandreae May 5, 2022
49efe73
Update p2panda-rs
sandreae May 5, 2022
57181be
Update test response strings
sandreae May 5, 2022
47f998d
Tests in EntryStore
sandreae May 5, 2022
bbbaa1f
Comments and arg for choosing no of entries in test_db
sandreae May 5, 2022
7e45877
Rename `document_view.rs` to `document.rs` in `store/`
sandreae May 5, 2022
62f5422
Merge branch 'development' into db-operation
sandreae May 5, 2022
5ccc359
Remove old comment
sandreae May 5, 2022
8ceb088
Remove wild SQL sketch
sandreae May 5, 2022
2104373
Use WIP branch in p2panda_rs
adzialocha May 5, 2022
5119e08
Update API for retreiving hashed version of document view id
adzialocha May 5, 2022
023a9d6
Rough formatting
adzialocha May 5, 2022
801da3a
Rename sql_store to provider module
adzialocha May 5, 2022
3c05a0b
Move db rows into models
adzialocha May 5, 2022
787c392
WIP: get_all_lipmaa_entries_for_entry() implementation
sandreae May 9, 2022
287da77
Update p2panda-rs branch
sandreae May 9, 2022
521ac28
Implement get_next_n_entries_after_seq in EntryStore
sandreae May 9, 2022
cdc818b
Complete implementation of get_all_lipmaa_entries_for_entry in EntryS…
sandreae May 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
152 changes: 91 additions & 61 deletions Cargo.lock

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion aquadoggo/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ bamboo-rs-core-ed25519-yasmf = "0.1.1"
crossbeam-queue = "0.3.5"
directories = "3.0.2"
envy = "0.4.2"
futures = "0.3.17"
hex = "0.4.3"
jsonrpc-v2 = { version = "0.10.1", features = [
"easy-errors",
Expand All @@ -32,7 +33,8 @@ log = "0.4.14"
openssl-probe = "0.1.4"
# We can not publish the `aquadoggo` crate yet, since `p2panda-rs` is an
# unpublished dependency.
p2panda-rs = { git = "https://github.com/p2panda/p2panda", branch = "main" }
# @TODO: This points at a WIP branch in p2panda_rs which is used as long as things are slightly unstable
p2panda-rs = { git = "https://github.com/p2panda/p2panda", branch = "aquadoggo-wip" }
rand = "0.8.4"
serde = { version = "1.0.130", features = ["derive"] }
serde_json = "1.0.67"
Expand All @@ -59,6 +61,7 @@ reqwest = { version = "0.11.9", default-features = false, features = [
"json",
"stream",
] }
rstest = "0.12.0"
tower-service = "0.3.1"
hyper = "0.14.17"
http = "0.2.6"
Expand Down
32 changes: 32 additions & 0 deletions aquadoggo/migrations/20220424125654_create-operations.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
-- SPDX-License-Identifier: AGPL-3.0-or-later

CREATE TABLE IF NOT EXISTS operations_v1 (
author VARCHAR(64) NOT NULL,
document_id VARCHAR(64) NOT NULL,
operation_id VARCHAR(64) NOT NULL UNIQUE,
entry_hash VARCHAR(68) NOT NULL UNIQUE,
action VARCHAR(16) NOT NULL,
schema_id TEXT NOT NULL,
previous_operations TEXT NULL,
-- FOREIGN KEY(entry_hash) REFERENCES entries(entry_hash),
PRIMARY KEY (operation_id)
);

-- With the above "previous_operations" column in operations_v1 table we technically no longer need this
-- relation table. Can we remove it or are there reasons it's useful?
CREATE TABLE IF NOT EXISTS previous_operations_v1 (
parent_operation_id VARCHAR(64) NOT NULL,
child_operation_id VARCHAR(64) NOT NULL,
FOREIGN KEY(parent_operation_id) REFERENCES operations_v1(operation_id)
FOREIGN KEY(child_operation_id) REFERENCES operations_v1(operation_id)
);

CREATE TABLE IF NOT EXISTS operation_fields_v1 (
operation_id VARCHAR(64) NOT NULL,
name VARCHAR(128) NOT NULL,
field_type TEXT NOT NULL,
value BLOB NULL,
FOREIGN KEY(operation_id) REFERENCES operations_v1(operation_id)
);

CREATE INDEX idx_operation_fields_v1 ON operation_fields_v1 (operation_id, name);
21 changes: 21 additions & 0 deletions aquadoggo/migrations/20220502013246_create-documents.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
-- SPDX-License-Identifier: AGPL-3.0-or-later

CREATE TABLE IF NOT EXISTS document_view_fields (
document_view_id VARCHAR(64) NOT NULL,
operation_id VARCHAR(64) NOT NULL,
name VARCHAR(128) NOT NULL
);

CREATE INDEX idx_document_view_fields ON document_view_fields (document_view_id, operation_id, name);

CREATE TABLE IF NOT EXISTS document_views (
document_view_id VARCHAR(64) NOT NULL UNIQUE,
schema_id TEXT NOT NULL,
PRIMARY KEY (document_view_id)
);

CREATE TABLE IF NOT EXISTS documents (
document_id VARCHAR(64) NOT NULL UNIQUE,
document_view_id VARCHAR(64) NOT NULL,
PRIMARY KEY (document_id)
);
17 changes: 17 additions & 0 deletions aquadoggo/src/db/errors.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
// SPDX-License-Identifier: AGPL-3.0-or-later

/// `OperationStore` errors.
#[derive(thiserror::Error, Debug)]
pub enum OperationStorageError {
/// Catch all error which implementers can use for passing their own errors up the chain.
#[error("Ahhhhh!!!!: {0}")]
Custom(String),
}

/// `DocumentStore` errors.
#[derive(thiserror::Error, Debug)]
pub enum DocumentViewStorageError {
/// Catch all error which implementers can use for passing their own errors up the chain.
#[error("Ahhhhh!!!!: {0}")]
Custom(String),
}
6 changes: 5 additions & 1 deletion aquadoggo/src/db/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,12 @@ use sqlx::any::{Any, AnyPool, AnyPoolOptions};
use sqlx::migrate;
use sqlx::migrate::MigrateDatabase;

pub mod errors;
pub mod models;
pub mod store;
pub mod provider;
pub mod stores;
pub mod traits;
pub mod utils;

/// Re-export of generic connection pool type.
pub type Pool = AnyPool;
Expand Down
Loading