Skip to content
This repository was archived by the owner on Aug 12, 2025. It is now read-only.

Metadata Database Part 1#117

Merged
rayokamoto merged 22 commits intomainfrom
metadata-db
Jun 3, 2025
Merged

Metadata Database Part 1#117
rayokamoto merged 22 commits intomainfrom
metadata-db

Conversation

@rishiad
Copy link
Contributor

@rishiad rishiad commented May 23, 2025

Stake guarded metadata database with CRDT to replace current DHT metadata system

TO-DO

  • remove dht stuff
  • implement metadatadb
  • incorporate metadatadb into upload routines
  • incorporate metadatadb into download routines
  • check if piece entries exist during organic uploads - save the bandwidth of retransmitting the pieces to miners.
  • make crsqlite lib directory configurable
  • update docs - include installation instructions for crsqlite

@rishiad rishiad marked this pull request as draft May 23, 2025 03:43
Shr1ftyy added 4 commits May 25, 2025 11:28
- purged PeerVerifier, StorbRecord, and StorbStore
- updated MetadataDB implementation to incorporate correct type conversions when inserting and querying to and from database
- added newtype for DateTime<Utc> (SqlDateTime) as part of the above change, and also TryForm for PieceType
- added command channel and handling to MetadataDB
- added tests for MetadataDB
- removed dht_dir param from settings.toml.example
- removed infohash, chunk, and piece insertion helpers
- using r2d2 to create connection pools to manage db connections
- refactored all insertion and query functions to be run through sender channels as `MetadataDBCommand`s
- refactored metadatdb insertion and query handler functions to be async
- updated metadatadb tests
- mostly purged dht stuff from all upload-related routines
- updated the upload processes in `upload.rs` - incorporated metadatadb.
- enhanced error handling and logging for database operations.
- updated metadatadb schema - removed redundant columns from certain tables.
- improved object metadata insertion logic.
- modified the `Validator` struct to include a sender for the metadata database.
- adjusted the `run_synthetic_challenges` method to utilize the new metadata database for chunk and piece retrieval.
- updated `MinerState` to have an `Arc` miner only - the  miner itself is no longer the critical sections for locks, but rather certain resources inside of the `Miner` (much like how components related to the validator are written to reduce blocking).
- updated validator docs
- downloading is temporarily disabled on the validator (to be addressed in upcoming commits).
Shr1ftyy added 3 commits June 1, 2025 03:44
- incorporated metadatadb into download routines
- added filenames to infohash tracker entries
- fixed issue with chunks and pieces being gathered out-of-order during uploads
- addressed some clippy warnings
- added checks to prevent uploading a piece if it already exists in the network
- fixed typos of metadata db sender variables
- added `get_piece` function to metadatadb w/ accompanying tests
- removed duplicate score attempt and success updating in store challenges
@Shr1ftyy Shr1ftyy changed the title Metadata Database Metadata Database Part 1 Jun 2, 2025
@Shr1ftyy Shr1ftyy added the enhancement New feature or request label Jun 2, 2025
@Shr1ftyy Shr1ftyy marked this pull request as ready for review June 2, 2025 05:28
@Shr1ftyy Shr1ftyy requested a review from rayokamoto June 2, 2025 05:28
Copy link
Contributor

@rayokamoto rayokamoto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing - I'm not the biggest fan of putting [u8; 32] everywhere for infohashes and piece hashes. Could we make types for these?

Shr1ftyy added 2 commits June 2, 2025 08:50
Copy link
Contributor

@rayokamoto rayokamoto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address the u8 slices not having types as well

@Shr1ftyy Shr1ftyy requested a review from rayokamoto June 3, 2025 08:11
@rayokamoto rayokamoto merged commit 95e51a3 into main Jun 3, 2025
1 check passed
@rayokamoto rayokamoto deleted the metadata-db branch June 3, 2025 11:46
@Shr1ftyy Shr1ftyy linked an issue Jun 11, 2025 that may be closed by this pull request
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MetadataDB Part I

3 participants