Split massa-db worker and exports #4010
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update cache sync cache. * Update compilation * Fix wrong variable usage. * Fix all compilation errors * Fix fmt * Change branch runtime
Versioning fixes
Co-authored-by: sydhds <sd@massa.net>
* Optim: If we need to interpolate a lot of cycles, only build the last ones * TEST RocksDB checkpoints * Added logs for ledger backups and recovery (check hash) * Moved the LedgerBatch to final_state, step 1 This does not build, I have to add the logic of apply_change to the given ledger_batch * Pass around a handle to the RocksDB DB instead of creating multiple ones * Resolve deadlock in bootstrap test * Changed AyncPool methods, bootstrap fails * Fixed bug in bootstrap test. * Resolved serializing issue with AsyncMessage in DB * fixed doc test * Big commit, see details inside desc - Created massa-db package for easier access to rocks_db in final_state - Put the AsyncPool in rocks_db - Re-designed the speculative async_pool to avoid querying rocks_db too much /!\ Tests fail! Need to investigate But now the execution tests are not in a deadlock state. * Added mini cache to async_pool, and used SetUpdateOrDelete * Update mini cache + doctest passes * Fix after merge * Boilerplate for new rocks_db in final_state * RocksDB for executed_ops and denunciations * Fmt + Clippy fix * Updated Pool message iterating * Handle the async_pool iteration differently Iteration per message > Iteration per key * Fixed after merge * Cleaned some things + docs * Recompute all caches on init of final_state * Additionnal cleaning (e.g. unused function) * Added unit test for rocksdb final_state * Add bool to reset the final_state on init * Fix test * Added back the test_take_batch for async_pool * fmt fix * Fix merge * Big commit, see description Put Final_state in a single column family. TODO: Handle bootstrap as a single step and prefixes (e.g. LedgerPrefix !) * clippy + fmt * Unified bootstrap of final_state * Put pos_state in rocks_db * Fixed prefix_iteration problem * another pass at iterator problems * Added last_slot_before_downtime in final_state * Fixed iteration error in pool cache reconstruction * Fixed deferred_credits and production_stats behaviour Still bootstrap test ignore * fmt after merge * Cleaned before working on bootstrap changes * Wrapped the rocksDB structure Needs several edits to existing code (e.g. bootstrap, write_batch) * Almost everything up to date Boostrap test still fails, have to debug with more logs! * Fixed some of the bootstrap bugs /!\ Some Deadlocks are a problem /!\ Several TODOs in the code (commented ou asserts...) * Started implementing Monotree * Spring cleaning (DBBatch type) * Fix cargo.lock after merge * Fixed deadlock in pos_state * clippy fixes * Fixed pos_state change fail in bootstrap test Also removed println! * Fixed monotree But Hash computations do not sync right on write_batch * Added checks to bootstraped db (is_db_valid()) * Fix after merge w/ testnet23 * changed monotree dep * Implemented XOR, test passes with it * Removed old todo comment * Added network restart documentation in massa-final-state crate lib.rs * tmp Signed-off-by: Litchi Pi <litchi.pi@proton.me> * fixup Signed-off-by: Litchi Pi <litchi.pi@proton.me> * assert it's not the initial bytes Signed-off-by: Litchi Pi <litchi.pi@proton.me> * Removed create_initial_cycle from new_derived_from_snapshot * Various fixes - Fix cargo.lock after massa-sc-runtime update - Don't create_initial_cycle if restarting from snapshot - Cargo fixes * fmt fix * Cleaned error handling * Cleaned unused tests, added comments * Fixed consensus mock in bootstrap test * Fixed after merge * Cleanup warnings + monotree, clear the current_hashmap in db * Fix bootstrap_test after last commit * Additional cleaning following self review Still need to review: - Bootstrap server - Final_state and massa_db - Read and writes to the DB in all sub-structures of final_state * Added a TODO comment * Updated ledger_db to use MassaDB helper functions directly * Added comments * Fix tests after merge * Finished self review, added TODO_PR comments for everything to fix before merge. * Fixed bad behaviour of bootstrap streaming of stream_batch * Added doc_comments for massa_db Also only put the cur_change_id in the db to avoid desync * Fixed bad final_state_test * Import lsmtree instead of new * removed final_state_hash * Removed slot in final_state * Removed initial_ledger_hash * recompute caches no longer duplicated. * Changed pos apply change documentation * Cleaning of un-necessary recomputes * Fix slot in bootstrap * Fixed weird behaviour for bootstrap * Try fix bootstrap * Fix in deadlock in downtime interpolation * fmt fix * Fix: only backup on a slot with thread == 0 * Fix slots logic and interpolation * fmt fix * fix tests after slot behaviour change in final_state * Removed println in final_state is_db_valid * Updated bootstrap deserializer args for state_length * Update op_exec_status on pruned executed_ops * Updated pos_state test, remove roll_counts at 0 * Fix after merge * typo fix * removed create_snapshot (unused) feature * revert node log level (from 3 to 2) * Use format!() to construct backup directory * Clean batch allocation in massa_db * Used assert_eq!() instead of panic!() in finalize * Use unreachable!() instead of todo!() in massa-hash * Added debug_assert_eq!() to check Hash size and LsmTree output_size match * Commit review suggestion Co-authored-by: Litchi Pi (Tim) <litchi.pi@proton.me> * Cargo fmt fix * Updated state_step inconsistant error messages * Updated XOR hash comment * Add mip_store in final_state (#3992) * Add mip_store in final_state * Add reset_db in MIP store * Fix bootstrap test - added the case of empty Versioning bootstrap We have to rethink of how the cursors work, see follow up issue #4000 * Rework shutdown start & shutdown end computation for final state check * Rework delete_prefix * Rework extend_from_db for MipStore --------- Co-authored-by: sydhds <sd@massa.net> Co-authored-by: Leo-Besancon <leoleo38@live.fr> * Fix after merge * Fix after merge * Improve error message if update_batches call fail * Fix lsmtree version in massa_db * Update MAX_BOOTSTRAPPED_NEW_ELEMENTS to 500 --------- Signed-off-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Litchi Pi <litchi.pi@proton.me> Co-authored-by: Sydhds <sylvain.delhomme@gmail.com> Co-authored-by: sydhds <sd@massa.net>
12 tasks
Still has lifetime issues and some trait methods missing, e.g. for bootstrap
|
Closing, I made a new branch for easier rebase in #4045 |
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this PR is to avoid having RocksDB as a dependency in all the crates that interact with the DB (massa-final-state, massa-async-pool, etc.).