Fix massa db change history handling #4533

Leo-Besancon · 2023-11-13T12:18:35Z

document all added functions
try in sandbox /simulation/labnet
- if part of node-launch, checked using the resync_check flag
unit tests on the added/changed features
- make tests compile
- make tests pass
add logs allowing easy debugging in case the changes caused problems
if the API has changed, update the API specification

Tested that:

Labnet with a 1Go ledger successfully bootstrap with final_history_length at 100
Labnet with a 1Go ledger fails bootstrap with error message (instead of desync) with final_history_length at 8
Without this PR, Labnet with a 1Go ledger "successfully bootstrap" and desync much more often with final_history_length at 8

To check: if < or <= to allow changes

Leo-Besancon · 2023-11-13T12:39:16Z

Note: before merging, we should calibrate the max_history_length value in the config.toml. 100 slots of history should be good with a timeout of 30 seconds for 1 bootstrap message (60 slots). We take some headroom in case more slots are finalized (unsteady network).

massa-db-worker/src/massa_db.rs

Dzmitryy1812 · 2023-11-14T05:30:51Z

document all added functions

try in sandbox /simulation/labnet

if part of node-launch, checked using the resync_check flag

unit tests on the added/changed features

make tests compile

make tests pass

add logs allowing easy debugging in case the changes caused problems

if the API has changed, update the API specification

Tested that:

Labnet with a 1Go ledger successfully bootstrap with final_history_length at 100

Labnet with a 1Go ledger fails bootstrap with error message (instead of desync) with final_history_length at 8

Without this PR, Labnet with a 1Go ledger "successfully bootstrap" and desync much more often with final_history_length at 8

sydhds

It's easier to read now thanks to the comment. Nice job!

* Buildnet WIP * Test dir + base case test * Mock Final State/Improve tests execution (#4512) * Use a controller for final state to be able to mock it * Fix all formatting issues. * Use more mocks in execution tests * Fix test nested call usage * improve some tests execution * Fix tests async message and compact the test of executed * Fix more execution tests * Fix more tests. * Update more tests * Finish fixing all executions tests. * Improve naming * add more comment * Fix comment * Minor addition to test_standalone_selection * Add test_invalid_roll_distribution * Minor test_invalid_roll_distribution change * fix clippy * Fix double cursor advance (#4531) Co-authored-by: sydhds <sydhds@gmail.com> * Fix massa db change history handling (#4533) * Initial fix and test To check: if < or <= to allow changes * fmt + clippy * fix comments * Remove debug prints * Cache miss error rename + u8::MAX * Added comments about Included and CacheMiss * Comment + Add StreamingStep::Ongoing to test * Fix previous tests * Bump self_cell from 1.0.1 to 1.0.2 (#4538) Bumps [self_cell](https://github.com/Voultapher/self_cell) from 1.0.1 to 1.0.2. - [Release notes](https://github.com/Voultapher/self_cell/releases) - [Commits](Voultapher/self_cell@v1.0.1...v1.0.2) --- updated-dependencies: - dependency-name: self_cell dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add test code coverage + checksums (#4470) * ci: add test code coverage * ci: add checksums to artifacts * fix: cargo clean * fix: trigger ci * ci: last chance * ci: skip macOS ... * ci: silver bullet cache * ci: add codecov * ci: clean coverage utility * ci: reset to main version * Keep only event count in ExecutionContextSnapshot Signed-off-by: Jean-François <jfm@laposte.net> * updates * update packages * make test_standalone_selection generic (#4539) * Create universes for bootstrap (#4528) * Use a controller for final state to be able to mock it * Fix all formatting issues. * Use more mocks in execution tests * Fix test nested call usage * improve some tests execution * Fix tests async message and compact the test of executed * Fix more execution tests * Fix more tests. * Update more tests * Finish fixing all executions tests. * Improve naming * Create universes for bootstrap * add more comment * Add bootstrap whitelist test correctly * Update to make a correct bootstrap flow * Change big bootstrap test and cleanup some unused variables * Version of bootstrap tests with mock well organized * Update bootstrap check database * fix all tests bootstrap * Fix tests massa db * Update better tests * modify condition * Fix readonly (#4543) * fix readonly * update proto dep * Fix bootstrap deser (#4537) * Add length_value check in deserializer * Update the logic TODO: - Fix UT fails - Remove debug logs - Refactor the serde of KV in a separate parser * fmt * Fix ser * Remove prints * Add duplicate key test in deserializer * Add faulty message tests * Fix & prints * Update to take serialized len into account for streamBatch fill * Rename + add comments * Use Enum for faulty case instead of int * Fix clippy * Adjust node APIs settings (#4536) * refactor: adjust APIs settings * refactor: revert bind public address * Improve ReadOnly estimations (#4545) * Fix read only call & execution estimations * Update comment * Set async pool size to 1000 and add async msg fixed costs (#4546) Signed-off-by: Jean-François <jfm@laposte.net> * Update genesis --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Jean-François <jfm@laposte.net> Co-authored-by: Thomas Plisson <thomas.plisson@epitech.eu> Co-authored-by: AurelienFT <32803821+AurelienFT@users.noreply.github.com> Co-authored-by: AurelienFT <aurelien.foucault@epitech.eu> Co-authored-by: Sydhds <sylvain.delhomme@gmail.com> Co-authored-by: sydhds <sydhds@gmail.com> Co-authored-by: Eitu33 <89928840+Eitu33@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Moncef AOUDIA <ma@massa.net> Co-authored-by: Jean-François <jfm@laposte.net> Co-authored-by: Damir Vodenicarevic <damipator@gmail.com> Co-authored-by: Moncef AOUDIA <mf.aoudia@gmail.com>

Leo-Besancon added 2 commits November 13, 2023 13:02

Initial fix and test

d285a63

To check: if < or <= to allow changes

fmt + clippy

fc4c342

Leo-Besancon requested review from sydhds and AurelienFT November 13, 2023 12:24

Leo-Besancon marked this pull request as ready for review November 13, 2023 12:24

fix comments

256a5c6

Leo-Besancon mentioned this pull request Nov 13, 2023

Node security meta-issue #4415

Open

45 tasks

Remove debug prints

e450019

sydhds reviewed Nov 13, 2023

View reviewed changes

Leo-Besancon added 4 commits November 14, 2023 08:21

Cache miss error rename + u8::MAX

179062e

Added comments about Included and CacheMiss

f1601a5

Comment + Add StreamingStep::Ongoing to test

dade0ab

Fix previous tests

e0a6613

sydhds approved these changes Nov 14, 2023

View reviewed changes

AurelienFT approved these changes Nov 14, 2023

View reviewed changes

Leo-Besancon merged commit b175e72 into buildnet_wip Nov 14, 2023
10 of 11 checks passed

Leo-Besancon deleted the fix_massa_db_change_history branch November 14, 2023 14:13

Leo-Besancon restored the fix_massa_db_change_history branch November 14, 2023 14:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix massa db change history handling #4533

Fix massa db change history handling #4533

Leo-Besancon commented Nov 13, 2023 •

edited

Leo-Besancon commented Nov 13, 2023 •

edited

Dzmitryy1812 commented Nov 14, 2023

sydhds left a comment

Fix massa db change history handling #4533

Fix massa db change history handling #4533

Conversation

Leo-Besancon commented Nov 13, 2023 • edited

Leo-Besancon commented Nov 13, 2023 • edited

Dzmitryy1812 commented Nov 14, 2023

sydhds left a comment

Choose a reason for hiding this comment

Leo-Besancon commented Nov 13, 2023 •

edited

Leo-Besancon commented Nov 13, 2023 •

edited