Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Import export trie log #6363

Merged
merged 51 commits into from Jan 22, 2024
Merged

Import export trie log #6363

merged 51 commits into from Jan 22, 2024

Commits on Dec 15, 2023

  1. Add x-trie-log subcommand for one-off backlog prune 

    Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @siladu @gfukushima
    siladu authored and gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    16c0a49 View commit details
    Browse the repository at this point in the history

  2. long -> int 

    Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @siladu @gfukushima
    siladu authored and gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    7dd4928 View commit details
    Browse the repository at this point in the history

  3. Removed banned method 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    bf2b098 View commit details
    Browse the repository at this point in the history

  4. Preload process stream in parallel 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    e67ae51 View commit details
    Browse the repository at this point in the history

  5. Drop unwanted trielogs and keep reatain layers only 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    9b4e0c9 View commit details
    Browse the repository at this point in the history

  6. Add output to user and cleanup refactor 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    0b9fe83 View commit details
    Browse the repository at this point in the history

  7. small tweak to display cf that had reference dropped by RocksDbSegmen… 

    …tIdentifier.reset()

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    426848e View commit details
    Browse the repository at this point in the history

  8. spotless 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    7401b59 View commit details
    Browse the repository at this point in the history

  9. Fix classes that changed package 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    1b7fb72 View commit details
    Browse the repository at this point in the history

  10. spotless 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    11e6b05 View commit details
    Browse the repository at this point in the history

  11. Code review 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    f2d01e2 View commit details
    Browse the repository at this point in the history

  12. Only clear DB when we have the exact amount of trie logs we want in m… 

    …emory

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 15, 2023
    Configuration menu
    Copy the full SHA
    04f1aaa View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2023

  1. Trielogs stream to and from file to avoid possibly OOM 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    2f01c5a View commit details
    Browse the repository at this point in the history

  2. Process trie logs in chunks to avoid OOM 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    56e4c8e View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2023

  1. save and read in batches to handle edge cases 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 19, 2023
    Configuration menu
    Copy the full SHA
    78561b0 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2023

  1. save and read files to/from database dir 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    42c72cf View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into x-trie-log-subcommand-2

    @gfukushima
    gfukushima committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    9961fc2 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2023

  1. add unit tests and PR review fixes 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    9389540 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into x-trie-log-subcommand-2

    @gfukushima
    gfukushima committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    e3d4fbc View commit details
    Browse the repository at this point in the history

  3. spdx 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    c7144fe View commit details
    Browse the repository at this point in the history

  4. Fix unit tests directory creation and deletion 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Dec 21, 2023
    Configuration menu
    Copy the full SHA
    20b0ba5 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. rename Xbonsai-trie-log-pruning-enabled to Xbonsai-limit-trie-logs-en… 

    …abled

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    586ab25 View commit details
    Browse the repository at this point in the history

  2. Import and export trie log subcommands 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    67e6f3d View commit details
    Browse the repository at this point in the history

  3. PR review 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    b9640e5 View commit details
    Browse the repository at this point in the history

  4. spotless 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    3bc1878 View commit details
    Browse the repository at this point in the history

Commits on Jan 7, 2024

  1. fix path resolver and added unit tests 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 7, 2024
    Configuration menu
    Copy the full SHA
    d47ddf5 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2024

  1. Merge branch 'main' into import-export-trie-log

    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    999edb6 View commit details
    Browse the repository at this point in the history

  2. fix unit test 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    1699fe4 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/import-export-trie-log' into imp… 

    …ort-export-trie-log
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    e679cb3 View commit details
    Browse the repository at this point in the history

  4. fix unit test 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    5d3b4f2 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into import-export-trie-log

    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    f839b75 View commit details
    Browse the repository at this point in the history

  6. Add import and export to list of subcommands under --x-trie-log 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    0caa4cf View commit details
    Browse the repository at this point in the history

  7. Merge remote-tracking branch 'origin/import-export-trie-log' into imp… 

    …ort-export-trie-log
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    2d5d31d View commit details
    Browse the repository at this point in the history

  8. Remove static from setup method 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    37df23e View commit details
    Browse the repository at this point in the history

  9. change option name and fix descriptions 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    5ce1800 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'main' into import-export-trie-log 

    # Conflicts:
#	besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java
#	besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogSubCommand.java
#	besu/src/test/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelperTest.java
    @gfukushima
    gfukushima committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    98423dc View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2024

  1. Fix subcommands descriptions 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    cf3a5e6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into import-export-trie-log

    @gfukushima
    gfukushima committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    c759bba View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. Remove old flag and move commands const into Unstable 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    5fb9413 View commit details
    Browse the repository at this point in the history

  2. Allow list of block hashes to passed as well as a file to be generate… 

    …d or imported

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    087c54b View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/import-export-trie-log' into imp… 

    …ort-export-trie-log
    @gfukushima
    gfukushima committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    4b033a3 View commit details
    Browse the repository at this point in the history

  4. Allow list of block hashes to passed as well as a file to be generate… 

    …d or imported

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    3a89ac3 View commit details
    Browse the repository at this point in the history

  5. Fix broken test when replaced the old option 

    Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
    @gfukushima
    gfukushima committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    9be7d13 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. import and export using rlp 

    Signed-off-by: Jason Frame <jason.frame@consensys.net>
    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    75d1c3b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into import-export-trie-log

    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    5eb4cda View commit details
    Browse the repository at this point in the history

  3. tests for exporting and importing multiple trielogs 

    Signed-off-by: Jason Frame <jason.frame@consensys.net>
    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    5425075 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'import-export-trie-log' of github.com:gfukushima/besu i… 

    …nto import-export-trie-log
    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    2de0b19 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into import-export-trie-log

    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    248a776 View commit details
    Browse the repository at this point in the history

  6. fix build 

    Signed-off-by: Jason Frame <jason.frame@consensys.net>
    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    55d653e View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into import-export-trie-log

    @jframe
    jframe committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    c9d38f0 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. Merge branch 'main' into import-export-trie-log

    @jframe
    jframe committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    b9d7620 View commit details
    Browse the repository at this point in the history