Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add recover rollup command #594

Merged
merged 2 commits into from Jul 31, 2023
Merged

add recover rollup command #594

merged 2 commits into from Jul 31, 2023

Conversation

jingchen2222
Copy link
Collaborator

@jingchen2222 jingchen2222 commented Jul 30, 2023 •
edited by pr-codex bot

PR-Codex overview

This PR focuses on adding support for recovering rollup nodes.

Detailed summary:

  • Added RecoverType enum to differentiate between index and rollup recovery
  • Modified RecoverConfig struct to include recover_type field
  • Updated Recover struct to include storage field
  • Added recover_stat method to recover database state
  • Modified recover_from_arweave_tx method to update mutation statistics for rollup nodes
  • Added is_recover_rollup method to check if the recovery is for a rollup node
  • Updated add_mutation method to update mutation statistics for rollup nodes
  • Added update_mutation_stat method to update mutation statistics for rollup nodes
  • Added Rollup variant to RecoverCommand enum in DB3Command struct
  • Modified DB3Command::build_wallet method to include RecoverType parameter
  • Added new command line option for rollup recovery in DB3Command::Recover match block

The following files were skipped due to too many changes: src/node/src/command.rs

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

@jingchen2222
Copy link
Collaborator Author

jingchen2222 commented Jul 30, 2023 •
edited

Step 1. Start rollup node, apply mutations and wait rollup interval ms to execute rollup

...
2023-07-30T01:40:15.907345Z  INFO db3_storage::db_store_v2: add database with addr 0x2179792c30a81a31bebda6ba6b42d68458b5caf1 from owner 0x33c7514a2c1e65d3cf478eae0d937c40f36112de
2023-07-30T01:40:15.918031Z  INFO db3_storage::db_store_v2: add collection with db_addr 0x2179792c30a81a31bebda6ba6b42d68458b5caf1, collection_name: col, from owner 0x33c7514a2c1e65d3cf478eae0d937c40f36112de
2023-07-30T01:40:27.784318Z  INFO db3_node::storage_node_light_impl: flush database meta done
2023-07-30T01:40:27.784343Z  INFO db3_node::rollup_executor: the next rollup start block 0 and the newest block 119
2023-07-30T01:40:27.811823Z  INFO db3_node::ar_toolbox: convert 1002 into recordbatch with memory 1352424
2023-07-30T01:40:27.915572Z  INFO db3_node::ar_toolbox: Start to upload_file with balance: Currency { arweave: 0, winston: 10000000000000000 }
2023-07-30T01:40:27.916595Z  INFO db3_storage::ar_fs: fee: 6135279054
2023-07-30T01:40:28.042427Z  INFO db3_storage::meta_store_client: update rollup step with tx 59lSUEyGAerojR6lpb1UJiiqVCIiS-SjIcFtAbswXQ0, network: 1
2023-07-30T01:40:28.043096Z  INFO db3_storage::meta_store_client: start update rollup step with tx 59lSUEyGAerojR6lpb1UJiiqVCIiS-SjIcFtAbswXQ0, network: 1
2023-07-30T01:40:28.082265Z  INFO db3_storage::meta_store_client: update rollup step done! tx hash: 0xa8ec…bc21
2023-07-30T01:40:33.088546Z  INFO db3_node::rollup_executor: the process rollup done with num mutations 1002, raw data size 1352424, compress data size 93532 and processed time 5 id 59lSUEyGAerojR6lpb1UJiiqVCIiS-SjIcFtAbswXQ0 ar cost 6135279054 and evm tx 0xa8ec9c70325afedb417745e06b9ee739b0414fa78543315986a9c2ab9e20bc21 and cost 269517736168551
2023-07-30T01:40:33.088806Z  INFO db3_node::rollup_executor: last gc block range [0, 0)
2023-07-30T01:40:33.088895Z  INFO db3_node::rollup_executor: not enough round to run gc
2023-07-30T01:40:38.061537Z  INFO db3_node::storage_node_light_impl: address 0x5f8c5ab8f57698e395bbd347b438019a072b872c used nonce 0
2023-07-30T01:40:38.088497Z  INFO db3_storage::db_store_v2: add database with addr 0x99006e9644e77fe47a05025da103c207ff23abd1 from owner 0x5f8c5ab8f57698e395bbd347b438019a072b872c
2023-07-30T01:40:38.099590Z  INFO db3_storage::db_store_v2: add collection with db_addr 0x99006e9644e77fe47a05025da103c207ff23abd1, collection_name: col, from owner 0x5f8c5ab8f57698e395bbd347b438019a072b872c
2023-07-30T01:41:02.501520Z  INFO db3_node::storage_node_light_impl: address 0x3cf54deda4f17ca6724a7985dfd85953aca25931 used nonce 0
2023-07-30T01:41:02.523827Z  INFO db3_storage::db_store_v2: add database with addr 0x6de0a9239b48d36ce0ca6db44f8f4010646bf726 from owner 0x3cf54deda4f17ca6724a7985dfd85953aca25931
2023-07-30T01:41:02.532568Z  INFO db3_storage::db_store_v2: add collection with db_addr 0x6de0a9239b48d36ce0ca6db44f8f4010646bf726, collection_name: col, from owner 0x3cf54deda4f17ca6724a7985dfd85953aca25931
2023-07-30T01:41:27.790448Z  INFO db3_node::storage_node_light_impl: flush database meta done
2023-07-30T01:41:33.093424Z  INFO db3_node::rollup_executor: the next rollup start block 119 and the newest block 248
2023-07-30T01:41:33.143170Z  INFO db3_node::ar_toolbox: convert 2004 into recordbatch with memory 2704296
2023-07-30T01:41:33.309248Z  INFO db3_node::ar_toolbox: Start to upload_file with balance: Currency { arweave: 0, winston: 9999993864720946 }
2023-07-30T01:41:33.310687Z  INFO db3_storage::ar_fs: fee: 11720670965
2023-07-30T01:41:33.417093Z  INFO db3_storage::meta_store_client: update rollup step with tx yP-0W2aFMyCzdaC776pdbapHH4JYlYi0Ui4X_jJg4DQ, network: 1
2023-07-30T01:41:33.417409Z  INFO db3_storage::meta_store_client: start update rollup step with tx yP-0W2aFMyCzdaC776pdbapHH4JYlYi0Ui4X_jJg4DQ, network: 1
2023-07-30T01:41:33.456945Z  INFO db3_storage::meta_store_client: update rollup step done! tx hash: 0xe155…beff
2023-07-30T01:41:38.461651Z  INFO db3_node::rollup_executor: the process rollup done with num mutations 2004, raw data size 2704296, compress data size 178681 and processed time 5 id yP-0W2aFMyCzdaC776pdbapHH4JYlYi0Ui4X_jJg4DQ ar cost 11720670965 and evm tx 0xe155bfb6d497d14082dfc153a7bedf7564a51f0f5399a99f6abccf274891beff and cost 139874663389986
2023-07-30T01:41:38.461966Z  INFO db3_node::rollup_executor: last gc block range [0, 0)
2023-07-30T01:41:38.462079Z  INFO db3_node::rollup_executor: not enough round to run gc
2023-07-30T01:45:12.778262Z  INFO db3_node::command:

Step 2. stop rollup node, clean the mutation and doc db

if [ -e ./mutation_db ]
then
    rm -rf ./mutation_db
fi

if [ -e ./state_db ]
then
    rm -rf ./state_db
fi
if [ -e ./doc_db ]
then
    rm -rf ./doc_db
fi

Step 3. start recover rollup

❯ ../target/debug/db3 recover rollup                                                 ─╯
2023-07-30T01:43:28.492278Z  INFO db3_node::command:
██████╗ ██████╗ ██████╗
██╔══██╗██╔══██╗╚════██╗
██║  ██║██████╔╝ █████╔╝
██║  ██║██╔══██╗ ╚═══██╗
██████╔╝██████╔╝██████╔╝
╚═════╝ ╚═════╝ ╚═════╝
@db3.network🚀🚀🚀
2023-07-30T01:43:28.492375Z  INFO db3_storage::state_store: open state store with path ./state_db
2023-07-30T01:43:28.494898Z  INFO db3_node::command: Arweave address gXJVsUCAmDqv9XeZui0MB2EdGPQEhN86QEnKY0_7vPc
2023-07-30T01:43:28.496224Z  INFO db3_node::command: Evm address 0x0xf39f…2266
2023-07-30T01:43:28.496309Z  INFO db3_storage::db_store_v2: open db store with path ./rollup_meta_db
2023-07-30T01:43:28.532715Z  INFO db3_storage::mutation_store: open mutation store with path ./mutation_db
2023-07-30T01:43:28.606957Z  INFO db3_node::recover: evm address 0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266
2023-07-30T01:43:28.608409Z  INFO db3_storage::ar_fs: recover ar key store from ./keys
2023-07-30T01:43:28.609458Z  INFO db3_storage::ar_fs: new ar filestore with url http://127.0.0.1:1984 and addr gXJVsUCAmDqv9XeZui0MB2EdGPQEhN86QEnKY0_7vPc
2023-07-30T01:43:28.609832Z  INFO db3_node::command: start recovering index node
2023-07-30T01:43:28.610197Z  INFO db3_node::recover: start recover from arweave
2023-07-30T01:43:28.610220Z  INFO db3_node::recover: recover the block state done, last block is 0
tx yP-0W2aFMyCzdaC776pdbapHH4JYlYi0Ui4X_jJg4DQ
tx 59lSUEyGAerojR6lpb1UJiiqVCIiS-SjIcFtAbswXQ0
2023-07-30T01:43:28.688696Z  INFO db3_storage::db_store_v2: add database with addr 0x2179792c30a81a31bebda6ba6b42d68458b5caf1 from owner 0x33c7514a2c1e65d3cf478eae0d937c40f36112de
2023-07-30T01:43:28.692491Z  INFO db3_storage::db_store_v2: add collection with db_addr 0x2179792c30a81a31bebda6ba6b42d68458b5caf1, collection_name: col, from owner 0x33c7514a2c1e65d3cf478eae0d937c40f36112de
2023-07-30T01:43:32.112968Z  INFO db3_storage::db_store_v2: add database with addr 0x99006e9644e77fe47a05025da103c207ff23abd1 from owner 0x5f8c5ab8f57698e395bbd347b438019a072b872c
2023-07-30T01:43:32.116761Z  INFO db3_storage::db_store_v2: add collection with db_addr 0x99006e9644e77fe47a05025da103c207ff23abd1, collection_name: col, from owner 0x5f8c5ab8f57698e395bbd347b438019a072b872c
2023-07-30T01:43:35.375824Z  INFO db3_storage::db_store_v2: add database with addr 0x6de0a9239b48d36ce0ca6db44f8f4010646bf726 from owner 0x3cf54deda4f17ca6724a7985dfd85953aca25931
2023-07-30T01:43:35.379325Z  INFO db3_storage::db_store_v2: add collection with db_addr 0x6de0a9239b48d36ce0ca6db44f8f4010646bf726, collection_name: col, from owner 0x3cf54deda4f17ca6724a7985dfd85953aca25931
tx yP-0W2aFMyCzdaC776pdbapHH4JYlYi0Ui4X_jJg4DQ
2023-07-30T01:43:38.672649Z  INFO db3_node::recover: recover from arweave done!

Step 4. start rollup node again

../target/debug/db3 rollup --block-interval=500 --admin-addr=0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 >rollup.log 2>&1 &

Step 5. test document database create/update/delete/query

@jingchen2222
Copy link
Collaborator Author

Future work

rollup integration test script

@jingchen2222 jingchen2222 changed the title add recover rollup command (WIP) add recover rollup command Jul 30, 2023
@codecov-commenter
Copy link

codecov-commenter commented Jul 30, 2023 •
edited

Codecov Report

Merging #594 (3ca4850) into main (dddbf4c) will not change coverage.
The diff coverage is n/a.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@           Coverage Diff           @@
##             main     #594   +/-   ##
=======================================
  Coverage   61.70%   61.70%           
=======================================
  Files           9        9           
  Lines         470      470           
  Branches       69       69           
=======================================
  Hits          290      290           
  Misses        180      180
Flag Coverage Δ
rust 61.70% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@jingchen2222 jingchen2222 requested a review from imotai July 30, 2023 05:35
imotai
imotai approved these changes Jul 31, 2023
View reviewed changes
Copy link
Contributor

@imotai imotai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imotai imotai merged commit 518b627 into main Jul 31, 2023
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@imotai imotai imotai approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@jingchen2222 @codecov-commenter @imotai