Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor!: read-write separation and merge rocksdb instance #1338

Merged
merged 4 commits into from
Aug 22, 2023
Merged

Conversation

KaoImin
Copy link
Contributor

@KaoImin KaoImin commented Aug 18, 2023

What this PR does / why we need it?

This PR contains two part:

  1. Make the storage and executor backend read-write separation.
  2. Merge the four RocksDB instance into one

The part 1 is the precondition of part 2.

Read-write separation

  1. Divide Storage trait into ReadOnlyStorage and Storage. The ReadOnlyStorage contains all the get* methods and the Storage trait inherit the ReadOnlyStorage trait.
  2. Divide ExecutorAdaptor trait into ReadOnlyExecutorAdaptor and ExecutorAdaptor. The difference is that the ExecutorAdaptor requires the ApplyBackend trait.
  3. The consensus module and executor module use Storage and API module and mempool module use ReadOnlyStorage
  4. The consensus module use ExecutorAdaptor and API module, interoperation module and mempool module use ReadOnlyExecutorAdaptor

Merge RocksDB instance

This PR abstracts the adapter module in the core/storage crate into a new standalone crate named core/db. The new crate contains memory DB and RocksDB wrapper functionalities. Both core/storage and core/executor now directly depend on core/db, which is a change from the previous setup. The former state_db, metadata_db, and header_cell_db are now represented as c8, c9, and c10 column families within the main RocksDB instance. This restructuring enhances modularity and efficiency in managing database components.

What is the impact of this PR?

⚠️This is a breaking change and have to clear the data.

The previous data folder layout:

├── axon.ver
├── consensus_wal
├── peer_store
│   ├── addr_manager.db
│   ├── ban_list.db
│   └── tmp
├── rocksdb
│   ├── block_data
│   ├── state_data
│   └── system_contract
│       ├── header_cell
│       └── metadata
└── txs_wal

The current data folder layout:

├── axon.ver
├── consensus_wal
├── peer_store
│   ├── addr_manager.db
│   ├── ban_list.db
│   └── tmp
├── rocksdb
└── txs_wal

Special notes for your reviewer

Because of this PR change so many files, review it commit by commit.

Which issue(s) this PR fixes:

CI Settings

CI Usage

Tip: Check the CI you want to run below, and then comment /run-ci.

CI Switch

  • Cargo Clippy
  • Coverage Test
  • E2E Tests
  • Code Format
  • Unit Tests
  • Web3 Compatible Tests
  • OCT 1-5 And 12-15
  • OCT 6-10
  • OCT 11
  • OCT 16-19
  • v3 Core Tests

@KaoImin KaoImin requested a review from a team as a code owner August 18, 2023 14:58
@KaoImin KaoImin marked this pull request as draft August 18, 2023 14:58
@github-actions github-actions bot added the chore label Aug 18, 2023
@KaoImin KaoImin changed the title chore: upgrade some crates refactor: read-write separation and merge rocksdb instance Aug 18, 2023
@KaoImin KaoImin requested review from blckngm and driftluo and removed request for ashuralyk August 18, 2023 15:30
@KaoImin KaoImin changed the title refactor: read-write separation and merge rocksdb instance refactor!: read-write separation and merge rocksdb instance Aug 18, 2023
@KaoImin KaoImin marked this pull request as ready for review August 18, 2023 15:56
@KaoImin KaoImin force-pushed the refactor-db branch 3 times, most recently from 084580c to abbb23d Compare August 20, 2023 16:20
@KaoImin

This comment was marked as off-topic.

@axon-bot

This comment was marked as outdated.

@KaoImin KaoImin force-pushed the refactor-db branch 2 times, most recently from 6d8fd32 to c6e57ce Compare August 21, 2023 10:41
@KaoImin

This comment was marked as off-topic.

@axon-bot

This comment was marked as outdated.

@KaoImin

This comment was marked as off-topic.

@axon-bot

This comment was marked as outdated.

driftluo
driftluo previously approved these changes Aug 22, 2023
@KaoImin

This comment was marked as off-topic.

@axon-bot
Copy link

axon-bot bot commented Aug 22, 2023

CI tests run on commit:

CI test list:

  • Code Format
  • Cargo Clippy
  • E2E Tests
  • OCT 1-5 And 12-15
  • OCT 6-10
  • OCT 11
  • OCT 16-19
  • Unit Tests
  • v3 Core Tests
  • Web3 Compatible Tests

Please check ci test results later.

@KaoImin KaoImin added this pull request to the merge queue Aug 22, 2023
Merged via the queue into main with commit 39868b9 Aug 22, 2023
29 checks passed
@KaoImin KaoImin deleted the refactor-db branch August 22, 2023 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants