Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crimson/os/seastore/rbm: make rbm support multiple shards #51770

Merged
merged 2 commits into from
Jul 7, 2023
Merged

crimson/os/seastore/rbm: make rbm support multiple shards #51770

merged 2 commits into from
Jul 7, 2023

Conversation

myoungwon
Copy link
Member

Signed-off-by: Myoungwon Oh myoungwon.oh@samsung.com

Contribution Guidelines

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

@myoungwon myoungwon added the DNM label May 26, 2023
@myoungwon myoungwon requested a review from a team as a code owner May 26, 2023 08:40
@myoungwon myoungwon removed the request for review from a team May 26, 2023 08:41
@myoungwon myoungwon force-pushed the wip-multicore-rbm branch 4 times, most recently from 7bec68c to a8549f3 Compare June 17, 2023 08:33
@github-actions github-actions bot added the tests label Jun 17, 2023
@myoungwon myoungwon removed the DNM label Jun 19, 2023
@myoungwon
Copy link
Member Author

myoungwon commented Jun 19, 2023 •
edited

@cyx1231st @liu-chunmei Can you take a look? The goal of this PR is to align RBM with recent multi shard updates.

@liu-chunmei liu-chunmei changed the title [WIP,DNM]: crimson/os/seastore/rbm: make rbm support multiple shrads [WIP,DNM]: crimson/os/seastore/rbm: make rbm support multiple shards Jun 20, 2023
liu-chunmei
liu-chunmei reviewed Jun 21, 2023
View reviewed changes
Copy link
Contributor

@liu-chunmei liu-chunmei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change commit type error: multiple shards

@myoungwon myoungwon force-pushed the wip-multicore-rbm branch 3 times, most recently from ae44dea to 9d367e5 Compare June 22, 2023 12:57
liu-chunmei
liu-chunmei reviewed Jun 26, 2023
View reviewed changes
Copy link
Contributor

@liu-chunmei liu-chunmei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except for one error need be fixed.

@myoungwon myoungwon changed the title [WIP,DNM]: crimson/os/seastore/rbm: make rbm support multiple shards [WIP]: crimson/os/seastore/rbm: make rbm support multiple shards Jun 27, 2023
@myoungwon
Copy link
Member Author

@liu-chunmei Done.

@myoungwon
Copy link
Member Author

@cyx1231st Please take a look when you are available.

@myoungwon myoungwon requested a review from cyx1231st July 4, 2023 11:24
cyx1231st
cyx1231st reviewed Jul 5, 2023
View reviewed changes
src/crimson/os/seastore/random_block_manager/rbm_device.h Outdated
return RBM_SUPERBLOCK_SIZE;
}

rbm_abs_addr get_journal_start() {
return shard_info.start_offset + get_journal_start_off();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why each shard's journal needs to reserve a RBM_SUPERBLOCK_SIZE?

IIUC there is only one superblock for all the shards, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. There is a superblock in RBM. Please see the below.

I'm thinking of a backup super block like other filesystems. This might help to check inconsistency and recovery. So, this part is reserved for now. Also, this can be used as the extension of the existing primary's superblock if the superblock size exceeds the block size. If we want to save the amount of space (super block size x #shard number) by removing a super block in other shards, this is also fine with me for now. But, we need to change the disk layout entirely if those blocks are needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering we still need to adjust the layout in the future.

We need to finally implement dynamic partition across shards, which means the data size for each shard may become dynamic in the future.

In that case, probably starting from | superblock(s) | circular journals | data | and static partition the data for each shard can be more generic ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree on that we need a further consideration for dynamic partition. I think there are two options as below. But, it seems that I need a study on the dynamic partition to find out pros and cons---there are already many studies, such as [1], that did a research on sharded partitions.

Option 1 (initial version, but working): | super 1 | journal 1 | data 1 | - | super 2 (reserved) | journal 2 | data 2 |
Option 2: | superblocks(s) | journal(s) | data |

Before doing that, how about merging this PR as is. I'll continue doing this work to find what the better solution is.

[1] Scaling a file system to many cores using an operation log. SOSP 2017

src/crimson/os/seastore/random_block_manager.h Outdated Show resolved Hide resolved
src/crimson/os/seastore/random_block_manager/rbm_device.h Outdated Show resolved Hide resolved
src/crimson/os/seastore/random_block_manager/rbm_device.cc Outdated Show resolved Hide resolved
src/crimson/os/seastore/random_block_manager/rbm_device.h Outdated Show resolved Hide resolved
src/crimson/os/seastore/random_block_manager/rbm_device.cc Outdated Show resolved Hide resolved
src/crimson/os/seastore/random_block_manager/nvme_block_device.cc Outdated Show resolved Hide resolved
cyx1231st
cyx1231st reviewed Jul 6, 2023
View reviewed changes
Copy link
Member

@cyx1231st cyx1231st left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except for comments

src/crimson/os/seastore/random_block_manager/rbm_device.h Outdated Show resolved Hide resolved
src/crimson/os/seastore/random_block_manager/rbm_device.h Outdated Show resolved Hide resolved
@myoungwon
crimson/os/seastore/rbm: make rbm support multiple shards
1d6beef 
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
@myoungwon
src/test/crimson/seastore/nvmedevice: update a test case based on rbm…
7a15d48 
… multi shard change

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>
@myoungwon
Copy link
Member Author

@cyx1231st Addressed.

Besides, please note the the following work items I have planed to working on.

  1. Data partial-overwrite (crimson/os/seastore: introduce delta-based overwrite #50419)
  2. Generalize replay logic, including scan_valid_records, and fixing invalid replay
  3. Deferred journal head update---current rbm update the head every time transaction is done
  4. Dynamic partition

@myoungwon myoungwon changed the title [WIP]: crimson/os/seastore/rbm: make rbm support multiple shards crimson/os/seastore/rbm: make rbm support multiple shards Jul 6, 2023
@myoungwon
Copy link
Member Author

jenkins test api

1 similar comment
@myoungwon
Copy link
Member Author

jenkins test api

@cyx1231st cyx1231st merged commit f8ad0e2 into ceph:main Jul 7, 2023
11 of 12 checks passed
@Matan-B Matan-B mentioned this pull request Oct 11, 2023
Merged
@myoungwon myoungwon mentioned this pull request Jan 8, 2024
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cyx1231st cyx1231st cyx1231st approved these changes

@liu-chunmei liu-chunmei liu-chunmei approved these changes

Assignees
No one assigned
Labels
crimson tests
Projects
None yet
Milestone
No milestone
3 participants
@myoungwon @cyx1231st @liu-chunmei