Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pacific: mon: fix mds metadata lost in one case. #54318

Merged
merged 1 commit into from Nov 22, 2023

Conversation

batrick
Copy link
Member

@batrick batrick commented Nov 3, 2023

backport tracker: https://tracker.ceph.com/issues/63414


backport of #53883
parent tracker: https://tracker.ceph.com/issues/63166

this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/main/src/script/ceph-backport.sh

In most cases, peon's pending_metadata is inconsistent with mon's db.
When a peon turns into leader, and at the same time a active mds stops,
the new leader may flush wrong mds metadata into db. So we meed to
update mds metadata from db at every fsmap change.

This phenomenon can be reproduce like this:
A Cluster with 3 mon and 3 mds (one active, other two standby), 6 osd.
step 1. stop two standby mds;
step 2. restart all mon; (make pending_medata consistent with db)
step 3. start other two mds
step 4. stop leader mon
step 5. run "ceph mds metadata" command to check mds metadata
step 6. stop active mds
step 7. run "ceph mds metadata" command to check mds metadata again

In step 7, we would find mds metadata lost.

Fixes: https://tracker.ceph.com/issues/63166
Signed-off-by: shimin <shimin@kuaishou.com>
(cherry picked from commit 56cbf3f)
@batrick batrick requested a review from a team as a code owner November 3, 2023 00:28
@batrick batrick added this to the pacific milestone Nov 3, 2023
@batrick batrick added the core label Nov 3, 2023
@github-actions github-actions bot added cephfs Ceph File System mon labels Nov 3, 2023
@batrick batrick modified the milestones: pacific, v16.2.15 Nov 8, 2023
Copy link
Contributor

@rishabh-d-dave rishabh-d-dave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuriw yuriw merged commit 91e6f18 into ceph:pacific Nov 22, 2023
8 checks passed
@batrick batrick deleted the wip-63414-pacific branch December 18, 2023 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants