Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG 2153695: osd: Fix osd-prepare-job for encrypted cluster #440

Merged
merged 2 commits into from
Dec 16, 2022

Conversation

Rakshith-R
Copy link
Member

Description of your changes:

osd: re-open encrypted disk during osd-prepare-job if closed

This commit implements this corner case during osd-prepare job.

The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.

Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.

When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.

Signed-off-by: Rakshith R rar@redhat.com
(cherry picked from commit bde286e)

osd: remove stale dm device during osd-prepare-job

During osd-prepare-job,

The encrypted device is found to be closed in some cases when
the OSD deployment has been removed manually accompanied
by any of following cases:

  • node reboot
  • csi managed PVC being unmounted etc

while re-opening the block to re-hydrate the OSDInfo,
the dm device <pvc-name>-block-dmcrypt clashes with
the one used by OSD pod which is stale by now.
This commit adds cmd to remove this stale dm device.
Error with output "No such device" is ignored.

Signed-off-by: Rakshith R rar@redhat.com
(cherry picked from commit 24fa9ed)

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide).
  • Skip Tests for Docs: If this is only a documentation change, add the label skip-ci on the PR.
  • Reviewed the developer guide on Submitting a Pull Request
  • Pending release notes updated with breaking and/or notable changes for the next minor release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

This commit implements this corner case during osd-prepare job.
```
The encrypted block is not opened, this is an extreme corner case
The OSD deployment has been removed manually AND the node rebooted
So we need to re-open the block to re-hydrate the OSDInfo.

Handling this case would mean, writing the encryption key on a
temporary file, then call luksOpen to open the encrypted block and
then call ceph-volume to list against the opened encrypted block.
We don't implement this, yet and return an error.
```
When underlying PVC for osd are CSI provisioned, the encrypted device
is closed when PVC is unmounted due to osd pod being deleted.
Therefore, this may occur more frequently and needs to be handled.
This commit implements the fix for the same.

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit bde286e)
@openshift-ci openshift-ci bot added bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Dec 15, 2022
@openshift-ci
Copy link

openshift-ci bot commented Dec 15, 2022

@Rakshith-R: This pull request references Bugzilla bug 2153695, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

2 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @nehaberry

In response to this:

BUG 2153695: osd: Fix osd-prepare-job for encrypted cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link

openshift-ci bot commented Dec 15, 2022

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: nehaberry.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@Rakshith-R: This pull request references Bugzilla bug 2153695, which is valid. The bug has been updated to refer to the pull request using the external bug tracker.

2 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @nehaberry

In response to this:

BUG 2153695: osd: Fix osd-prepare-job for encrypted cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

During osd-prepare-job,

The encrypted device is found to be closed in some cases when
the OSD deployment has been removed manually accompanied
by any of following cases:
- node reboot
- csi managed PVC being unmounted etc

while re-opening the block to re-hydrate the OSDInfo,
the dm device `<pvc-name>-block-dmcrypt` clashes with
the one used by OSD pod which is stale by now.
This commit adds cmd to remove this stale dm device.
Error with output "No such device" is ignored.

Signed-off-by: Rakshith R <rar@redhat.com>
(cherry picked from commit 24fa9ed)
@Rakshith-R
Copy link
Member Author

/bugzilla refresh

/cc @travisn @BlaineEXE @red-hat-storage/rook-admins

cc @sunilheggodu

@openshift-ci openshift-ci bot requested review from BlaineEXE, travisn and a team December 16, 2022 09:28
@openshift-ci
Copy link

openshift-ci bot commented Dec 16, 2022

@Rakshith-R: This pull request references Bugzilla bug 2153695, which is valid.

2 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @nehaberry

In response to this:

/bugzilla refresh

/cc @travisn @BlaineEXE @red-hat-storage/rook-admins

cc @sunilheggodu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link

openshift-ci bot commented Dec 16, 2022

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: nehaberry.

Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@Rakshith-R: This pull request references Bugzilla bug 2153695, which is valid.

2 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @nehaberry

In response to this:

/bugzilla refresh

/cc @travisn @BlaineEXE @red-hat-storage/rook-admins

cc @sunilheggodu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@travisn
Copy link

travisn commented Dec 16, 2022

/approve
/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 16, 2022
@openshift-ci
Copy link

openshift-ci bot commented Dec 16, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Rakshith-R, travisn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 16, 2022
@travisn travisn merged commit e9e0b59 into red-hat-storage:release-4.10 Dec 16, 2022
@openshift-ci
Copy link

openshift-ci bot commented Dec 16, 2022

@Rakshith-R: All pull requests linked via external trackers have merged:

Bugzilla bug 2153695 has been moved to the MODIFIED state.

In response to this:

BUG 2153695: osd: Fix osd-prepare-job for encrypted cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-unspecified Referenced Bugzilla bug's severity is unspecified for the PR. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
2 participants