Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach #25733

jsafrane · 2020-12-02T17:15:37Z

FibreChannel volume plugin misses one important step when removing a device: "multipath -f". It flushes all multipath buffers to its individual paths. Without it, a filesystem on the device may get corrupted.

~~/hold~~

The PR is not upstream yet, see Fix FibreChannel volume plugin corrupting filesystem on detach kubernetes/kubernetes#97013
- We may need to merge it even before though, upstream is frozen...
~~I need to test it.~~ Testsed with iSCSI instead of FC.

openshift-ci-robot · 2020-12-02T17:15:41Z

@jsafrane: This pull request references Bugzilla bug 1903524, which is invalid:

expected the bug to target the "4.7.0" release, but it targets "3.11.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-12-02T17:16:22Z

@jsafrane: This pull request references Bugzilla bug 1903524, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (3.11.z) matches configured target release for branch (3.11.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

… on detach FibreChannel volume plugin misses one important step when removing a device: "multipath -f". It flushes all multipath buffers to its individual paths. Without it, a filesystem on the device may get corrupted.

openshift-ci-robot · 2020-12-02T21:00:26Z

@jsafrane: This pull request references Bugzilla bug 1903524, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (3.11.z) matches configured target release for branch (3.11.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

If a FibreChannel device is used as a block volume, we should flush its I/O before deleting its device. It is not strictly necessary when it's used as a filesystem (mount), but it won't hurt either.

tsmetana

/lgtm

jsafrane · 2020-12-03T12:54:51Z

/retest

jsafrane · 2020-12-03T16:30:08Z

/hold cancel
upstream is still frozen.

openshift-ci-robot · 2020-12-03T16:30:55Z

@jsafrane: This pull request references Bugzilla bug 1903524, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (3.11.z) matches configured target release for branch (3.11.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jsafrane · 2020-12-07T09:32:56Z

/retest

jsafrane · 2020-12-07T12:29:56Z

Test failures:

cmd: Docker registry lookup failed: toomanyrequests: You have reached your pull rate limit. (of a ruby image).
extended_clusterup: oc get is -n openshift ruby ... expecting any result and text 'latest'; re-trying every 1s until completion or 600.000s: the command timed out. I believe it's docker pull limit too.

extended_conformance_install: most probably it's this failure:

                        "message": "runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized", 
                        "reason": "KubeletNotReady", 
                        "status": "False", 
                        "type": "Ready"
...
1. Hosts:    localhost
 Play:     Restart nodes
 Task:     Wait for node to be ready
 Message:  Failed without returning a message.
 ...

This looks like failing since 2020-11-12.

e2e-gcp: Failed: Failed to pull image "gcr.io/k8s-authenticated-test/serve-hostname-amd64:1.0": rpc error: code = Unknown desc = repository gcr.io/k8s-authenticated-test/serve-hostname-amd64 not found: does not exist or no pull access. Some recent e2e-gcp tests are green, might be a flake.

/retest

sdodson · 2020-12-07T13:42:15Z

/test artifacts

sdodson · 2020-12-07T13:43:14Z

/override ci/openshift-jenkins/extended_conformance_install
Bug on that https://bugzilla.redhat.com/show_bug.cgi?id=1902067
/override ci/openshift-jenkins/cmd

openshift-ci-robot · 2020-12-07T13:43:24Z

@sdodson: Overrode contexts on behalf of sdodson: ci/openshift-jenkins/cmd, ci/openshift-jenkins/extended_conformance_install

In response to this:

/override ci/openshift-jenkins/extended_conformance_install
Bug on that https://bugzilla.redhat.com/show_bug.cgi?id=1902067
/override ci/openshift-jenkins/cmd

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-12-07T13:56:55Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: jsafrane, tsmetana

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-12-07T15:05:07Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-12-07T15:45:47Z

@jsafrane: All pull requests linked via external trackers have merged:

openshift/origin#25733

Bugzilla bug 1903524 has been moved to the MODIFIED state.

In response to this:

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from sjenning and smarterclayton December 2, 2020 17:15

openshift-ci-robot added the vendor-update Touching vendor dir or related files label Dec 2, 2020

jsafrane changed the base branch from master to release-3.11 December 2, 2020 17:16

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Dec 2, 2020

jsafrane force-pushed the fc-fixcorrupted-fs branch from d1385fc to 466689c Compare December 2, 2020 17:24

jsafrane force-pushed the fc-fixcorrupted-fs branch from 466689c to ad59887 Compare December 2, 2020 17:25

Flush FibreChannel devices before deleting

da05315

If a FibreChannel device is used as a block volume, we should flush its I/O before deleting its device. It is not strictly necessary when it's used as a filesystem (mount), but it won't hurt either.

tsmetana approved these changes Dec 3, 2020

View reviewed changes

openshift-ci-robot assigned tsmetana Dec 3, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 3, 2020

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2020

sdodson added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 7, 2020

openshift-merge-robot merged commit 8852839 into openshift:release-3.11 Dec 7, 2020

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach #25733

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach #25733

jsafrane commented Dec 2, 2020 •

edited

openshift-ci-robot commented Dec 2, 2020

openshift-ci-robot commented Dec 2, 2020

openshift-ci-robot commented Dec 2, 2020

tsmetana left a comment

jsafrane commented Dec 3, 2020

jsafrane commented Dec 3, 2020

openshift-ci-robot commented Dec 3, 2020

jsafrane commented Dec 7, 2020

jsafrane commented Dec 7, 2020 •

edited

sdodson commented Dec 7, 2020

sdodson commented Dec 7, 2020

openshift-ci-robot commented Dec 7, 2020

openshift-ci-robot commented Dec 7, 2020

openshift-bot commented Dec 7, 2020

openshift-ci-robot commented Dec 7, 2020

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach #25733

Bug 1903524: [3.11] UPSTREAM: 97013: Fix FibreChannel volume plugin corrupting filesystem on detach #25733

Conversation

jsafrane commented Dec 2, 2020 • edited

openshift-ci-robot commented Dec 2, 2020

openshift-ci-robot commented Dec 2, 2020

openshift-ci-robot commented Dec 2, 2020

tsmetana left a comment

Choose a reason for hiding this comment

jsafrane commented Dec 3, 2020

jsafrane commented Dec 3, 2020

openshift-ci-robot commented Dec 3, 2020

jsafrane commented Dec 7, 2020

jsafrane commented Dec 7, 2020 • edited

sdodson commented Dec 7, 2020

sdodson commented Dec 7, 2020

openshift-ci-robot commented Dec 7, 2020

openshift-ci-robot commented Dec 7, 2020

openshift-bot commented Dec 7, 2020

openshift-ci-robot commented Dec 7, 2020

jsafrane commented Dec 2, 2020 •

edited

jsafrane commented Dec 7, 2020 •

edited