Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new iSCSI refcounter #80091

Merged
merged 2 commits into from
Jul 24, 2019
Merged

Add new iSCSI refcounter #80091

merged 2 commits into from
Jul 24, 2019

Conversation

bertinatto
Copy link
Member

@bertinatto bertinatto commented Jul 12, 2019 •
edited

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR adds a new refcounter mechanism to iSCSI storage plugin so that sessions are not logged out too early.

The previous refcounter looked at the mounted volumes to determine whether the session could be logged out or not. This works well with filesystem volumes, however, we can't use this mechanism for block volumes because they are not necessarily mounted in the host.

This new refcounter looks at the volume directories created/deleted by the kubelet instead.

Which issue(s) this PR fixes:

Fixes #74313

Does this PR introduce a user-facing change?:

NONE

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 12, 2019
@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 12, 2019
@bertinatto
Copy link
Member Author

/assign @bswartz @j-griffith

@bertinatto
Copy link
Member Author

I'll check if it's possible to add a test for this.

@bswartz bswartz mentioned this pull request Jul 15, 2019
Closed
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 18, 2019
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 19, 2019
@bertinatto
Copy link
Member Author

/assign @jsafrane

@bertinatto
Copy link
Member Author

@bswartz, @j-griffith, @jsafrane:

For now I'm mostly looking for feedback on the refcounter's strategy.

If you think the way it's counting the volumes is the way to go, I'll go ahead and forward and finish this up.

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 19, 2019
j-griffith
j-griffith reviewed Jul 19, 2019
View reviewed changes
pkg/volume/iscsi/iscsi_util.go
volumeMount := m.Name()
prefix := portal + "-" + iqn
if strings.HasPrefix(volumeMount, prefix) {
counter++
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this relying on lun to be the ref counter? That won't work in a RWM case for a block device I don't think

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I understand the question correctly, but the prefix doesn't have the lun, so the counter doesn't rely on it (only on portal + iqn).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was worried there was a uniqueness check based on lun number somewhere else in the changes. Thanks for clarifying.

@bertinatto bertinatto changed the title [WIP] Add new iSCSI refcounter Add new iSCSI refcounter Jul 22, 2019
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 22, 2019
@bertinatto
Copy link
Member Author

CC @humblec

bertinatto
bertinatto commented Jul 24, 2019
View reviewed changes
pkg/volume/iscsi/iscsi_util.go
@@ -718,10 +695,16 @@ func (util *ISCSIUtil) DetachBlockISCSIDisk(c iscsiDiskUnmapper, mapPath string)
if _, err = os.Stat(devicePath); err != nil {
return fmt.Errorf("failed to validate devicePath: %s", devicePath)
}
// check if the dev is using mpio and if so mount it via the dm-XX device
if mappedDevicePath := c.deviceUtil.FindMultipathDeviceForDevice(devicePath); mappedDevicePath != "" {
devicePath = mappedDevicePath
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was not used anywhere.

bertinatto
bertinatto commented Jul 24, 2019
View reviewed changes
pkg/volume/iscsi/iscsi_util.go

// If device is no longer used, see if need to logout the target
if isSessionBusy(c.iscsiDisk.plugin.host, portals[0], iqn) {
return nil
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to prevent races, the lock here is necessary just like in DetachDisk.

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 24, 2019 •
edited

@bertinatto: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-e2e-gce-iscsi-serial 44d7510 link /test pull-kubernetes-e2e-gce-iscsi-serial
pull-kubernetes-e2e-gce-iscsi 44d7510 link /test pull-kubernetes-e2e-gce-iscsi

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bertinatto
Copy link
Member Author

/test pull-kubernetes-kubemark-e2e-gce-big

@jsafrane
Copy link
Member

/approve
@bswartz @j-griffith @humblec, last chance to review, it looks good to me.

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2019
@j-griffith
Copy link
Contributor

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bertinatto, j-griffith, jsafrane

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@j-griffith
Copy link
Contributor

This looks good to me, I was holding off on the l g t m to give @bswartz a chance to chime in.

@bswartz
Copy link
Contributor

bswartz commented Jul 24, 2019

I don't object to this -- it looks like an improvement. It's not clear to me that the affects both raw block and filesystem volumes, but if so, then that's good.

I hope we consider looking at using sysfs for refcounting in the future. That would be a larger change, but probably even more reliable than this improvement.

@j-griffith
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 24, 2019
@k8s-ci-robot k8s-ci-robot merged commit 9c90337 into kubernetes:master Jul 24, 2019
@humblec
Copy link
Contributor

humblec commented Jul 25, 2019

Apologies, late to the party :(. Thanks @jsafrane and @j-griffith for reviewing it!

@jsafrane jsafrane mentioned this pull request Aug 1, 2019
Closed
@bertinatto bertinatto mentioned this pull request Nov 4, 2019
Closed
6 tasks
@ralfonso ralfonso mentioned this pull request Nov 27, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@j-griffith j-griffith j-griffith left review comments

@humblec humblec Awaiting requested review from humblec

@jingxu97 jingxu97 Awaiting requested review from jingxu97

Assignees

@j-griffith j-griffith

@bswartz bswartz

@jsafrane jsafrane

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Iscsi raw block support logs out from target too early
6 participants
@bertinatto @k8s-ci-robot @jsafrane @j-griffith @bswartz @humblec