RBD in-tree plugin migration to CSI driver using migration translation lib #95361

humblec · 2020-10-07T12:34:30Z

Signed-off-by: Humble Chirammal hchiramm@redhat.com

What type of PR is this?

In support of csi-migration proposal here:
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/csi-migration.md

this help with migration of in-tree RBD plugin ( kubernetes.io/rbd) to RBD CSI driver ( rbd.csi.ceph.com ).

Fixes #kubernetes/enhancements#2923

/kind feature
/kind sig-storage

Updates # #91246

-->

 This release enables in-tree RBD migration to CSI driver with a couple of feature gates.  These featuregates are alpha in this release.

- `CSIMigrationRBD`: when enable, it will redirect traffic from in tree rbd plugin ( kubernetes.io/rbd )  to CSI driver ( rbd.csi.ceph.com) , default to `false` now.
- `IntreePluginRBDUnregister`: Disables the RBD in-tree driver

The featuregates can be enabled by:

1. Adding the feature flag to the kube-controller-manager `--feature-gates=CSIMigrationRBD=true` 
2. Adding the feature flag to the kubelet config:

featureGates:
  CSIMigrationRBD: true


As a Kubernetes cluster operator that administers storage, here are the prerequisites that you must complete before you attempt migration to the RBD CSI driver:

* You must install the Ceph CSI driver (rbd.csi.ceph.com), v3.5.0 or above, into your Kubernetes cluster.
* Considering the clusterID field is a required parameter for CSI driver for its operations, but in-tree StorageClass has monitors field as a required parameter, a Kubernetes storage admin has to create a clusterID based on the monitors hash ( ex:#echo -n '<monitors_string>' | md5sum) in the CSI config map and keep the monitors under this clusterID configuration.
* Also, if the value of adminId in the in-tree Storageclass is different from admin, the adminSecretName mentioned in the in-tree Storageclass has to be patched with the base64 value of the adminId parameter value, otherwise this step can be skipped.

fedebongio · 2020-10-08T20:08:13Z

/remove-sig api-machinery

huchengze · 2020-11-27T01:23:32Z

@humblec Hello! Are you still working on this?

I found that to finish rbd csi migration, you not only need to update the code in kubernetes, but also some code in external-provisioner .

If you don't have enough time to do this, I can help.

humblec · 2021-02-08T10:40:21Z

@humblec Hello! Are you still working on this?

I found that to finish rbd csi migration, you not only need to update the code in kubernetes, but also some code in external-provisioner .

If you don't have enough time to do this, I can help.

@huchengze due to some personal emergency I was away so the delay.. Will be revisiting soon.

fejta-bot · 2021-05-09T11:13:58Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

fejta-bot · 2021-06-08T11:38:53Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

humblec · 2021-11-15T18:18:21Z

/approve /hold

@msau42 please unhold once the feature gate is added to

kubernetes/pkg/scheduler/framework/plugins/volumebinding/binder.go

Line 1018 in ed3d9f7

func isCSIMigrationOnForPlugin(pluginName string) bool {

@ahg-g @msau42 addressed above, ptal.. thanks in advance. 👍

sttts · 2021-11-15T18:41:24Z

/approve

for the kcm part

k8s-ci-robot · 2021-11-15T19:47:28Z

@humblec: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required
pull-kubernetes-bazel-test	1095740a1ce3a1f32631cc152ecafe88c8195b42	link	`/test pull-kubernetes-bazel-test`
pull-kubernetes-node-e2e	1095740a1ce3a1f32631cc152ecafe88c8195b42	link	`/test pull-kubernetes-node-e2e`
pull-kubernetes-bazel-build	1095740a1ce3a1f32631cc152ecafe88c8195b42	link	`/test pull-kubernetes-bazel-build`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-11-15T20:22:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahg-g, humblec, jsafrane, mrunalp, msau42, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kube-controller-manager/OWNERS~~ [sttts]
~~cmd/kubelet/OWNERS~~ [mrunalp]
~~pkg/features/OWNERS~~ [ahg-g,jsafrane,msau42]
~~pkg/scheduler/OWNERS~~ [ahg-g]
~~pkg/volume/OWNERS~~ [jsafrane,msau42]
~~staging/src/k8s.io/csi-translation-lib/OWNERS~~ [msau42]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

msau42 · 2021-11-15T20:39:01Z

/lgtm

humblec · 2021-11-16T02:04:53Z

/retest pull-kubernetes-e2e-gce-storage-slow

k8s-ci-robot · 2021-11-16T02:04:55Z

@humblec: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test pull-kubernetes-conformance-kind-ga-only-parallel
/test pull-kubernetes-dependencies
/test pull-kubernetes-dependencies-go-canary
/test pull-kubernetes-e2e-gce
/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-gce-big-performance
/test pull-kubernetes-e2e-gce-canary
/test pull-kubernetes-e2e-gce-large-performance
/test pull-kubernetes-e2e-gce-large-performance-canary
/test pull-kubernetes-e2e-gce-network-proxy-http-connect
/test pull-kubernetes-e2e-gce-no-stage
/test pull-kubernetes-e2e-gce-ubuntu
/test pull-kubernetes-e2e-gce-ubuntu-containerd
/test pull-kubernetes-e2e-gce-ubuntu-containerd-canary
/test pull-kubernetes-e2e-kind
/test pull-kubernetes-e2e-kind-ipv6
/test pull-kubernetes-files-remake
/test pull-kubernetes-integration
/test pull-kubernetes-integration-go-canary
/test pull-kubernetes-kubemark-e2e-gce-scale
/test pull-kubernetes-node-e2e-alpha
/test pull-kubernetes-node-e2e-containerd
/test pull-kubernetes-typecheck
/test pull-kubernetes-unit
/test pull-kubernetes-unit-go-canary
/test pull-kubernetes-verify
/test pull-kubernetes-verify-go-canary
/test pull-kubernetes-verify-govet-levee

The following commands are available to trigger optional jobs:

/test check-dependency-stats
/test pull-kubernetes-conformance-image-test
/test pull-kubernetes-conformance-kind-ga-only
/test pull-kubernetes-conformance-kind-ipv6-parallel
/test pull-kubernetes-cross
/test pull-kubernetes-e2e-aks-engine-azure-disk-windows-containerd
/test pull-kubernetes-e2e-aks-engine-azure-disk-windows-dockershim
/test pull-kubernetes-e2e-aks-engine-azure-file-windows-containerd
/test pull-kubernetes-e2e-aks-engine-azure-file-windows-dockershim
/test pull-kubernetes-e2e-aks-engine-gpu-windows-dockershim
/test pull-kubernetes-e2e-aks-engine-windows-containerd
/test pull-kubernetes-e2e-aks-engine-windows-dockershim
/test pull-kubernetes-e2e-capz-azure-disk
/test pull-kubernetes-e2e-capz-azure-disk-vmss
/test pull-kubernetes-e2e-capz-azure-file
/test pull-kubernetes-e2e-capz-azure-file-vmss
/test pull-kubernetes-e2e-capz-conformance
/test pull-kubernetes-e2e-capz-ha-control-plane
/test pull-kubernetes-e2e-capz-windows-dockershim
/test pull-kubernetes-e2e-containerd-gce
/test pull-kubernetes-e2e-gce-alpha-features
/test pull-kubernetes-e2e-gce-correctness
/test pull-kubernetes-e2e-gce-csi-serial
/test pull-kubernetes-e2e-gce-device-plugin-gpu
/test pull-kubernetes-e2e-gce-iscsi
/test pull-kubernetes-e2e-gce-iscsi-serial
/test pull-kubernetes-e2e-gce-kubetest2
/test pull-kubernetes-e2e-gce-network-proxy-grpc
/test pull-kubernetes-e2e-gce-storage-disruptive
/test pull-kubernetes-e2e-gce-storage-slow
/test pull-kubernetes-e2e-gce-storage-snapshot
/test pull-kubernetes-e2e-gci-gce-autoscaling
/test pull-kubernetes-e2e-gci-gce-ingress
/test pull-kubernetes-e2e-gci-gce-ipvs
/test pull-kubernetes-e2e-iptables-azure-dualstack
/test pull-kubernetes-e2e-ipvs-azure-dualstack
/test pull-kubernetes-e2e-kind-canary
/test pull-kubernetes-e2e-kind-dual-canary
/test pull-kubernetes-e2e-kind-ipv6-canary
/test pull-kubernetes-e2e-kind-ipvs-dual-canary
/test pull-kubernetes-e2e-kind-multizone
/test pull-kubernetes-e2e-kops-aws
/test pull-kubernetes-e2e-ubuntu-gce-network-policies
/test pull-kubernetes-e2e-windows-gce
/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-local-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e-kubetest2
/test pull-kubernetes-node-crio-e2e
/test pull-kubernetes-node-crio-e2e-kubetest2
/test pull-kubernetes-node-e2e
/test pull-kubernetes-node-e2e-alpha-kubetest2
/test pull-kubernetes-node-e2e-containerd-features
/test pull-kubernetes-node-e2e-containerd-features-kubetest2
/test pull-kubernetes-node-e2e-containerd-kubetest2
/test pull-kubernetes-node-e2e-kubetest2
/test pull-kubernetes-node-e2e-podutil
/test pull-kubernetes-node-kubelet-eviction
/test pull-kubernetes-node-kubelet-eviction-kubetest2
/test pull-kubernetes-node-kubelet-serial
/test pull-kubernetes-node-kubelet-serial-containerd
/test pull-kubernetes-node-kubelet-serial-containerd-kubetest2
/test pull-kubernetes-node-kubelet-serial-cpu-manager
/test pull-kubernetes-node-kubelet-serial-cpu-manager-kubetest2
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv1
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv2
/test pull-kubernetes-node-kubelet-serial-dockershim
/test pull-kubernetes-node-kubelet-serial-hugepages
/test pull-kubernetes-node-kubelet-serial-kubetest2
/test pull-kubernetes-node-kubelet-serial-memory-manager
/test pull-kubernetes-node-kubelet-serial-topology-manager
/test pull-kubernetes-node-kubelet-serial-topology-manager-kubetest2
/test pull-kubernetes-node-memoryqos-cgrpv2
/test pull-kubernetes-node-swap-fedora
/test pull-kubernetes-node-swap-ubuntu
/test pull-kubernetes-unit-experimental
/test pull-publishing-bot-validate

Use /test all to run the following jobs that were automatically triggered:

pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-dependencies
pull-kubernetes-e2e-gce-100-performance
pull-kubernetes-e2e-gce-alpha-features
pull-kubernetes-e2e-gce-csi-serial
pull-kubernetes-e2e-gce-storage-slow
pull-kubernetes-e2e-gce-storage-snapshot
pull-kubernetes-e2e-gce-ubuntu-containerd
pull-kubernetes-e2e-kind
pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-integration
pull-kubernetes-node-e2e-containerd
pull-kubernetes-typecheck
pull-kubernetes-unit
pull-kubernetes-verify
pull-kubernetes-verify-govet-levee

In response to this:

/retest pull-kubernetes-e2e-gce-storage-slow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

humblec · 2021-11-16T02:05:36Z

/test pull-kubernetes-e2e-gce-storage-slow

msau42 · 2021-11-16T02:11:12Z

/hold cancel

Kubernetes adds the RBD CSI migration functionality via CSI migration translation lib. Ref# kubernetes/kubernetes#95361 This commit add the migration details to volumes.md and feature-gates.md Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

humblec · 2022-02-08T02:53:01Z

Demo : part 1 (existPVC) shows PVCs created before enabling migration and part2 shows PVCs created after migration.
https://drive.google.com/file/d/1yfcngCEyrOPDG5tOg7gva0wVJqNELRGc/view

Ref# kubernetes#95361 kubernetes#111137 Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

Kubernetes adds the RBD CSI migration functionality via CSI migration translation lib. Ref# kubernetes/kubernetes#95361 This commit add the migration details to volumes.md and feature-gates.md Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

k8s-ci-robot requested review from piosz and Random-Liu October 7, 2020 12:36

humblec changed the title ~~Migration library for RBD intree to CSI volume~~ [WIP] Migration library for RBD intree to CSI volume Oct 7, 2020

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 7, 2020

humblec mentioned this pull request Oct 7, 2020

Migrate cephfs and rbd intree plugins by CSI Migration path. #91246

Closed

k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Oct 8, 2020

spiffxp added do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Nov 5, 2020

pacoxu mentioned this pull request Nov 26, 2020

Is there a plan to add rbd csimigration controller? #96880

Closed

ehashman added this to In Progress in SIG Node PR Triage Jan 6, 2021

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 8, 2021

humblec mentioned this pull request Mar 15, 2021

1.16.3. Error: "failed to create rbd image: executable file not found in $PATH, command output: " #85454

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 9, 2021

mrunalp approved these changes Nov 15, 2021

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 15, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 15, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 16, 2021

k8s-ci-robot merged commit 66c342b into kubernetes:master Nov 16, 2021

k8s-ci-robot added this to the v1.23 milestone Nov 16, 2021

humblec mentioned this pull request Nov 17, 2021

Add RBD CSI migration section to the storage volumes guide kubernetes/website#30530

Closed

humblec mentioned this pull request Nov 17, 2021

Add RBD CSI migration section to the storage volumes guide kubernetes/website#30533

Merged

cici37 mentioned this pull request Dec 16, 2021

Update CHANGELOG to add missing release notes. #107084

Merged

pacoxu mentioned this pull request Jan 12, 2022

The feature gate csiMigrationRBD should be CSIMigrationRBD #107488

Open

humblec added a commit to humblec/kubernetes that referenced this pull request Sep 21, 2022

Add humblec to approvers and reviewers list for Ceph driver

c15757d

Ref# kubernetes#95361 kubernetes#111137 Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

humblec mentioned this pull request Sep 21, 2022

Add humblec to approvers and reviewers list for Ceph driver #112640

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RBD in-tree plugin migration to CSI driver using migration translation lib #95361

RBD in-tree plugin migration to CSI driver using migration translation lib #95361

humblec commented Oct 7, 2020 •

edited

fedebongio commented Oct 8, 2020

huchengze commented Nov 27, 2020

humblec commented Feb 8, 2021

fejta-bot commented May 9, 2021

fejta-bot commented Jun 8, 2021

humblec commented Nov 15, 2021

sttts commented Nov 15, 2021

k8s-ci-robot commented Nov 15, 2021 •

edited

k8s-ci-robot commented Nov 15, 2021

msau42 commented Nov 15, 2021

humblec commented Nov 16, 2021

k8s-ci-robot commented Nov 16, 2021

humblec commented Nov 16, 2021

msau42 commented Nov 16, 2021

humblec commented Feb 8, 2022 •

edited

RBD in-tree plugin migration to CSI driver using migration translation lib #95361

RBD in-tree plugin migration to CSI driver using migration translation lib #95361

Conversation

humblec commented Oct 7, 2020 • edited

fedebongio commented Oct 8, 2020

huchengze commented Nov 27, 2020

humblec commented Feb 8, 2021

fejta-bot commented May 9, 2021

fejta-bot commented Jun 8, 2021

humblec commented Nov 15, 2021

sttts commented Nov 15, 2021

k8s-ci-robot commented Nov 15, 2021 • edited

k8s-ci-robot commented Nov 15, 2021

msau42 commented Nov 15, 2021

humblec commented Nov 16, 2021

k8s-ci-robot commented Nov 16, 2021

humblec commented Nov 16, 2021

msau42 commented Nov 16, 2021

humblec commented Feb 8, 2022 • edited

humblec commented Oct 7, 2020 •

edited

k8s-ci-robot commented Nov 15, 2021 •

edited

humblec commented Feb 8, 2022 •

edited