rbd: add replication capability & service to csiaddons server #3305

Rakshith-R · 2022-08-16T05:53:59Z

csi-addons server will advertise replication capability and
replication service will run with csi-addons server too.

Updates: #3277

Signed-off-by: Rakshith R rar@redhat.com

Testing is done with entire csi-addons controller, rook-ceph setup, Everything works perfectly.

I0816 10:51:09.662986       1 utils.go:195] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC call: /replication.Controller/PromoteVolume
I0816 10:51:09.663087       1 utils.go:197] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:09.671059       1 omap.go:88] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b got omap values: (pool="replicapool", namespace="", name="csi.volume.dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"): map[csi.imageid:13b3a033ed4f csi.imagename:csi-vol-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b csi.volname:pvc-d04b0cf4-bacd-41ac-a1fc-84ba3aa67f69 csi.volume.owner:rook-ceph]
I0816 10:51:09.713428       1 replicationcontrollerserver.go:615] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b attempting to tickle dummy image for restarting RBD schedules
I0816 10:51:10.775350       1 utils.go:206] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC response: {}

I0816 10:23:43.411151       1 connection.go:154] Connecting to unix:///csi/csi-addons.sock
I0816 10:23:44.412209       1 client.go:79] Probing CSI driver for readiness
I0816 10:23:44.412228       1 connection.go:183] GRPC call: /identity.Identity/Probe
I0816 10:23:44.412234       1 connection.go:184] GRPC request: {}
I0816 10:23:44.414265       1 connection.go:186] GRPC response: {"ready":{"value":true}}
I0816 10:23:44.414380       1 connection.go:187] GRPC error: <nil>
I0816 10:23:44.415085       1 connection.go:183] GRPC call: /identity.Identity/GetIdentity
I0816 10:23:44.415100       1 connection.go:184] GRPC request: {}
I0816 10:23:44.415713       1 connection.go:186] GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"canary"}
I0816 10:23:44.415794       1 connection.go:187] GRPC error: <nil>
I0816 10:23:44.424010       1 csiaddonsnode.go:99] CSIAddonsNode 'rook-ceph/csi-rbdplugin-provisioner-5c4c4876db-vf9tz' successfully created
I0816 10:23:44.424209       1 server.go:90] Listening for CSI-Addons requests on address: 172.17.0.11:9070
I0816 10:23:44.440485       1 connection.go:183] GRPC call: /identity.Identity/GetCapabilities
I0816 10:23:44.440505       1 connection.go:184] GRPC request: {}
I0816 10:23:44.442059       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"ReclaimSpace":{"type":1}}},{"Type":{"NetworkFence":{"type":1}}},{"Type":{"VolumeReplication":{"type":1}}}]}
I0816 10:23:44.442339       1 connection.go:187] GRPC error: <nil>
I0816 10:39:43.527025       1 connection.go:183] GRPC call: /identity.Identity/GetCapabilities
I0816 10:39:43.527037       1 connection.go:184] GRPC request: {}
I0816 10:39:43.529601       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"ReclaimSpace":{"type":1}}},{"Type":{"NetworkFence":{"type":1}}},{"Type":{"VolumeReplication":{"type":1}}}]}
I0816 10:39:43.529749       1 connection.go:187] GRPC error: <nil>
I0816 10:51:09.478678       1 connection.go:183] GRPC call: /replication.Controller/EnableVolumeReplication
I0816 10:51:09.478717       1 connection.go:184] GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":{"userID":"csi-rbd-provisioner","userKey":"AQCTa/tiAEPQCRAAQlzzu+5AlGTPXijj7qACfw=="},"volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:09.659917       1 connection.go:186] GRPC response: {}
I0816 10:51:09.659962       1 connection.go:187] GRPC error: <nil>
I0816 10:51:09.662515       1 connection.go:183] GRPC call: /replication.Controller/PromoteVolume
I0816 10:51:09.662527       1 connection.go:184] GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":{"userID":"csi-rbd-provisioner","userKey":"AQCTa/tiAEPQCRAAQlzzu+5AlGTPXijj7qACfw=="},"volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:10.775573       1 connection.go:186] GRPC response: {}
I0816 10:51:10.775613       1 connection.go:187] GRPC error: <nil>

VolRep service will be completely removed from controllerserver in v3.8 and will only be accessible from csi-addons setup.

Will be followed up with #3314

nixpanic · 2022-08-16T07:21:04Z

What are the TODO's for this PR before it will be out of draft?

I expect at least the following:

drop Kubernetes controller
move logic to internal/csi-addons/rbd
add minimal unit tests like internal/csi-addons/rbd/reclaimspace_test.go

Rakshith-R · 2022-08-16T07:51:06Z

What are the TODO's for this PR before it will be out of draft?

I expect at least the following:

drop Kubernetes controller

move logic to internal/csi-addons/rbd

add minimal unit tests like internal/csi-addons/rbd/reclaimspace_test.go

I was thinking to keep the replication services with controller server too for this release and log warning with its usage.
I'll open an issue to track all this for v3.8.

I'll move the pr out of draft after some more testing.

Rakshith-R · 2022-08-16T10:53:32Z

Testing is done, Everything works perfectly.

I0816 10:51:09.662986       1 utils.go:195] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC call: /replication.Controller/PromoteVolume
I0816 10:51:09.663087       1 utils.go:197] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:09.671059       1 omap.go:88] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b got omap values: (pool="replicapool", namespace="", name="csi.volume.dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"): map[csi.imageid:13b3a033ed4f csi.imagename:csi-vol-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b csi.volname:pvc-d04b0cf4-bacd-41ac-a1fc-84ba3aa67f69 csi.volume.owner:rook-ceph]
I0816 10:51:09.713428       1 replicationcontrollerserver.go:615] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b attempting to tickle dummy image for restarting RBD schedules
I0816 10:51:10.775350       1 utils.go:206] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC response: {}

I0816 10:23:43.411151       1 connection.go:154] Connecting to unix:///csi/csi-addons.sock
I0816 10:23:44.412209       1 client.go:79] Probing CSI driver for readiness
I0816 10:23:44.412228       1 connection.go:183] GRPC call: /identity.Identity/Probe
I0816 10:23:44.412234       1 connection.go:184] GRPC request: {}
I0816 10:23:44.414265       1 connection.go:186] GRPC response: {"ready":{"value":true}}
I0816 10:23:44.414380       1 connection.go:187] GRPC error: <nil>
I0816 10:23:44.415085       1 connection.go:183] GRPC call: /identity.Identity/GetIdentity
I0816 10:23:44.415100       1 connection.go:184] GRPC request: {}
I0816 10:23:44.415713       1 connection.go:186] GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"canary"}
I0816 10:23:44.415794       1 connection.go:187] GRPC error: <nil>
I0816 10:23:44.424010       1 csiaddonsnode.go:99] CSIAddonsNode 'rook-ceph/csi-rbdplugin-provisioner-5c4c4876db-vf9tz' successfully created
I0816 10:23:44.424209       1 server.go:90] Listening for CSI-Addons requests on address: 172.17.0.11:9070
I0816 10:23:44.440485       1 connection.go:183] GRPC call: /identity.Identity/GetCapabilities
I0816 10:23:44.440505       1 connection.go:184] GRPC request: {}
I0816 10:23:44.442059       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"ReclaimSpace":{"type":1}}},{"Type":{"NetworkFence":{"type":1}}},{"Type":{"VolumeReplication":{"type":1}}}]}
I0816 10:23:44.442339       1 connection.go:187] GRPC error: <nil>
I0816 10:39:43.527025       1 connection.go:183] GRPC call: /identity.Identity/GetCapabilities
I0816 10:39:43.527037       1 connection.go:184] GRPC request: {}
I0816 10:39:43.529601       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"ReclaimSpace":{"type":1}}},{"Type":{"NetworkFence":{"type":1}}},{"Type":{"VolumeReplication":{"type":1}}}]}
I0816 10:39:43.529749       1 connection.go:187] GRPC error: <nil>
I0816 10:51:09.478678       1 connection.go:183] GRPC call: /replication.Controller/EnableVolumeReplication
I0816 10:51:09.478717       1 connection.go:184] GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":{"userID":"csi-rbd-provisioner","userKey":"AQCTa/tiAEPQCRAAQlzzu+5AlGTPXijj7qACfw=="},"volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:09.659917       1 connection.go:186] GRPC response: {}
I0816 10:51:09.659962       1 connection.go:187] GRPC error: <nil>
I0816 10:51:09.662515       1 connection.go:183] GRPC call: /replication.Controller/PromoteVolume
I0816 10:51:09.662527       1 connection.go:184] GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":{"userID":"csi-rbd-provisioner","userKey":"AQCTa/tiAEPQCRAAQlzzu+5AlGTPXijj7qACfw=="},"volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:10.775573       1 connection.go:186] GRPC response: {}
I0816 10:51:10.775613       1 connection.go:187] GRPC error: <nil>

nixpanic · 2022-08-17T16:07:06Z

Testing is done, Everything works perfectly.

I0816 10:51:09.662986       1 utils.go:195] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC call: /replication.Controller/PromoteVolume
I0816 10:51:09.663087       1 utils.go:197] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:09.671059       1 omap.go:88] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b got omap values: (pool="replicapool", namespace="", name="csi.volume.dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"): map[csi.imageid:13b3a033ed4f csi.imagename:csi-vol-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b csi.volname:pvc-d04b0cf4-bacd-41ac-a1fc-84ba3aa67f69 csi.volume.owner:rook-ceph]
I0816 10:51:09.713428       1 replicationcontrollerserver.go:615] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b attempting to tickle dummy image for restarting RBD schedules
I0816 10:51:10.775350       1 utils.go:206] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b GRPC response: {}

I0816 10:23:43.411151       1 connection.go:154] Connecting to unix:///csi/csi-addons.sock
I0816 10:23:44.412209       1 client.go:79] Probing CSI driver for readiness
I0816 10:23:44.412228       1 connection.go:183] GRPC call: /identity.Identity/Probe
I0816 10:23:44.412234       1 connection.go:184] GRPC request: {}
I0816 10:23:44.414265       1 connection.go:186] GRPC response: {"ready":{"value":true}}
I0816 10:23:44.414380       1 connection.go:187] GRPC error: <nil>
I0816 10:23:44.415085       1 connection.go:183] GRPC call: /identity.Identity/GetIdentity
I0816 10:23:44.415100       1 connection.go:184] GRPC request: {}
I0816 10:23:44.415713       1 connection.go:186] GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"canary"}
I0816 10:23:44.415794       1 connection.go:187] GRPC error: <nil>
I0816 10:23:44.424010       1 csiaddonsnode.go:99] CSIAddonsNode 'rook-ceph/csi-rbdplugin-provisioner-5c4c4876db-vf9tz' successfully created
I0816 10:23:44.424209       1 server.go:90] Listening for CSI-Addons requests on address: 172.17.0.11:9070
I0816 10:23:44.440485       1 connection.go:183] GRPC call: /identity.Identity/GetCapabilities
I0816 10:23:44.440505       1 connection.go:184] GRPC request: {}
I0816 10:23:44.442059       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"ReclaimSpace":{"type":1}}},{"Type":{"NetworkFence":{"type":1}}},{"Type":{"VolumeReplication":{"type":1}}}]}
I0816 10:23:44.442339       1 connection.go:187] GRPC error: <nil>
I0816 10:39:43.527025       1 connection.go:183] GRPC call: /identity.Identity/GetCapabilities
I0816 10:39:43.527037       1 connection.go:184] GRPC request: {}
I0816 10:39:43.529601       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"ReclaimSpace":{"type":1}}},{"Type":{"NetworkFence":{"type":1}}},{"Type":{"VolumeReplication":{"type":1}}}]}
I0816 10:39:43.529749       1 connection.go:187] GRPC error: <nil>
I0816 10:51:09.478678       1 connection.go:183] GRPC call: /replication.Controller/EnableVolumeReplication
I0816 10:51:09.478717       1 connection.go:184] GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":{"userID":"csi-rbd-provisioner","userKey":"AQCTa/tiAEPQCRAAQlzzu+5AlGTPXijj7qACfw=="},"volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:09.659917       1 connection.go:186] GRPC response: {}
I0816 10:51:09.659962       1 connection.go:187] GRPC error: <nil>
I0816 10:51:09.662515       1 connection.go:183] GRPC call: /replication.Controller/PromoteVolume
I0816 10:51:09.662527       1 connection.go:184] GRPC request: {"parameters":{"mirroringMode":"snapshot"},"secrets":{"userID":"csi-rbd-provisioner","userKey":"AQCTa/tiAEPQCRAAQlzzu+5AlGTPXijj7qACfw=="},"volume_id":"0001-0009-rook-ceph-0000000000000001-dc2ae0f8-1d4f-11ed-b7f0-0242ac11000b"}
I0816 10:51:10.775573       1 connection.go:186] GRPC response: {}
I0816 10:51:10.775613       1 connection.go:187] GRPC error: <nil>

How did you test this? With the csi-addons tool from the csi-addons sidecar? Is there a PR that adds the functionality to https://github.com/csi-addons/kubernetes-csi-addons/tree/main/cmd/csi-addons ?

Rakshith-R · 2022-08-18T04:30:17Z

How did you test this? With the csi-addons tool from the csi-addons sidecar? Is there a PR that adds the functionality to https://github.com/csi-addons/kubernetes-csi-addons/tree/main/cmd/csi-addons ?

I've written code to add vol-rep controller under kubernetes csi addons, I've tested with fully working setup of csi-addons controller, rook-ceph, vrc and vr deployed.
The WIP code is here https://github.com/rakshith-r/kubernetes-csi-addons/tree/add-vr
I'll be opening a pr at kubernetes-csi-addons shortly.

humblec · 2022-08-18T05:29:25Z

This looks good .. Appreciated if you can quickly update the PR description with the details on the things covered and the roadmap of changes planned for future releases @Rakshith-R 👍

Rakshith-R · 2022-08-18T05:36:36Z

This looks good .. Appreciated if you can quickly update the PR description with the details on the things covered and the roadmap of changes planned for future releases @Rakshith-R +1

done ptal, thanks

internal/rbd/replicationcontrollerserver.go

internal/rbd/driver/driver.go

mergify · 2022-08-18T07:41:58Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

csi-addons/spec is updated to latest to get replication identity capability. Signed-off-by: Rakshith R <rar@redhat.com>

csi-addons server will advertise replication capability and replication service will run with csi-addons server too. Signed-off-by: Rakshith R <rar@redhat.com>

Pull request has been modified.

Rakshith-R added the ci/skip/e2e skip running e2e CI jobs label Aug 16, 2022

mergify bot added the component/rbd Issues related to RBD label Aug 16, 2022

Rakshith-R force-pushed the mv-vr branch from 5c8a838 to f38a0f5 Compare August 16, 2022 06:10

Rakshith-R force-pushed the mv-vr branch from f38a0f5 to b1807a6 Compare August 16, 2022 07:47

Rakshith-R force-pushed the mv-vr branch from b1807a6 to c2b493f Compare August 16, 2022 09:57

Rakshith-R marked this pull request as ready for review August 16, 2022 10:53

Rakshith-R requested review from nixpanic, Madhu-1, ShyamsundarR and a team August 16, 2022 10:53

Rakshith-R linked an issue Aug 17, 2022 that may be closed by this pull request

Move volume replication operator to kubernetes-csi-addons #3277

Closed

2 tasks

Rakshith-R requested a review from a team August 18, 2022 04:30

Rakshith-R mentioned this pull request Aug 18, 2022

rbd: complete migration of volrep to csi-addons #3314

Closed

5 tasks

Rakshith-R requested a review from humblec August 18, 2022 06:40

nixpanic added the dependency/csi-addons Related to a CSI-Addons issue or enhancement label Aug 18, 2022

nixpanic requested changes Aug 18, 2022

View reviewed changes

internal/rbd/replicationcontrollerserver.go Outdated Show resolved Hide resolved

internal/rbd/driver/driver.go Show resolved Hide resolved

Rakshith-R force-pushed the mv-vr branch from c2b493f to a6811dc Compare August 18, 2022 07:29

Rakshith-R requested a review from nixpanic August 18, 2022 07:29

nixpanic previously approved these changes Aug 18, 2022

View reviewed changes

humblec previously approved these changes Aug 18, 2022

View reviewed changes

Rakshith-R added 2 commits August 18, 2022 13:15

rebase: update csi-addons/spec to latest

afe1e3c

csi-addons/spec is updated to latest to get replication identity capability. Signed-off-by: Rakshith R <rar@redhat.com>

rbd: add replication capability & service to csiaddons server

2ab5b85

csi-addons server will advertise replication capability and replication service will run with csi-addons server too. Signed-off-by: Rakshith R <rar@redhat.com>

Rakshith-R force-pushed the mv-vr branch from a6811dc to 2ab5b85 Compare August 18, 2022 07:45

Rakshith-R requested review from nixpanic and humblec August 18, 2022 07:46

nixpanic approved these changes Aug 18, 2022

View reviewed changes

humblec approved these changes Aug 18, 2022

View reviewed changes

mergify bot merged commit 19e4146 into ceph:devel Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rbd: add replication capability & service to csiaddons server #3305

rbd: add replication capability & service to csiaddons server #3305

Rakshith-R commented Aug 16, 2022 •

edited

Loading

nixpanic commented Aug 16, 2022

Rakshith-R commented Aug 16, 2022

Rakshith-R commented Aug 16, 2022

nixpanic commented Aug 17, 2022

Rakshith-R commented Aug 18, 2022

humblec commented Aug 18, 2022

Rakshith-R commented Aug 18, 2022

mergify bot commented Aug 18, 2022

rbd: add replication capability & service to csiaddons server #3305

rbd: add replication capability & service to csiaddons server #3305

Conversation

Rakshith-R commented Aug 16, 2022 • edited Loading

nixpanic commented Aug 16, 2022

Rakshith-R commented Aug 16, 2022

Rakshith-R commented Aug 16, 2022

nixpanic commented Aug 17, 2022

Rakshith-R commented Aug 18, 2022

humblec commented Aug 18, 2022

Rakshith-R commented Aug 18, 2022

mergify bot commented Aug 18, 2022

Rakshith-R commented Aug 16, 2022 •

edited

Loading