open ebs node installation failling with registration error

[pardhasm]

Hi,

I followed the following article https://openebs.io/blog/openebs-dynamic-volume-provisioning-on-zfs/ to install zfs-localpv in my K8s cluster.

It keeps failing with the following error.

│ csi-node-driver-registrar I0429 16:06:11.322451       1 main.go:110] Version: v1.2.0-0-g6ef000ae                                                                                                         │
│ csi-node-driver-registrar I0429 16:06:11.322531       1 main.go:120] Attempting to open a gRPC connection with: "/plugin/csi.sock"                                                                       │
│ csi-node-driver-registrar I0429 16:06:11.322545       1 connection.go:151] Connecting to unix:///plugin/csi.sock                                                                                         │
│ csi-node-driver-registrar I0429 16:06:11.322870       1 main.go:127] Calling CSI driver to discover driver name                                                                                          │
│ csi-node-driver-registrar I0429 16:06:11.323255       1 connection.go:180] GRPC call: /csi.v1.Identity/GetPluginInfo                                                                                     │
│ csi-node-driver-registrar I0429 16:06:11.323287       1 connection.go:181] GRPC request: {}                                                                                                              │
│ csi-node-driver-registrar I0429 16:06:11.326587       1 connection.go:183] GRPC response: {"name":"zfs.csi.openebs.io","vendor_version":"master-02bc587:04-28-2020"}                                     │
│ csi-node-driver-registrar I0429 16:06:11.327524       1 connection.go:184] GRPC error: <nil>                                                                                                             │
│ csi-node-driver-registrar I0429 16:06:11.327544       1 main.go:137] CSI driver name: "zfs.csi.openebs.io"                                                                                               │
│ csi-node-driver-registrar I0429 16:06:11.327615       1 node_register.go:58] Starting Registration Server at: /registration/zfs.csi.openebs.io-reg.sock                                                  │
│ csi-node-driver-registrar I0429 16:06:11.327798       1 node_register.go:67] Registration Server started at: /registration/zfs.csi.openebs.io-reg.sock                                                   │
│ csi-node-driver-registrar I0429 16:06:11.471563       1 main.go:77] Received GetInfo call: &InfoRequest{}                                                                                                │
│ csi-node-driver-registrar I0429 16:06:12.471906       1 main.go:77] Received GetInfo call: &InfoRequest{}                                                                                                │
│ csi-node-driver-registrar I0429 16:06:12.821947       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration │
│  failed with err: error updating Node object with CSI driver node info: error updating node: timed out waiting for the condition; caused by: detected topology value collision: driver reported "kuberne │
│ tes.io/hostname":"ip-10-99-90-189.ap-south-1.compute.internal" but existing label is "kubernetes.io/hostname":"ip-10-99-90-189",}                                                                        │
│ csi-node-driver-registrar E0429 16:06:12.822033       1 main.go:89] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: error updating Node object with │
│  CSI driver node info: error updating node: timed out waiting for the condition; caused by: detected topology value collision: driver reported "kubernetes.io/hostname":"ip-10-99-90-189.ap-south-1.comp │
│ ute.internal" but existing label is "kubernetes.io/hostname":"ip-10-99-90-189", restarting registration container.                                                                                       │
│ csi-node-driver-registrar stream closed

I haven't modified anything. Using the vanilla config

[pawanpraka1]

@pardhasm, I think your setup has different nodename and hostname. The driver is trying to update the node lable "kubernetes.io/hostname" with the value as "ip-10-99-90-189", but it already has different label on the node. Can you verify this.

kubectl get nodes --show-labels

I have already planned to fix this, waiting for #94 to be merged. Then I will raise a PR to fix this issue.

[pardhasm]

These are the labels for the instance

NAME                                          STATUS   ROLES    AGE     VERSION   LABELS
ip-10-99-90-189.ap-south-1.compute.internal   Ready    <none>   3h13m   v1.15.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5ad.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=ap-south-1,failure-domain.beta.kubernetes.io/zone=ap-south-1a,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-99-90-189,kubernetes.io/os=linux,spotinst.io/node-lifecycle=spot

Thanks for the quick response. When is the planned date for the next release?

[pawanpraka1]

We release ZFS-LocalPV every month. The next release(0.7.0) is planned on May 15th. I have aleady raised the PR (#101) to fix this.

[pawanpraka1]

@pardhasm can you let us know your use case? How you want to use ZFS-LocalPV, what specific feature you are looking for?

[pardhasm]

@pawanpraka1 I'm planning to run stateful services like elasticsearch, percona and redis on my K8s cluster utilising all ZFS features.

[pawanpraka1]

@pardhasm, PR (#101) to fix this issue is merged. You can go ahead and try this out (zfs-driver:ci image) and see if this is working now. Please note that operator (https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml) is using zfs-drive:ci tag with image pull policy as ifnotpresent. So it will not pull the image again if it has already pulled zfs-driver:ci image before. You can change the imagepullpolicy (

zfs-localpv/deploy/zfs-operator.yaml

Line 1039 in 49dc997

imagePullPolicy: IfNotPresent

) to Always and try. There are two places in the yaml where zfs-driver:ci tag is used. We have to change the imagepullpolicy at both the places.

[pawanpraka1]

@pardhasm Let me know if you are facing any issue with this. This should work now.

[pawanpraka1]

@pardhasm we have released 0.7.0 version for ZFS-LocalPV toady. Please let us know if you are still facing this issue. Here is the 0.7.0 operator yaml

https://raw.githubusercontent.com/openebs/zfs-localpv/v0.7.x/deploy/zfs-operator.yaml

[pawanpraka1]

closing this, please reopen if this issue still persist.