Using Kubernetes v1.20.0, getting "unexpected error getting claim reference: selfLink was empty, can't make reference"

[nealman13]

Using Kubernetes v1.20.0

Attempting to create pvc's, they all remain in "pending" status

Doing a kubectl logs nfs-client-provisioner gives this:

I1210 14:42:01.396466 1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs
I1210 14:42:01.396534 1 controller.go:631] Starting provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.496922 1 controller.go:680] Started provisioner controller fuseim.pri/ifs_nfs-client-provisioner-64b7476494-p4fcm_dcfca333-3af5-11eb-8248-5aed4ceb7af7!
I1210 14:42:01.497152 1 controller.go:987] provision "default/pvc1" class "managed-nfs-storage": started
I1210 14:42:01.497157 1 controller.go:987] provision "default/test-claim" class "managed-nfs-storage": started
E1210 14:42:01.500487 1 controller.go:1004] provision "default/pvc1" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference
E1210 14:42:01.500502 1 controller.go:1004] provision "default/test-claim" class "managed-nfs-storage": unexpected error getting claim reference: selfLink was empty, can't make reference

selfLink has been disabled in v1.20.0
kubernetes/enhancements#1164

[fangedhex]

I can confirm that I have this problem too on my new cluster using Kubernetes v1.20 with those same logs/errors when creating a test pvc.

[nealman13]

Current workaround is to edit /etc/kubernetes/manifests/kube-apiserver.yaml

Under here:

spec:
  containers:
  - command:
    - kube-apiserver

Add this line:
- --feature-gates=RemoveSelfLink=false

The do this:
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

(I had to do it twice to get it to work)

[petermicuch]

Yep, exactly the same issue. In k8s 1.20 they have removed selfLink as per release notes.

Adding selfLink manually does not help, as that is removed by API server immediatelly.

[fangedhex]

I used @nealman13's workaround until a fix is made and it works now. Thanks !

[petermicuch]

Just to add to what @nealman13 provided. Before first leader election log, there is this error:

E1210 15:41:30.047572       1 event.go:259] Could not construct reference to: '&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"nfss", SelfLink:"", UID:"75df6842-e437-4604-9034-2d4d932373be", ResourceVersion:"19431", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63743203519, loc:((nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"control-plane.alpha.kubernetes.io/leader":"{\"holderIdentity\"eb-af59-dac986fde3f6\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2020-12-10T15:41:30Z\",\"renewTime\":\"2020-12-10T15:41:30Z\",\"leaderTransitions\":1}"}, OwnerReference)(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]v1.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'Le2d51-3afe-11eb-af59-dac986fde3f6 became leader'
I1210 15:41:30.047754       1 leaderelection.go:194] successfully acquired lease default/fuseim.pri-ifs

[petermicuch]

I think the problem is in using the old client-go by this replace, but I am not 100% sure if that is the only legacy we pull here.

Same problem was discussed in #94660 and this was suggested solution.

[james-d-elliott]

Just a note about /etc/kubernetes/manifests/kube-apiserver.yaml. When running the control-plane as an in-cluster service you should not need to apply changes to this file ever (via kubectl apply). As soon as you save the change to disk the apiserver pods will detect it and should conduct a Rolling Update. You can watch the progress with watch kubectl get pods --namespace kube-system. You can confirm the behavior after deleting the provisioner pod and letting it restart.

[iming0319]

The workaround will no longer work after 1.20.
"selfLink is a URL representing this object. Populated by the system. Read-only. DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release."

[zaneclaes]

The "fix" now causes a CrashLoopBackoff on kube-apiserver, per @iming0319 .
Is there any way to get NFS working with Kubernetes 1.20.1?

[yonatankahana]

The "fix" now causes a CrashLoopBackoff on kube-apiserver, per @iming0319 .
Is there any way to get NFS working with Kubernetes 1.20.1?

have you tried #26 ?

[zaneclaes]

have you tried #26 ?

It does not appear to be merged. So, no, I have not pulled and built that PR (yet).

[groundhog2k]

@zaneclaes: I created a fork of nfs-subdir-external-provisioner, merged the PR from @petermicuch and adapted the action workflow to create an automated build + multi-arch. docker image - try v3.2.0 of https://hub.docker.com/r/groundhog2k/nfs-subdir-external-provisioner/tags?page=1&ordering=last_updated - It should work for Kubernetes >= V1.20.x.

[olcayseker]

I have tried your fork with feature-gate trick but pvc stucks in pending state. Kubernetes version is 1.20.0 log message is;

Waiting for a volume to be created, either by external provisioner "cluster.local/nfs-client-provisioner" or manually created by system administrator

[groundhog2k]

@olcayseker: I don't know which fork you mean, but if you are talking about https://github.com/groundhog2k/nfs-subdir-external-provisioner, i can say it's definetly working. I tested in on ARM64 and AMD64/x86 with and without feature-gate switch and using it since 15 days on my mixed-arch. cluster without any problems. Have you looked at the logs of the nfs-provisioner pod when PVC is in pending state?

[azui007]

Current workaround is to edit /etc/kubernetes/manifests/kube-apiserver.yaml

Under here:

spec:
  containers:
  - command:
    - kube-apiserver

Add this line:

• --feature-gates=RemoveSelfLink=false

The do this:
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml
kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

(I had to do it twice to get it to work)

it work for me ,thank you

[petermicuch]

No more workarounds please :-) - this issue was closed for a reason guys! Just use new available docker image: gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0 and it works fine. Do not edit kube-apiserver.yaml, there is no need to.

[cleverlzc]

the same error, trying

[abdennour]

same error

class "nfs-client": unexpected error getting claim reference: selfLink was empty, can't make reference

my env:

• k8s: 1.20
• helm chart stable/nfs-client-provisioner 1.2.8 ( app versopn 3.1.0)

[petermicuch]

@abdennour - you should read the comments above. This issue is closed and there is new helm chart and new container image existing that fixed this issue. Please try to update with helm.

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

:~$ helm search repo nfs
NAME                                                    CHART VERSION   APP VERSION     DESCRIPTION
nfs-subdir-external-provisioner/nfs-subdir-exte...      4.0.14          4.0.2           nfs-subdir-external-provisioner is an automatic...

[abdennour]

Thank you @petermicuch
It's working BUT I think it requires a doc for migration.
So far there are 2 steps before applying helm:

1. if you are using private/proxy registry make sure to point to the new image in gcr instead of the old one in quay.
2. delete MANUALLY 2 resources : the storageclass & the deployment

Then run helm apply

[petermicuch]

@abdennour - feel free to create PR with your migration description, I think maintainers will accept it if it adds value to the community. But please create separate feature request for that one. Thank you.

[PhillipDowney]

@olcayseker: I don't know which fork you mean, but if you are talking about https://github.com/groundhog2k/nfs-subdir-external-provisioner, i can say it's definetly working. I tested in on ARM64 and AMD64/x86 with and without feature-gate switch and using it since 15 days on my mixed-arch. cluster without any problems. Have you looked at the logs of the nfs-provisioner pod when PVC is in pending state?

Worked for me also , Storage class definition simpler as well

[finance-dataspider]

@zaneclaes: I created a fork of nfs-subdir-external-provisioner, merged the PR from @petermicuch and adapted the action workflow to create an automated build + multi-arch. docker image - try v3.2.0 of https://hub.docker.com/r/groundhog2k/nfs-subdir-external-provisioner/tags?page=1&ordering=last_updated - It should work for Kubernetes >= V1.20.x.

this works! thanks a lot!!!!

[ashkarstudy]

@petermicuch
Thank you. It worked on 1.27.0

[lIlIlllllmeng]

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.100.100:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.100.100
    - --allow-privileged=true
    - --audit-log-format=json
    - --audit-log-maxage=7
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/kubernetes/audit.log
    - --audit-policy-file=/etc/kubernetes/audit-policy.yml
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-aggregator-routing=true
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --feature-gates=
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/22
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=RemoveSelfLink=false

The connection to the server apiserver.cluster.local:6443 was refused - did you specify the right host or port?

[devopstraining99]

Current workaround is to edit /etc/kubernetes/manifests/kube-apiserver.yaml

Under here:

spec:
  containers:
  - command:
    - kube-apiserver

Add this line: - --feature-gates=RemoveSelfLink=false

The do this: kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

(I had to do it twice to get it to work)

This solutions doesn't work in kubernetes > 1.26

I updated the image to - gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0

and it worked thanks to @petermicuch for the solutions

try this @lIlIlllllmeng