unable to create directory to provision new pv: mkdir /persistentvolumes/openshift-image-registry-registry-pvc-pvc-8617bc9a-0321-44d7-9192-5823df3e277f: permission denied

[tomazb]

Hello!

I added extra disk to helpernode and mount it at /exports, because I was under impression that all the nfs storage would be there.
Later I used "helpernodecheck nfs-setup" to setup nfs provisioner which did not work due to this error:

I0413 12:00:09.119057 1 controller.go:987] provision "openshift-image-registry/registry-pvc" class "nfs-storage-provisioner": started I0413 12:00:09.123036 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"openshift-image-registry", Name:"registry-pvc", UID:"8617bc9a-0321-44d7-9192-5823df3e277f", APIVersion:"v1", ResourceVersion:"325736", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "openshift-image-registry/registry-pvc" W0413 12:00:09.124249 1 controller.go:746] Retrying syncing claim "openshift-image-registry/registry-pvc" because failures 4 < threshold 15 E0413 12:00:09.124289 1 controller.go:761] error syncing claim "openshift-image-registry/registry-pvc": failed to provision volume with StorageClass "nfs-storage-provisioner": unable to create directory to provision new pv: mkdir /persistentvolumes/openshift-image-registry-registry-pvc-pvc-8617bc9a-0321-44d7-9192-5823df3e277f: permission denied I0413 12:00:09.124336 1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"openshift-image-registry", Name:"registry-pvc", UID:"8617bc9a-0321-44d7-9192-5823df3e277f", APIVersion:"v1", ResourceVersion:"325736", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "nfs-storage-provisioner": unable to create directory to provision new pv: mkdir /persistentvolumes/openshift-image-registry-registry-pvc-pvc-8617bc9a-0321-44d7-9192-5823df3e277f: permission denied

Where is this location defined? Should this be under /exports?

[christianh814]

@tomazb Hi!

So it looks like it's looking for /export specifically https://github.com/RedHatOfficial/ocp4-helpernode/blob/master/templates/nfs-provisioner-deployment.yaml.j2#L32

You can either edit the deployment oc edit deploy nfs-client-provisioner -n nfs-provisioner and change the path or remount the drive to /export

EDIT: You also may have to play with permissions/selinux if you're remounting.

[tomazb]

I have the /export on the appliance:

[root@helper ~]# ls -ldZ /export/
drwxr-xr-x. 2 root root system_u:object_r:usr_t:s0 6 Apr 12 19:54 /export/

[root@helper ~]# restorecon -Rv /export/

[root@helper ~]# ls -ldZ /export/
drwxr-xr-x. 2 root root system_u:object_r:usr_t:s0 6 Apr 12 19:54 /export/

[root@helper ~]# showmount -e localhost
Export list for localhost:
/export *

---

I only mounted an additional drive in the same place.

Nothing else was changed.

[christianh814]

So I just ran a test and I had no issues creating a PVC

# oc get pvc -A
NAMESPACE                  NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
openshift-image-registry   registry-pvc   Bound    pvc-09cc6439-5883-445a-bd0d-9f1f2663d40e   20Gi       RWX            nfs-storage-provisioner   3m2s

# ll /export/
total 0
drwxrwxrwx. 2 nobody nobody 6 Apr 14 09:02 openshift-image-registry-registry-pvc-pvc-09cc6439-5883-445a-bd0d-9f1f2663d40e

I did find one bug, and it's a documentation bug found when running helpernodecheck nfs-info so that'll be fixed in the next devel release.

You need to run

oc patch configs.imageregistry.operator.openshift.io cluster --type=json -p '[{"op": "remove", "path": "/spec/storage/emptyDir" }]'

Then run

oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"pvc":{ "claim": "registry-pvc"}}}}'

^ I'll add that to the info script.

But back to your issue...

One thing I did noteice is that your /export dir doesn't have the right ownership (nfsnobody for EL7 or nobody for EL 8)

# ll -d /export/
drwxrwxrwx. 3 nobody nobody 92 Apr 14 09:02 /export/

Try changing the ownership to either nfsnobody or nobody. You might want to chmod 777 for testing (this is insecure, but just do it to test it)

If that works I'll add that to the notes too

[tomazb]

[root@helper ~]# oc get pvc -A NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE openshift-image-registry registry-pvc Bound pvc-0a8e5e96-4d83-4f07-8830-6f2460899f73 20Gi RWX nfs-storage-provisioner 59s
Thanks! It worked.

I changed the owner to nobody and followed the steps you provided here.

[christianh814]

Thanks, I merged some changes to the devel branch too d39c3de