openstack: cannot provision volumes when multiple AZs available

[teutonet]

Version

$ openshift-install version
openshift-install v4.2.10
built from commit 6ed04f65b0f6a1e11f10afe658465ba8195ac459
release image quay.io/openshift-release-dev/ocp-release@sha256:dc2e38fb00085d6b7f722475f8b7b758a0cb3a02ba42d9acf8a8298a6d510d9c

Platform:

openstack

What happened?

Installation of a openshift cluster on openstack was successful.

Creating PersistentVolumes using openstack cinder fails:

Failed to provision volume with StorageClass "standard": failed to create a 1 GB volume: 
Bad request with: [POST https://REDACTED:8776/v3/REDACTED/volumes], error message: {"badRequest": {"message": "Availability zone 'Zone2' is invalid.", "code": 400}}

This is due to the openstack compute service has multiple availability zones such as Zone1 and Zone2 but the openstack blockstorage service has only a single AZ in this case nova.

Once the AZ is set on the storage class to nova the creation of volumes works as expected, but attaching the volume to a pod does not work as the scheduler tries to ensure that the volume AZ matches the workers AZ and fails:

0/6 nodes are available: 3 node(s) had taints that the pod didn't tolerate, 3 node(s) had volume node affinity conflict.
pv "pvc-6631ed79-20dc-11ea-9d14-fa163e4d02f1" node affinity doesn't match node "ocp-jbfgw-worker-nvc5p": No matching NodeSelectorTerms

According to an older openshift 3 guide setting ignore-volume-az=yes in the cloud provider config and rebooting the whole cluster should fix this issue.

Unfortunately adding the property to the cloud provider config and rebooting the whole cluster did not solve this issue.

Only relabelling all Nodes to match the openstack blockstorage service AZ worked.

What you expected to happen?

Installation of openshift on openstack with multiple compute AZs and single volume AZ should include the ignore-volume-az property in the cloud provider config, e. g.:

cloud.conf:

[Global]
auth-url           = 
....
domain-name = Default

[BlockStorage]
ignore-volume-az = yes

And the default storage class should include the volume AZ, e. g.:

StorageClass standard:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
parameters:
  availability: nova
provisioner: kubernetes.io/cinder
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

How to reproduce it (as minimally and precisely as possible)?

Install openshift on a openstack cluster with mutliple compute AZs and a single volume AZ.
Create a Pod using a PersistentVolume:

test.yaml:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
        claimName: task-pv-claim
  containers:
    - name: task-pv-container
      image: alpine
      command: ["/bin/sh", "-c", "--"]
      args: [ "while true; do sleep 30; done;" ]
      volumeMounts:
        - mountPath: "/data"
          name: task-pv-storage

Both the PV and the Pod should be created successfully and the Pod should be in running state.

$ oc create -n teuto -f test.yaml
persistentvolumeclaim/task-pv-claim created
pod/task-pv-pod created

$ oc get pvc,pv,pod  -n teuto
NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
persistentvolumeclaim/task-pv-claim   Bound    pvc-7ac96859-21a8-11ea-9ef4-fa163e117977   3Gi        RWO            standard         61s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS     REASON   AGE
persistentvolume/pvc-7ac96859-21a8-11ea-9ef4-fa163e117977   3Gi        RWO            Delete           Bound    teuto/task-pv-claim        standard                  60s

NAME              READY   STATUS    RESTARTS   AGE
pod/task-pv-pod   1/1     Running   0          61s

References

• OpenShift 3 Guide for Multiple Compute AZs and Single Volume AZ

[pierreprinetti]

/label platform/openstack
/close

OpenShift 4 support for Availability Zones is planned, but not implemented yet. We hope to be able to bring it soon.

Closing as duplicate of #2391; feel free to comment further or reopen if needed.

[openshift-ci-robot]

@pierreprinetti: Closing this issue.

In response to this:

/label platform/openstack
/close

OpenShift 4 support for multiple Availability Zones is planned, but not implemented yet. We hope to be able to bring it soon.

Closing as duplicate of #2391; feel free to comment further or reopen if needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MartijnStraatman]

Why was this feature which is available in 3.11 not implemented from the beginning?
I mean the "ignore_volume_az" option. We are also using this. Migrating to 4.x will take even longer now.

[MartijnStraatman]

@teutonet How dit you relabel the nodes. I adjusted this label on each node:

failure-domain.beta.kubernetes.io/zone: nova

But somehow pods stay unschedulabe with the same message. I also change the scheduler-policy by removing the "NoVolumeZoneConflict" predicate. But no luck.

[teutonet]

@MartijnStraatman I did the following steps in order:

1. added ignore_volume_az to cloud provider config and rebooted, with no effect.
2. changed Node label failure-domain.beta.kubernetes.io/zone to nova
3. created a new StorageClass with parameter for availability zone:

...
parameters:
  availability: nova

4. create PV using the new StorageClass

In 99% of PV creation it worked without any issue, but sometimes the PersistentVolume has the old AZ labels, deleting the labels helped to fix the issue.

Hope this works for you!

[teutonet]

@MartijnStraatman on a fresh installed cluster I could workaround this issue by just relabeling the nodes to Availability Zone nova using the following command:

$ oc get nodes -ojsonpath='{.items[*].metadata.name}' | \
   xargs -n 1 \
      oc patch node --type='json' \
           -p='[{"op":"replace", "path":"/metadata/labels/failure-domain.beta.kubernetes.io~1zone", "value":"nova"}]'
node/cluster1-1337-master-0 patched
node/cluster1-1337-master-1 patched
node/cluster1-1337-master-2 patched
node/cluster1-1337-worker-abc patched
node/cluster1-1337-worker-def patched
node/cluster1-1337-worker-ghi patched

[MartijnStraatman]

@teutonet thx. Will try that. I changed the default scheduler by removing the

NoVolumeZoneConflict Predicate