EBS volume resize fails: volume not found

[edmeister]

We recently set up a postgres cluster on AWS on a kubernetes cluster set up with Typhoon using this configuration:

kind: "postgresql"
apiVersion: "acid.zalan.do/v1"

metadata:
  name: "iot-historics"
  namespace: "postgres-operator"
  labels:
    team: iot

spec:
  teamId: "iot"
  postgresql:
    version: "12"
  numberOfInstances: 2
  enableMasterLoadBalancer: true
  enableReplicaLoadBalancer: true
  volume:
    size: "200Gi"
  users:
    iot-historics: []
  preparedDatabases:
    iot_historics:
      defaultUsers: true
      extensions:
        timescaledb: public
  
  resources:
    requests:
      cpu: 1000m
      memory: 2048Mi
    limits:
      cpu: 1000m
      memory: 2048Mi

Operator version is 1.5.0 installed with helm.

Our operator has an IAM role attached, granting full ec2 and ebs permissions (this is of course too broad, but we had permission issues during this test).

Since we hit the limit of the disk, we tried to expand the volume by changing the size in the manifest and reapplying it. Unfortunately this seems to fail. The operator logs show

time="2020-07-08T07:56:37Z" level=debug msg="syncing persistent volumes" cluster-name=postgres-operator/iot-historics pkg=cluster
time="2020-07-08T07:56:38Z" level=debug msg="updating persistent volume \"pvc-bae2f27e-ac44-42ba-a5f7-aa24aaa8becc\" to 300" cluster-name=postgres-operator/iot-historics pkg=cluster
time="2020-07-08T07:56:39Z" level=warning msg="error while syncing cluster state: could not sync persistent volumes: could not sync volumes: could not resize EBS volume \"vol-0b0407e7caa5e083a\": could not get information about the volume: InvalidVolume.NotFound: The volume 'vol-0b0407e7caa5e083a' does not exist.\n\tstatus code: 400, request id: 6804d969-3971-46a6-b0d3-66cda100a629" cluster-name=postgres-operator/iot-historics pkg=cluster

When we check the AWS console, the volume does exist.

When we update the PVC manually, the resize gets executed succesfully though.

Any idea what might be causing this?

[FxKu]

With IAM role attached you mean you have set it in the configuration? Does resizing not work at all?

Just recently we merged support to resize via changing the PVC, see #958. Maybe this could also help you in the future.

[FxKu]

Closing this one now, as with the new v1.6.0 release you can now use the K8s-native resize mode via PVCs.