ACK-RDS DBCluster adds invalid AllocatedStorage to aurora cluster

[devonhk]

Describe the bug
The DBCluster resource always adds a spec.allocatedStorage: 1 when none is specified. The controller then fails to create aurora-postgresql type clusters since the allocatedStorage field isn't supported.

Steps to reproduce

Create a simple DBCluster:

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: test-db-instance
spec:
  autoMinorVersionUpgrade: true
  databaseName: test_db
  dbClusterIdentifier: test-db-instance
  dbClusterParameterGroupName: default.aurora-postgresql16
  dbSubnetGroupName: test-db-instance
  deletionProtection: true
  enableCloudwatchLogsExports: []
  enableIAMDatabaseAuthentication: false
  enablePerformanceInsights: false
  engine: aurora-postgresql
  engineVersion: '16.6'
  masterUserPassword:
    key: DB_PASSWORD
    name: test-db-instance
  masterUsername: postgres
  vpcSecurityGroupIDs:
    - sg-CHANGE_ME

Expected outcome

• The ACK-RDS controller shouldn't mutate the DBCluster resource's spec.allocatedStorage field.
• The DBCluster should be created.

Actual outcome

Here's the live DBCluster resource (I redacted fields that weren't relevant).
We can see that spec.allocatedStorage was set to 1 and that the DBCluster creation failed since that field was included in the payload sent to AWS.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: test-db-instance
spec:
  allocatedStorage: 1
[...]
status:
[...]
  conditions:
    - message: "InvalidParameterCombination: AllocatedStorage isn't supported for DB engine aurora-postgresql.\n\tstatus code: 400, request id: [...]"
      status: 'True'
      type: ACK.Terminal
[...]

Environment

• Kubernetes version: 1.31
• Using EKS (yes/no), if so version?: eks.21
• AWS service targeted (S3, RDS, etc.): RDS

[github-actions]

Hello @devonhk 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[rushmash91]

Thanks for reporting this, @devonhk!

You're correct, the controller is incorrectly adding spec.allocatedStorage: 1 to the DBCluster resource for Aurora engine types.

This happens because the AWS RDS API itself always returns AllocatedStorage: 1 in its response for Aurora clusters (since their storage isn't fixed). The ACK controller code currently copies this value from the API response back into the Kubernetes resource's spec.

https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_ClusterPendingModifiedValues.html

Will work on a patch to prevent the controller from writing AllocatedStorage back to the spec when the engine is Aurora.

[devonhk]

Will work on a patch to prevent the controller from writing AllocatedStorage back to the spec when the engine is Aurora.

Thanks for the quick reply 🙏.

I did some additional testing and got this error message:

    - message: "InvalidParameterCombination: AllocatedStorage isn't supported for DB engine aurora-postgresql.\n\tstatus code: 400, request id: [...]"

I think it might be related to a race condition. I checked my cluster and we have a few aurora-postresql DBClusters that have successfully been created with spec.allocatedStorage: 1.

So I patched the DBCluster that had originally failed, by removing the spec.allocatedStorage property and it successfully created the RDS cluster.

So it seems like most of the time aurora DBClusters can be created. Not sure if this info helps with your bugfix 🤷‍♂.

[rushmash91]

Hey @devonhk, Do you mind sharing the manifest that worked for you vs the one that leads to the terminal condition? Want to understand what you mean by race condition.

While I do see the allocatedStorage appear, I am not encountering any terminal condition while creating the cluster.

[devonhk]

They are the same manifest.

The manifest in the initial bug report was the one that returned a 400 response.

All I did to resolve the terminal condition was to kubectl edit the resource and remove spec.allocatedStorage.

If you try creating tens of DBClusters you might be able to reproduce the bug.

We have a few DBCluster resources in our cluster, but this was the first time I encountered this type of creation error, which is why I suggested there might be a race condition.

[rushmash91]

Thank you for the detailed explanation! We might need to remove the field and test the race condition.

cc @a-hilaly @michaelhtm

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale