[rds-controller] Finalizers keep telling DBCluster to remove availabilityZones

[lanandra]

Describe the bug
When I ran kubectl diff, finalizers will show that any availability zone that not defined in the manifest will be removed. Although previously it has been applied many times

finalizers:
   - finalizers.rds.services.k8s.aws/DBCluster
-  generation: 14
+  generation: 15
@@ -22,7 +22,6 @@
   availabilityZones:
   - us-east-2b
   - us-east-2c
-  - us-east-2a
   backupRetentionPeriod: 3
   copyTagsToSnapshot: false
   databaseName: my-database

Steps to reproduce

1. After check the diff, tried to apply the manifest
2. After manifest has been applied, tried to run the diff again
3. kubectl diff still show that finalizers in DBCluster will remove the same availability zone that not defined in the manifest

Expected outcome
Finalizers will not show the same deletion output in the diff after it has been applied

Environment

• Kubernetes version: 1.32
• Using EKS (yes/no), if so version? yes, v1.32.3-eks-bcf3d70
• AWS service targeted (S3, RDS, etc.): RDS

[github-actions]

Hello @lanandra 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[lanandra]

For additional context, below are some snippet from the configuration and actual resource in the cluster.

Configuration in manifest:

---
apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: my-database
  annotations:
    services.k8s.aws/deletion-policy: retain
spec:
  --redacted--
  engine: aurora-mysql
  engineVersion: "8.0.mysql_aurora.3.08.1"
  --redacted--
  availabilityZones:
    - us-east-2b
    - us-east-2c
 --redacted

Although in the actual resource yaml, it still show availability zone us-east-2a

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
--redacted--
spec:
  --redaxcted--
  availabilityZones:
  - us-east-2b
  - us-east-2c
  - us-east-2a
  --redacted--

Not sure if we need to remove/edit manually/directly again in the cluster?

[knottnt]

@lanandra It looks like the behavior you're running into is related to this issue. The ModifyDbClusterInput struct doesn't support setting availability zones so the RDS controller can't send the new spec values to AWS. However, it looks like the RDS controller isn't treating DB Cluster availability zones as immutable and setting an ACK condition to indicate the issue.

[lanandra]

@lanandra It looks like the behavior you're running into is related to this issue. The ModifyDbClusterInput struct doesn't support setting availability zones so the RDS controller can't send the new spec values to AWS. However, it looks like the RDS controller isn't treating DB Cluster availability zones as immutable and setting an ACK condition to indicate the issue.

So currently, we would say that it is intended? until there is some change or update in RDS controller so that it can send the new spec values as mentioned above

[knottnt]

Yes, it's not unexpected that the controller isn't updating the availability zones after the initial creation of the DB cluster. The reason the new spec values for availabilityZones aren't sent is that the RDS API's ModifyDBCluster endpoint doesn't expose a parameter to change the availability zones.

There's is some unexpected behavior here though. The controller should be setting an ACK condition on the DB Cluster resource indicating that the availability zones can't be changed.

[lanandra]

Yes, it's not unexpected that the controller isn't updating the availability zones after the initial creation of the DB cluster. The reason the new spec values for availabilityZones aren't sent is that the RDS API's ModifyDBCluster endpoint doesn't expose a parameter to change the availability zones.

There's is some unexpected behavior here though. The controller should be setting an ACK condition on the DB Cluster resource indicating that the availability zones can't be changed.

Assuming in our DBCluster configuration, we only set the AZ to two of the AZ instead of all the AZ in region

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
--redacted--
spec:
  --redacted--
  availabilityZones:
    - us-east-2b
    - us-east-2c
  --redacted--

We actually need to define all the AZ and if we just want to use two of the AZ, then it will be reflected from the configuration of subnet that define in spec.dbSubnetGroupName?

dbSubnetGroupName: my-private-subnet

[knottnt]

@lanandra Testing creation of an Aurora DB cluster with only two AZs set and a dbSubnetGroupName with subnets in only two AZs, like the sample below, I still see the DB cluster report three AZs after creation due to RDS setting a defaulted value.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: sample-aurora-cluster
  annotations:
    services.k8s.aws/deletion-policy: retain
spec:
  engine: aurora-mysql
  engineVersion: "8.0.mysql_aurora.3.08.1"
  dbClusterIdentifier: sample-aurora-cluster
  dbSubnetGroupName: "test-subnet-group" // subnet group with only two AZs (us-east-2a and us-east-2b)
  availabilityZones:
    - us-east-2a
    - us-east-2b
 ...

However, when attempting to create a DB instance in the DB cluster with an AZ without a subnet in the dbSubnetGroupName I see the creation of the instance fail with the following error.

operation error RDS: CreateDBInstance, https response error StatusCode: 400, RequestID: f1ab5076-5ac2-421a-8875-47105ce77497, InvalidVPCNetworkStateFault: You can't create a db.t3.medium database instance because there are no subnets with available IP addresses in the specified Availability Zone us-east-2c. Create a new subnet in us-east-2c and target that Availability Zone again, or retry the launch with no availability zone, or choose from these Availability Zones: us-east-2a, us-east-2b.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: sample-mysql-instance
  annotations:
    services.k8s.aws/deletion-policy: retain
spec:
  availabilityZone: us-east-2c
  dbInstanceIdentifier: sample-mysql-instance
  dbInstanceClass: db.t3.medium
  engine: aurora-mysql
  dbClusterIdentifier: sample-aurora-cluster

[lanandra]

@lanandra Testing creation of an Aurora DB cluster with only two AZs set and a dbSubnetGroupName with subnets in only two AZs, like the sample below, I still see the DB cluster report three AZs after creation due to RDS setting a defaulted value.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
name: sample-aurora-cluster
annotations:
services.k8s.aws/deletion-policy: retain
spec:
engine: aurora-mysql
engineVersion: "8.0.mysql_aurora.3.08.1"
dbClusterIdentifier: sample-aurora-cluster
dbSubnetGroupName: "test-subnet-group" // subnet group with only two AZs (us-east-2a and us-east-2b)
availabilityZones:
- us-east-2a
- us-east-2b
...
However, when attempting to create a DB instance in the DB cluster with an AZ without a subnet in the dbSubnetGroupName I see the creation of the instance fail with the following error.

operation error RDS: CreateDBInstance, https response error StatusCode: 400, RequestID: f1ab5076-5ac2-421a-8875-47105ce77497, InvalidVPCNetworkStateFault: You can't create a db.t3.medium database instance because there are no subnets with available IP addresses in the specified Availability Zone us-east-2c. Create a new subnet in us-east-2c and target that Availability Zone again, or retry the launch with no availability zone, or choose from these Availability Zones: us-east-2a, us-east-2b.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
name: sample-mysql-instance
annotations:
services.k8s.aws/deletion-policy: retain
spec:
availabilityZone: us-east-2c
dbInstanceIdentifier: sample-mysql-instance
dbInstanceClass: db.t3.medium
engine: aurora-mysql
dbClusterIdentifier: sample-aurora-cluster

What if we define spec.availabilityZones with all three AZs? but in the dbSubnetGroupName, it only consist of two Azs

[knottnt]

@lanandra Trying to create a DB cluster with three regions set and a dbSubnetGroupName with only two AZs I see the create fail with the below error message.

InvalidVPCNetworkStateFault: Availability zone '[us-east-2c]' is unavailable in this region, please choose another zone set.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: sample-aurora-cluster
  annotations:
    services.k8s.aws/deletion-policy: retain
spec:
  engine: aurora-mysql
  engineVersion: "8.0.mysql_aurora.3.08.1"
  dbClusterIdentifier: sample-aurora-cluster
  dbSubnetGroupName: "test-subnet-group" // subnet group with only two AZs (us-east-2a and us-east-2b)
  availabilityZones:
    - us-east-2a
    - us-east-2b
    - us-east-2c

[lanandra]

@lanandra Trying to create a DB cluster with three regions set and a dbSubnetGroupName with only two AZs I see the create fail with the below error message.

InvalidVPCNetworkStateFault: Availability zone '[us-east-2c]' is unavailable in this region, please choose another zone set.

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
name: sample-aurora-cluster
annotations:
services.k8s.aws/deletion-policy: retain
spec:
engine: aurora-mysql
engineVersion: "8.0.mysql_aurora.3.08.1"
dbClusterIdentifier: sample-aurora-cluster
dbSubnetGroupName: "test-subnet-group" // subnet group with only two AZs (us-east-2a and us-east-2b)
availabilityZones:
- us-east-2a
- us-east-2b
- us-east-2c

I see, so set all the AZs also not needed when we only have two AZs defined in DB subnet group.
The issue circle back caused by the struct in RDS as you mentioned above. Thanks for the insight 👍

[knottnt]

Created a new issue to track the lack of immutability for DBCluster.AvailabilityZones.

[knottnt]

@lanandra: Happy to help! I'm going to close this issue in favor of the new issue linked in my previous comment. If you have any other questions or issues related to this feel free to re-open this issue or create a new one.