Skip to content

[RDS controller]: constant loop of changing the db in the cloud #2695

@gecube

Description

@gecube

We are using ACK Controllers to manage our database resources, and we are observing a persistent issue: after a DBinstance is created, it regularly falls back into the configuring cycle.
In the controller logs we constantly see messages like:

{"level":"info","ts":"2025-11-17T08:05:49.709Z","logger":"ackrt","msg":"desired resource state has changed","kind":"DBInstance","namespace":"infra-production","name":"database-2025-11-09","account":"966321756598","role":"arn:aws:iam::966321756598:role/ack-ec2-controller-tooling","region":"eu-west-2","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","AutoMinorVersionUpgrade"]},"A":null,"B":true},{"Path":{"Parts":["Spec","CACertificateIdentifier"]},"A":null,"B":"rds-ca-rsa2048-g1"},{"Path":{"Parts":["Spec","LicenseModel"]},"A":null,"B":"general-public-license"},{"Path":{"Parts":["Spec","MultiAZ"]},"A":null,"B":false},{"Path":{"Parts":["Spec","PreferredBackupWindow"]},"A":null,"B":"00:56-01:26"},{"Path":{"Parts":["Spec","PreferredMaintenanceWindow"]},"A":null,"B":"sat:23:48-sun:00:18"},{"Path":{"Parts":["Spec","StorageEncrypted"]},"A":null,"B":false},{"Path":{"Parts":["Spec","StorageThroughput"]},"A":null,"B":0},{"Path":{"Parts":["Spec","StorageType"]},"A":null,"B":"gp2"}]}

This behavior clearly doesn’t seem normal. While the controller repeatedly attempts to “configure” the instance, the database may become unavailable — which is unacceptable in production.
It would be very helpful to have guidance or a built-in mechanism to prevent this repeating reconfiguration loop.
Additionally, we see a related issue when AutoMinorVersionUpgrade is enabled:
the cloud database automatically upgrades to a newer minor version, while the version in the ACK manifest remains unchanged. After that, any further changes applied via the manifest get stuck, because the version mismatch prevents updates from proceeding.
Could you please clarify whether this is expected behavior, and whether there are recommended practices to avoid these situations?

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.service/rdsIndicates issues or PRs that are related to rds-controller.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions