v1.2.22 release of dynamodb-controller deleted preconfigured tableReplica

[mattzech]

Describe the bug
We had a team who created a DynamoDB Table in production last year which was then configured with a replica in us-east-1 via AWS (since this was not supported in Table spec yet). When the new dynamodb controller release went out to this cluster, the generated Table resource had its spec.tableReplicas set to null, which then deleted the pre-created replica in us-east-1, causing downtime for this application
Steps to reproduce

• While using v1.2.22, adopt a DynamoDB table with a table replica already configured in another region
• Upgrade to v1.2.22 and check AWS for replica table (observed deletion in cloudtrail)

Expected outcome
Somehow there needs to be better protection for Tables where this field is not set

Environment

• Kubernetes version v1.29.11
• Using EKS (yes/no), if so version? no
• AWS service targeted (S3, RDS, etc.) DynamoDB

[github-actions]

Hello @mattzech 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[mattzech]

Some logs from around that time:

2025-03-26T20:27:02.506Z	DEBUG	cache.namespace	created namespace	{"name": "application-namespace"}
2025-03-26T20:27:27.878Z	INFO	adoption.adopted-reconciler	starting adoption reconciliation	{"target_group": "dynamodb.services.k8s.aws", "target_kind": "Table", "namespace": "application-namespace", "name": "application-namespace-dynamodb", "generation": 1}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	> r.Sync	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	>> r.resetConditions	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	<< r.resetConditions	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	>> rm.ResolveReferences	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	<< rm.ResolveReferences	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	>> rm.EnsureTags	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	<< rm.EnsureTags	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	>> rm.ReadOne	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.024Z	DEBUG	ackrt	>>> rm.sdkFind	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.256Z	DEBUG	ackrt	>>>> rm.setResourceAdditionalFields	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.256Z	DEBUG	ackrt	>>>>> rm.getResourceTagsPagesWithContext	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.263Z	DEBUG	ackrt	<<<<< rm.getResourceTagsPagesWithContext	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.263Z	DEBUG	ackrt	>>>>> rm.getResourceTTLWithContext	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.275Z	DEBUG	ackrt	<<<<< rm.getResourceTTLWithContext	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.275Z	DEBUG	ackrt	>>>>> rm.getResourcePointInTimeRecoveryWithContext	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	<<<<< rm.getResourcePointInTimeRecoveryWithContext	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	<<<< rm.setResourceAdditionalFields	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	<<< rm.sdkFind	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	<< rm.ReadOne	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	>> r.updateResource	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	INFO	ackrt	desired resource state has changed	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2, "diff": [{"Path":{"Parts":["Spec","TableReplicas"]},"A":null,"B":[{"globalSecondaryIndexes":[{"indexName":"gs1"}],"regionName":"us-east-1","tableClassOverride":"STANDARD"}]}]}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	>>> rm.Update	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	>>>> rm.customUpdateTable	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	>>>>> rm.syncReplicas	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	>>>>>> rm.newUpdateTableReplicaUpdatesOneAtATimePayload	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2}
2025-03-26T20:27:28.294Z	DEBUG	ackrt	deleting replica in region	{"kind": "Table", "namespace": "application-namespace", "name": "application-namespace", "account": "1112223334445", "role": "arn:aws:iam::1112223334445:role/openshift-ack-controllers", "region": "eu-west-1", "is_adopted": true, "generation": 2, "table": "application-config-service-prod-live", "region": "us-east-1"}

[rushmash91]

Hi @mattzech ,

Thank you for reporting this issue. I wanted to let you know that the syncReplicas had been moved to v1.3.0 minor release.

If you update to v1.2.23, the controller will no longer attempt to sync table replicas, which should prevent any unintended replicas syncing. The complete table replica management has been moved to the v1.3.0 minor release.

For now, you can upgrade to v1.2.23 and your existing table replicas will remain untouched.
Please let us know if you have any questions or concerns.

[a-hilaly]

closing as 1.3.0 is released now