Mark errors raised after a successful create - #740
Conversation
sdkCreate issues the create API call and then, for many resources, further calls that apply tags, attributes or associated sub-resources. When one of those fails the resource exists in AWS but sdkCreate still returns an error, and the reconciler removes the ACK finalizer because it cannot tell that apart from a create that never happened. The next reconciliation finds an existing resource with no finalizer and terminally conditions it as not managed by ACK, orphaning it. Defer a wrap once the create call has returned, so every error raised below it is marked as a post-create failure and the reconciler keeps the finalizer. Because the defer mutates the named return, it also covers the returns inside inlined hook code, which no hook has to be edited for. Place it after setStatusDefaults so ko is in scope, and backfill created from ko when a hook returned nil. Without that the identifiers from the create response are never persisted, and for a server-generated identifier the next reconciliation reads NotFound and creates a second resource. WrapPostCreateError only wraps AWS API errors, so ackerr.NotFound and the ackrequeue signals that hooks return here pass through unchanged. Requires a runtime release containing WrapPostCreateError. Issue: aws-controllers-k8s/community#2849
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: gustavodiaz7722 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@gustavodiaz7722: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
The controller-test failures here are the runtime dependency, not a defect in the change. Converting to draft until aws-controllers-k8s/runtime#268 is released. Every job that regenerates a controller and builds it fails, while the jobs that do not — I could not pull the job log text to quote it directly, but I reproduced exactly this locally: a regenerated ec2-controller compiles with a Sequence to land this:
Note runtime#268 is currently blocked by a separate, pre-existing CI break — mockery v2.53.3 cannot run under the Go 1.27.1 image — fixed in aws-controllers-k8s/runtime#269. |
Issue #2849
Depends on aws-controllers-k8s/runtime#268. Do not merge before that is
released — generated code calls
ackerr.WrapPostCreateError, so regeneratinga controller against a runtime without it fails to compile:
Description of changes:
sdkCreateissues the create API call and then, for many resources, furthercalls that apply tags, attributes or associated sub-resources. When one of those
fails the resource exists in AWS but
sdkCreatestill returns an error, and thereconciler removes the ACK finalizer because it cannot tell that apart from a
create that never happened. The next reconciliation finds an existing resource
with no finalizer and terminally conditions it as not managed by ACK, orphaning
it. runtime#268 has the full analysis.
This defers a wrap once the create call has returned, so every error raised
below it is marked as a post-create failure:
Three things about this shape are deliberate.
A defer rather than edits at each return. Scanning every generated
sdkCreatein the fleet, 41 of 277 resources can fail after a successfulcreate, and 28 of those return
nil, errfrom inlined hook code. Because thedefer mutates the named return, it covers those returns without any hook
template being edited.
Placed after
setStatusDefaultssokois in scope. That matters for thecreatedbackfill below. The region above it is output field mapping; every oneof the 41 real post-create failures sits at or after this point, including the
set_output_custom_method_nameblock and thesdk_create_post_set_outputhook.Backfilling
created. When a hook returnednil, generatedmanager.Createsubstitutesdesired, so the identifiers from the createresponse are never persisted. For a server-generated identifier the next
reconciliation reads
NotFoundand creates a second resource, leaking thefirst. Returning
koprevents that.WrapPostCreateErroronly wraps AWS API errors, so theackerr.NotFoundguards and
ackrequeuesignals that hooks return here pass through unchanged.Of the resources that return a requeue after a successful create, none carries
an AWS error inside it.
Not covered:
custom_implementationon the create operation short-circuits atthe top of
sdkCreate, before this defer is registered. Two resources use it —elasticache/Snapshot(CustomCreateSnapshot) andapigatewayv2/API(
customCreateApi) — and need the wrap applied by hand in their controllerrepos.
Testing
go build ./...and the fullgo test ./...suite pass.Regenerating ec2-controller produces the defer in all 20 resources that have a
create operation, with no other change beyond the expected
version.goandack-generate-metadata.yamltimestamps. Insecurity_group/sdk.goit landsahead of both the
requiredFieldsMissingForSGRuleguard anddeleteDefaultSecurityGroupRule, which is the call that triggers #2849. Theregenerated controller compiles against runtime#268.
The regenerated controller was then deployed to an EKS cluster and the
post-create failure injected two ways (an IAM deny of
ec2:RevokeSecurityGroupEgress, and an ingress rule referencing a non-existentsecurity group). The finalizer was retained, the condition was
ACK.Recoverablerather than terminal, exactly one security group was createdper CR, the resource self-healed once the permission was restored, and deleting
the CRs removed the security groups from AWS. Full before/after in runtime#268.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.