Informational: AWS us-east-1 EC2 Launch Template API degradation (2026-07-06) causing RequestLimitExceeded and stuck NodeClaims

[rob-howie-depop]

Purpose

Informational, for operators hitting sudden RequestLimitExceeded / stuck NodeClaims today (2026-07-06). This is not a Karpenter bug: there is an active AWS-side EC2 Launch Template API issue in us-east-1, and Karpenter's symptoms are distinctive enough that people will land here looking for answers (see the fresh report on #7961 today). Fine to close once AWS resolves the event.

What is happening

AWS Health Dashboard (EC2, us-east-1, account-specific: No), posted 05:45 PDT / 12:45 UTC:

Increased Launch Template API Error Rates
We are investigating increased error rates when calling EC2 Launch Template APIs in US-EAST-1 Region. During this time, affected customers may experience errors when creating, modifying, or referencing launch templates. Other AWS services that rely on launch templates may also be impacted.

Karpenter resolves launch templates on every node launch, so it is directly in the blast radius.

We observed the actual impact starting well before AWS's acknowledgement, across 6 independent AWS accounts (one EKS cluster each, all us-east-1, Karpenter v1.12.1, EKS 1.36):

• ~09:51 UTC: HTTP 500 InternalError responses on DescribeLaunchTemplates / CreateLaunchTemplate / DeleteLaunchTemplate in all accounts (also observed a shorter precursor burst on 2026-07-04 19:27-20:25 UTC)
• from ~10:06 UTC: HTTP 503 RequestLimitExceeded load-shedding on the same API family
• sibling EC2 APIs (DescribeImages, DescribeInstances, CreateFleet when reached) remained healthy throughout

Symptom signature in Karpenter

NodeClaims stuck, then churning on the registration timeout:

Launched=Unknown   RequestLimitExceeded: Error getting launch template configs: Request limit exceeded
Registered=Unknown NodeNotFound: Node not registered with cluster

Controller log:

"Reconciler error","controller":"nodeclaim.lifecycle",...,"error":"launching nodeclaim, creating instance,
creating nodeclaim, getting launch template configs, getting launch templates, describing launch templates,
operation error EC2: DescribeLaunchTemplates, exceeded maximum number of attempts, 3,
https response error StatusCode: 503, ... api error RequestLimitExceeded: Request limit exceeded."

Then "terminating due to timeout" reason=launch_timeout timeout=5m0s, NodeClaim deleted, replacement created immediately, repeat (~5.5 min cadence). NodePool status shows NodeRegistrationHealthy=False with the same message. Only clusters with pending-pod demand visibly break; idle clusters just show elevated SDK retry counts (karpenter_aws_sdk_go_request_attempt_total minus karpenter_aws_sdk_go_request_total).

How to confirm it is this outage and not your configuration

1. A single fresh CLI call is throttled even from an idle account: aws ec2 describe-launch-templates --region us-east-1 returns RequestLimitExceeded (for us it failed even with AWS_RETRY_MODE=adaptive AWS_MAX_ATTEMPTS=10), while aws ec2 describe-images / describe-instances succeed instantly.
2. CloudTrail shows Server.InternalError (errorCode) on a large fraction of launch template calls, including from AWS service-linked roles (we saw AWSServiceRoleForConfig affected), at unremarkable call volumes.
3. Multiple unrelated AWS accounts degrade within minutes of each other.
4. The event appears on the AWS Health Dashboard (it is marked account-specific: No).

Mitigation notes

• There is no client-side fix; recovery is on AWS. Track the Health Dashboard event.
• The NodeClaim churn loop is self-limiting (one claim in flight per starving NodePool; the 5m launch_timeout garbage-collects stuck claims), and it stops on its own when the API recovers or pending demand drains. We did not need to scale Karpenter down or delete NodeClaims.
• Do not chase internal culprits (deploys, noisy neighbours, quota changes) if the probe above reproduces; we burned time on that before CloudTrail attribution ruled everything internal out.
• Optional hardening that reduces retry pressure but does not fix the outage: AWS_RETRY_MODE=adaptive on the controller (honored via LoadDefaultConfig), and batcher tuning (batchMaxDuration / batchIdleDuration).
• Related structural issues that shape behavior under this kind of throttling: CreateFleet batcher amplifies rate limit failures and lacks internal throttle protection #9175 (CreateFleet batcher amplifies rate limit failures), NodeClaims stuck in Launched: Unknown after transient throttle recover slowly kubernetes-sigs/karpenter#3034 (NodeClaims stuck Launched: Unknown after transient throttle), Karpenter failed to get rate limit token #7961 (client-side retry-quota exhaustion; has a same-day report of this event), Getting Client.InvalidLaunchTemplateName.NotFoundException For Automatically Deleted Launch Template #7941 (launch template delete-on-cache-eviction).

Environment

• Karpenter v1.12.1 (chart 1.12.1), EKS 1.36, us-east-1, 6 clusters in 6 independent AWS accounts
• Impact window so far: 09:51 UTC 2026-07-06 to ongoing at time of filing

[shearn89]

Thanks for writing this up, I've spent a couple hours poking around and trying to work out why I keep getting paged for US environments 😁. Glad I'm not the only one!

[TheRubick]

Seems like to be a global issue: https://health.aws.amazon.com/health/status

Credits to my senior colleague: https://github.com/Mais316 :)

[rob-howie-depop]

Thanks @shearn89 I asked Claude to draft it up during our incident call - I didn't see any issues in the provider, and it's the first place I normally go, so I was hoping to short-circuit other people's panicked investigations! 😆

[DerekFrank]

Converting to a discussion. We can keep conversations about improvements in the more specific issues