Skip to content

fix(eks): clear OCI repo/version after local pull for Helm v4 compatibility#37142

Merged
mergify[bot] merged 2 commits into
mainfrom
fix/eks-helm-oci-v4-compat
Mar 19, 2026
Merged

fix(eks): clear OCI repo/version after local pull for Helm v4 compatibility#37142
mergify[bot] merged 2 commits into
mainfrom
fix/eks-helm-oci-v4-compat

Conversation

@aemada-aws
Copy link
Copy Markdown
Contributor

@aemada-aws aemada-aws commented Mar 2, 2026

Issue # (if applicable)

Closes cdklabs/awscdk-asset-kubectl#2681
Closes #37143

Reason for this change

Helm v4 (bundled in kubectl-v35) changed LocateChart behavior: when --repo is set, the local path check is skipped entirely and Helm tries to resolve the chart from the repo URL. Previously Helm v3 prioritized local chart paths over --repo (v3.19 source).

This caused OCI helm chart deployments to fail with "failed to perform FetchReference on source: invalid reference" because the CDK handler was passing both a local chart path AND --repo/--version to helm upgrade after pulling the OCI chart locally. Helm v4 no longer falls back to the local path when --repo is provided (v4 source).

Additionally, EKS dropped AL2 AMI support starting from Kubernetes 1.33. Since the integ test version helpers now dynamically pick the latest K8s version (currently 1.35), all tests using the default AMI type (AL2_x86_64) or AL2_x86_64_GPU fail with:

"AMI Type AL2_x86_64 is only supported for kubernetes versions 1.32 or earlier"

These tests need to either migrate to AL2023 AMI types or be pinned to k8s 1.32.

Description of changes

  1. Fix OCI helm chart handling (v1 + v2 handlers): Clear repository and version after pulling the OCI chart locally, so helm upgrade only receives the local chart path without --repo/--version. This is correct for both Helm v3 and v4 since the chart is already at the right version locally.

  2. Dynamic K8s version selection in integ test helpers: The getClusterVersionConfig helper in both aws-eks and aws-eks-v2 integ tests now dynamically picks the latest version from the versionMap instead of hardcoding it.

  3. Helm asset integ tests cover latest and latest-1: Both v1 and v2 integ.eks-helm-asset.ts now create two stacks — one with the latest K8s version and one with latest-1.

  4. Added non-OCI chart coverage: Both helm-asset integ tests now also install a chart from a standard HTTPS Helm repository.

  5. AL2 to AL2023 AMI migration for EKS integ tests (required because AL2 is unsupported from k8s 1.33+):

    • integ.helm-chart-logging: Use defaultCapacity: 0 + explicit AL2023 nodegroup
    • integ.eks-windows-ng: Change Linux nodegroup from AL2_X86_64 to AL2023_X86_64_STANDARD
    • integ.eks-inference: Pin to k8s 1.32 — the library's EksOptimizedImage class hardcodes AL2 SSM paths (amazon-linux-2-gpu/) for GPU/Inferentia/Trainium node types, so AL2023 cannot be used without a library change
    • integ.eks-inference-nodegroup: Use AL2023_X86_64_NEURON for inference nodegroups, AL2023_X86_64_STANDARD for default capacity, add region constraint (inf2.xlarge not available in ca-central-1)
    • integ.eks-service-account-sdk-call: Use defaultCapacity: 0 + explicit AL2023 nodegroup
    • integ.eks-helm-asset (aws-eks): Add service account for ec2-chart (matching aws-eks-v2 pattern, fixes atomic helm install timeout)
  6. New integ.eks-default-capacity test: Explicit test for the defaultCapacity > 0 codepath, pinned to k8s 1.32 since the CDK library defaults to AL2_x86_64 as the first AMI type for x86_64 instances. This ensures the default capacity path remains tested. (Not needed for eks-v2 since it defaults to AUTOMODE which doesn't create a nodegroup.)

Describe any new or updated permissions being added

  • integ.eks-helm-asset (aws-eks): Added OIDC provider + service account with AmazonEC2FullAccess for ec2-chart (matching existing aws-eks-v2 pattern)

Description of how you validated changes

All integ tests deployed and validated successfully locally:

# Tests validated (all SUCCESS):
yarn integ \
  test/aws-eks/test/integ.eks-bottlerocket-ng.js \
  test/aws-eks/test/integ.eks-cluster-handlers-vpc.js \
  test/aws-eks/test/integ.eks-cluster-imported.js \
  test/aws-eks/test/integ.eks-cluster-private-endpoint.js \
  test/aws-eks/test/integ.eks-helm-asset.js \
  test/aws-eks/test/integ.eks-inference.js \
  test/aws-eks/test/integ.eks-inference-nodegroup.js \
  test/aws-eks/test/integ.eks-service-account-sdk-call.js \
  test/aws-eks/test/integ.eks-windows-ng.js \
  test/aws-eks/test/integ.helm-chart-logging.js \
  test/aws-eks/test/integ.eks-default-capacity.js \
  test/aws-eks-v2/test/integ.eks-helm-asset.js \
  --disable-update-workflow --update-on-failed --force \
  --parallel-regions us-east-1 us-west-2 eu-west-1 ap-south-1 ca-central-1 sa-east-1

Checklist


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions github-actions Bot added the p2 label Mar 2, 2026
@aws-cdk-automation aws-cdk-automation requested a review from a team March 2, 2026 23:20
@mergify mergify Bot added the contribution/core This is a PR that came from AWS. label Mar 2, 2026
@mergify mergify Bot temporarily deployed to automation March 2, 2026 23:21 Inactive
@mergify mergify Bot temporarily deployed to automation March 2, 2026 23:21 Inactive
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 2, 2026

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
This security report is NOT a review blocker. Please try merge from main to avoid findings unrelated to the PR.


TestsPassed ✅SkippedFailed
Security Guardian Results936 ran936 passed
TestResult
No test annotations available

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 2, 2026

⚠️ Experimental Feature: This security report is currently in experimental phase. Results may include false positives and the rules are being actively refined.
This security report is NOT a review blocker. Please try merge from main to avoid findings unrelated to the PR.


TestsPassed ☑️SkippedFailed ❌️
Security Guardian Results with resolved templates936 ran930 passed6 failed
TestResult
Security Guardian Results with resolved templates
packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-bottlerocket-ng.js.snapshot/aws-cdk-eks-cluster-bottlerocket-ng-test.template.json
iam-role-root-principal-needs-conditions.guard❌ failure
packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-cluster-imported.js.snapshot/aws-cdk-eks-import-cluster-test.template.json
iam-role-root-principal-needs-conditions.guard❌ failure
packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-cluster-private-endpoint.js.snapshot/aws-cdk-eks-cluster-private-endpoint-test.template.json
iam-role-root-principal-needs-conditions.guard❌ failure
packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-helm-asset.js.snapshot/aws-cdk-eks-helm-test-prev.template.json
iam-role-root-principal-needs-conditions.guard❌ failure
packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-helm-asset.js.snapshot/aws-cdk-eks-helm-test.template.json
iam-role-root-principal-needs-conditions.guard❌ failure
packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ.eks-windows-ng.js.snapshot/aws-cdk-eks-cluster-windows-ng-test.template.json
iam-role-root-principal-needs-conditions.guard❌ failure

@aemada-aws aemada-aws added bug This issue is a bug. p1 and removed p2 labels Mar 2, 2026
@aemada-aws aemada-aws marked this pull request as ready for review March 2, 2026 23:23
@aemada-aws aemada-aws added the pr/needs-integration-tests-deployment Requires the PR to deploy the integration test snapshots. label Mar 2, 2026
@aemada-aws aemada-aws had a problem deploying to deployment-integ-test March 2, 2026 23:23 — with GitHub Actions Error
@aws-cdk-automation aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Mar 3, 2026
…art fixes

- Update tests using AL2_x86_64 AMI type to AL2023_x86_64_STANDARD (k8s 1.35 dropped AL2 support)
- Pin integ.eks-inference to k8s 1.32 (EksOptimizedImage hardcodes AL2 SSM paths)
- Use AL2023_X86_64_NEURON for inference nodegroups
- Add service account for ec2-chart in integ.eks-helm-asset (matching eks-v2 pattern)
- Add region constraint for integ.eks-inference-nodegroup (inf2.xlarge not in ca-central-1)
- Add new integ.eks-default-capacity test to cover defaultCapacity codepath on k8s 1.32
@aemada-aws aemada-aws removed the pr/needs-integration-tests-deployment Requires the PR to deploy the integration test snapshots. label Mar 19, 2026
@aws-cdk-automation aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Mar 19, 2026
@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Mar 19, 2026

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@mergify
Copy link
Copy Markdown
Contributor

mergify Bot commented Mar 19, 2026

Merge Queue Status

  • Entered queue2026-03-19 09:49 UTC · Rule: default-squash
  • Checks passed · in-place
  • Merged2026-03-19 09:49 UTC · at 798c7bf7412b7d87adaa18b631ea137b65c247c6

This pull request spent 6 seconds in the queue, with no time running CI.

Required conditions to merge

@github-actions
Copy link
Copy Markdown
Contributor

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

bug This issue is a bug. contribution/core This is a PR that came from AWS. p1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(eks): OCI Helm chart deployments fail with "invalid reference" on kubectl-v35 (Helm v4) v35 helm fails to install oci charts

3 participants