jupyterhub blueprint gets 400 when pulling charts

[asmacdo]

Description

While following this blog to deploy the jupyterhub blueprint , Terraform fails with the error:

module.eks_blueprints_addons.module.karpenter.helm_release.this[0]: Creating...
╷
│ Error: could not download chart: unexpected status from HEAD request to https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.32.1: 400 Bad Request
│ 
│   with module.eks_blueprints_addons.module.karpenter.helm_release.this[0],
│   on .terraform/modules/eks_blueprints_addons.karpenter/main.tf line 9, in resource "helm_release" "this":
│    9: resource "helm_release" "this" {
│ 
╵

• [ X] ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

[X] these steps

Before you submit an issue, please perform the following for Terraform examples:

1. Remove the local `.terraform` directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): `rm -rf .terraform/`
2. Re-initialize the project root to pull down modules: `terraform init`
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]:
  module.eks is the one that fails, on latest main (e64db20)

• Terraform & providers version:

Terraform v1.6.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.27.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.2
+ provider registry.terraform.io/hashicorp/helm v2.12.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.24.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/time v0.9.2
+ provider registry.terraform.io/hashicorp/tls v4.0.5

Reproduction Code [Required]

git clone https://github.com/awslabs/data-on-eks/
# chmod install.sh
cd ai-ml/jupyterhub && ./install.sh

Steps to reproduce the behavior:

Not using workspaces
I did clear the local cache

Expected behavior

terraform completes

Actual behavior

400 error on HEAD request to ecr for helm charts

Additional context

I tried a handful of things:

1. Updated helm from 3.11 to 3.12, and also 3.13.2
2. Troubleshooting guide for failure to download chart
3. Clearing .terraform
4. bumping the karpenter helm chart version from v0.31.2 to v0.32.2

This is my first experience with terraform so theres a good chance of user error ;)

Other versions that might be relevant:

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:09:36 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:08 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

helm version
version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

 helm version
version.BuildInfo{Version:"v3.13.2", GitCommit:"2a2fb3b98829f1e0be6fb18af2f6599e0f4e8243", GitTreeState:"clean", GoVersion:"go1.20.10"}

[asmacdo]

I was able to replicate with helm directly using 3.13.1

helm install oci://public.ecr.aws/karpenter/karpenter  --version v0.32.1 --generate-name

Error: INSTALLATION FAILED: unexpected status from HEAD request to https://public.ecr.aws/v2/karpenter/karpenter/manifests/v0.32.1: 400 Bad Request

Updating to 3.13.2 Fixed it for Helm, but I continued to get the same error from Terraform. So I updated terraform

terraform version
Terraform v1.6.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.27.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.2
+ provider registry.terraform.io/hashicorp/helm v2.12.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.24.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/time v0.9.2
+ provider registry.terraform.io/hashicorp/tls v4.0.5

Still having the same issue, but I am convinced this isn't a data-on-eks issue. I'll refile against https://github.com/hashicorp/terraform-provider-helm.

[asmacdo]

The issue was fixed upstream, reopening to bump the helm provider requirement.