Helm lookup function not working for resources created in the same chart

[teeheehs]

We are encountering an issue with the Helm lookup function when trying to reference resources that are created within the same chart during installation. It appears that the lookup function is unable to find the newly created resources, leading to installation failures.

We have a chart that creates CIDR resources and then attempts to lookup those resources in subsequent parts of the chart. However, due to Helm's caching or some unknown snapshot of the Kubernetes API (??), the lookup function fails to find the newly created CIDR resources later in the Chart's execution.

Output of helm version:

version.BuildInfo{Version:"v3.14.2", GitCommit:"c309b6f0ff63856811846ce18f3bdc93d2b4d54b", GitTreeState:"clean", GoVersion:"go1.21.7"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"26+", GitVersion:"v1.26.15-dispatcher", GitCommit:"5490d28d307425a9b05773554bd5c037dbf3d492", GitTreeState:"clean", BuildDate:"2024-04-18T22:39:05Z", GoVersion:"go1.21.9", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25",

Cloud Provider/Platform (AKS, GKE, Minikube etc.): Google

Here are the relevant snippets from our chart where we are encountering this issue:

../some-template.yaml:

# Fetches the CIDR directly from the allocated CIDR resource
{{- $podCidrClaim := (lookup "private.widget/v1alpha1" "CIDRClaim" (printf "%s-cluster" $clusterName) (printf "%s-pod-cidr" $clusterName)) }}
pods:
  {{- if $podCidrClaim }}
  cidrBlocks:
    - {{ index $podCidrClaim.status.ipv4AllocationStatus.cidrBlocks 0 }}
  {{- else }}
  cidrBlocks:
    - "0.0.0.0/24"
  {{- end }}

The initial installation fails because the lookup function is unable to find the newly created CIDR resources generated earlier in the same chart (as pre-install hooks). However, the immediately-after and subsequent helm upgrade succeeds because the CIDRs are then present in the cluster when the upgrade process begins.

We would like to confirm if there is some sort of caching of the kubernetes API or some "snapshot" of the cluster's resources at the beginning of the initial installation or subsequent upgrade process, and if this is a known limitation of the lookup function or if there are any recommended workarounds to handle this scenario.

Any guidance or insights would be greatly appreciated. Thank you in advance for your help!

Similar to #10186, but uniq function will not work here..

[mattfarina]

I can explain what's happening here. It's not a caching issue.

First, Helm generates the manifests from the templates. All of them are handled at the same time. This is when the lookup function is executed. Since nothing from this chart has been placed in the cluster, there is nothing to lookup.

Second, Helm sends all of the generated manifests to the Kubernetes API at once. There is some ordering to it but it is still one large batch.

Kubernetes is designed to be declarative. You pass everything to it and the controllers should continuously reconcile until everything is in a healthy state.

Unfortunately, this doesn't always work. In this case it looks like you want things to behave in 2 steps. One the resource for the CIDR has completed getting the CIDRs and has them on the status then update the pods to use them. There is no good way to do this in Helm. The more idiomatic way to do this in k8s is with another CRD and custom controller over all those things.

[teeheehs]

Thanks for the explanation!

Here's some additional context:

We're creating CIDR resources using pre-install hooks that include Jobs to verify their validity by checking the .status. These hooks ensure the CIDRs are ready before proceeding:

wait-for-cidr-hj59d Waiting for CIDR Claims to be ready...
wait-for-cidr-hj59d CIDR Claims are ready. Proceeding with installation.

In subsequent templates, we attempt to lookup these CIDRs using lookup. However, this fails (per above) because the resources haven't been deployed yet when Helm initially processes the templates.

Given this, I'm wondering if splitting the chart into two parts would be a viable alternative to creating a new controller:

Chart 1: Create the CIDR resources and verify.
Chart 2: Dependent on Chart 1, create all resources that require the CIDR information.

Would this approach allow successful lookup of the CIDRs from Chart 2, or would the templating in Chart 2 still encounter the unwanted default/empty results?

[teeheehs]

Resolved the issue by splitting the original chart into two separate charts:

• Chart 1: Creates and verifies CIDR resources using pre-install hooks and Jobs.
• Chart 2: References CIDR resources created by Chart 1 using the lookup function.

Installing Chart 1 first ensures CIDR resources are available for Chart 2. Having Chart 1 as a dependency in Chart 2 did not work due to conflicting values and overrides from Chart 1 being templated into Chart 2. Overall, this approach avoids conflicts and eliminates the need for a custom controller.

The choice was to ship a wonky chart using:

1. install (fail the lookup)
2. upgrade (lookup passes)

..or structure two separate Chart install commands to be run sequentially. I opted for the latter as it seems cleaner.

[teeheehs]

We can close this issue as I have what I need. Could we consider an update to the lookup docs to include information about this limitation?

The lookup function shouldn't reference resources created within the same chart during initial install because Helm templates all manifests in a single batch prior to creating any resources. This results in lookup failures during the initial installation, but subsequent upgrades will succeed. Consider splitting your chart into smaller, more manageable charts if you need to reference resources created within the same chart.

I'd be happy to draft a small change for inclusion if agreeable. Appreciate the help!