Skip to content

Manually rebuild eso chart - #22

Merged
nicklathe merged 1 commit into
mainfrom
nicklathe/rebuild-eso-chart
Aug 7, 2026
Merged

Manually rebuild eso chart#22
nicklathe merged 1 commit into
mainfrom
nicklathe/rebuild-eso-chart

Conversation

@nicklathe

@nicklathe nicklathe commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

TL;DR

There's a fair amount of Claude generated PR description below that is helpful, but difficult to follow. Here's the TL;DR:

  • PR Allow db creds declared in Cfn to be secrets #21 added dataFrom to get secrets from the Cloud Formation Stack. These are picked up automatically on EC2, but weren't getting included as an ExternalSecret in the cluster
  • I verified with Claude that Argo was syncing correctly
  • It was conformed that apps/infra/standard-envtypes/chart/charts/eso-per-env-0.1.0.tgz is what Argo (via Helm) uses, however this is stale and hasn't been rebuilt to include the previous PR changes (it also includes a couple other PR changes, as it was 3 commits behind since being last rebuilt)
  • This PR rebuilds that chart
  • There is a followup decision to be made on if we continue to vendor the zipped chart or if we want Helm to build it each time. More research and understanding is needed there.

Why

PR #21 added a second dataFrom entry to the per-namespace ExternalSecret so
that CfnStack/<env>/* secrets sync into cdo-external-secrets. It merged, Argo
synced it, and nothing happened.

The evidence:

kubectl get application standard-envtypes -n argocd -o jsonpath='{.status.sync.status}{.status.health.status}'
Synced Healthy

kubectl get application standard-envtypes -n argocd -o jsonpath='{.status.sync.revisions}'
["4a22fdf...","4a22fdf..."] # the merge commit of #21

git show 4a22fdf:apps/infra/charts/eso-per-env/templates/_envtype.tpl | grep -c CfnStack

the change is in the source at that revision

kubectl get externalsecret cdo-external-secrets -n staging -o jsonpath='{.spec.dataFrom[*].find.path}'
staging/cdo/ # ...but only one path is live

Synced to the correct commit, correct source at that commit, wrong rendered output.

The cause is apps/infra/standard-envtypes/chart/charts/eso-per-envtype-0.1.0.tgz.
Helm renders subcharts from the charts/ directory, not from the file:// source
path, so the committed archive is what actually produces manifests. argocd-repo-server
does run helm dependency build before templating — but that command only fetches or
packages dependencies that are missing. The archive was already present at the
version pinned in Chart.lock, so helm left it untouched and rendered a copy of
_envtype.tpl last packaged in April.

Every layer behaved correctly and the net result was that source edits to
_envtype.tpl silently did nothing.

What

Repackage the vendored subchart from source:

cd apps/infra/standard-envtypes/chart && rm -f charts/*.tgz && helm dependency build

One binary file changes. Chart.lock is untouched — the subchart version is
unchanged at 0.1.0, so this is a repackage, not a version bump.

Reviewing a binary diff

The archive is tracked via Git LFS (*.tgz filter=lfs), so the diff is unreadable.
To see what's inside:

tar xzOf apps/infra/standard-envtypes/chart/charts/eso-per-envtype-0.1.0.tgz

eso-per-envtype/templates/_envtype.tpl | diff - apps/infra/charts/eso-per-env/templates/_envtype.tpl

That should now be empty. Before this PR it differed by three commits' worth of edits.

What else ships with this

The archive was stale by three commits, not one, so rebuilding also lands the two
that preceded #21:

  • 02b7896 — added creationPolicy / deletionPolicy / conversionStrategy /
    decodingStrategy to the manifests
  • 1021fba — removed them again

Net effect: those four fields disappear from the desired state. All four are ESO
CRD defaults, so the API server repopulates them on the live object and there is no
behavioral change — but Argo may report drift on them at first sync if the existing
managed-fields ignores don't cover them. Cosmetic if it happens.

Verification

After sync, allow one 5m ESO refreshInterval, then:

kubectl get externalsecret cdo-external-secrets -n staging -o jsonpath='{.spec.dataFrom[*].find.path}'

expect: staging/cdo/ CfnStack/staging/

kubectl get secret cdo-external-secrets -n staging -o jsonpath='{.data}' | jq -r 'keys[]' | grep db_endpoint

expect: db_endpoint_proxy_reader, db_endpoint_proxy_reporting, etc.

kubectl rollout restart deploy/cdo-active-job-worker -n staging

The restart is required — envFrom values are injected once at container start.

Confirmed locally: rendering the chart directory as-is with the old archive produced
zero CfnStack references; with the rebuilt archive it produces the expected two
dataFrom paths for staging, test, levelbuilder, and production, with the adhoc
ClusterExternalSecret unchanged.

Follow-up

This repo is in a half-vendored state: the archive is tracked so it wins, but nothing
regenerates it, so template edits are silent no-ops. Three commits died this way before
anyone noticed.

eso-per-envtype is a library chart with one template file, one consumer, in the same
repo, referenced by file://. Vendoring buys nothing and costs a lockfile, an LFS
binary, and this failure mode. Next PR moves _envtype.tpl into
apps/infra/standard-envtypes/chart/templates/ and deletes the subchart, the
dependency, Chart.lock, and charts/ — which should render byte-identically.

Until that lands, anyone editing _envtype.tpl must rebuild the archive in the same
commit.

🤖 Generated with Claude Code

Signed-off-by: Nick Lathe <nick.lathe@code.org>
@nicklathe
nicklathe merged commit e564fb8 into main Aug 7, 2026
@nicklathe
nicklathe deleted the nicklathe/rebuild-eso-chart branch August 7, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants