example: the wallet, shoppingcart and billing walkthroughs fail on kind on every run since 2026-10-22, waiting for Grafana's data sources, so nothing can merge

[Zaba505]

What happens

Since 2026-10-22 02:09 UTC, three walkthrough jobs fail on every run, on every branch:

• Wallet walkthrough (kind)
• Shoppingcart walkthrough (kind)
• Billing walkthrough (kind)

Each one fails in its functional tests, at the check that Grafana serves its Mimir, Loki and Tempo data sources. Build, which the default branch's ruleset requires, passes only when every job before it passes. So no pull request can merge, and neither can anything in the merge queue. Re-running the failed jobs doesn't help.

Every attempt so far has failed all three jobs. That is 21 of 21 job runs:

Run	Where	Attempts
39709832274	PR #1272	3, from 02:09, 02:21 and 02:32
39710262143	Renovate PR (#1235)	1, from 02:14
39711824443	PR #1273	3, from 02:31, 02:41 and 03:00

From the latest attempt of #1273. Billing (job):

--- FAIL: TestBilling_sends_a_runs_telemetry_to_the_LGTM_stack (304.15s)
    --- FAIL: TestBilling_sends_a_runs_telemetry_to_the_LGTM_stack/Grafana_has_all_three_as_data_sources (299.97s)
        --- FAIL: TestBilling_sends_a_runs_telemetry_to_the_LGTM_stack/Grafana_has_all_three_as_data_sources/tempo (299.97s)
                Messages:   last: GET /api/datasources/uid/tempo/health: 404 Not Found: {"statusCode":404,"messageId":"plugin.notRegistered","message":"Plugin not registered"}
        --- FAIL: TestBilling_sends_a_runs_telemetry_to_the_LGTM_stack/Grafana_has_all_three_as_data_sources/loki (0.00s)
                Messages:   last: Get "http://127.0.0.1:43369/api/datasources/uid/loki/health": context deadline exceeded

Wallet (job):

--- FAIL: TestWallet_sends_its_telemetry_to_the_LGTM_stack_on_kind (242.37s)
        Messages:   Mimir, Loki and Tempo as Grafana's data sources: context deadline exceeded (last error: Get "http://127.0.0.1:41975/api/datasources/uid/mimir/health": context deadline exceeded)

Shoppingcart (job):

--- FAIL: TestShoppingcart_sends_its_telemetry_to_the_LGTM_stack_on_kind (252.54s)
        Messages:   Mimir, Loki and Tempo as Grafana's data sources: context deadline exceeded (last error: <nil>)

What's known

This hasn't been triaged. What follows comes from the runs:

• The same code passed two hours earlier. story(grpcserver): a stream opened again when Shutdown ends one lands on another pod #1272, fix(deps): update patch and digest versions #1235 and story(flaky): sweep every package's unit tests, each run pinned to the cores of its CPU count #1273 all branch from 08b44fa, and none of them changes anything under example/*/deploy or example/*/functional. main's push run of 08b44fa, 39697194966, passed all three jobs at 00:02 UTC. The kind overlays last changed in story(example): send the wallet's and the ledger's telemetry to the LGTM stack in kind #1248 (6351c86).
• The Grafana image is the same. It is docker.io/grafana/grafana:13.2.4, and the failed runs pulled sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f, which Docker Hub last pushed on 2026-10-13.
• Grafana reports the Tempo data source's plugin as not registered. The billing check waits out its whole deadline on plugin.notRegistered for tempo. Its loki and mimir subtests then fail at once, on the spent deadline, so they don't show whether those two are registered. The wallet's and the shoppingcart's checks run out their 4-minute telemetry deadline on the same data sources.
• Tempo, Loki and Mimir themselves get the telemetry. The wallet's and the shoppingcart's tests check that before they check Grafana, and in the failed runs they got past it: the trace in Tempo, the records in Loki, the series in Mimir.
• The other kind jobs pass, but they don't check Grafana. Order status, Databases and Mailer pass in the same runs. Only the wallet's, the shoppingcart's and the billing's functional tests ask Grafana for its data sources, so those passes don't show whether Grafana works there.
• The runs don't show why. The job's "Show the … namespace" step prints kubectl get all and the pods' descriptions, not Grafana's logs.

Not yet known: why Grafana doesn't register the plugin, whether it does so on a local kind cluster, and whether anything outside the repository changed between 00:02 and 02:09 UTC.

Expected

The three jobs pass on a commit that passed before, as main's run of 08b44fa did. Grafana serves the Mimir, Loki and Tempo data sources that each kind overlay provisions.

References

• run 39697194966, the last run that passed all three jobs
• the Grafana deployments and their data sources: example/wallet/deploy/kind/grafana.yaml and grafana/datasources.yaml, and the same files under example/shoppingcart and example/billing
• the checks: wallet, shoppingcart, billing
• the pull requests it blocks so far: story(grpcserver): a stream opened again when Shutdown ends one lands on another pod #1272, fix(deps): update patch and digest versions #1235, story(flaky): sweep every package's unit tests, each run pinned to the cores of its CPU count #1273
• earlier, and maybe unrelated: on story(example): send the wallet's and the ledger's telemetry to the LGTM stack in kind #1248's own run, 39686280964, the wallet's telemetry test ran out the same 4-minute deadline on an earlier check, "the ledger's metrics from pod … in Mimir"

[Zaba505]

Triage

Verdict: a reproducible bug in the walkthroughs' kind overlays, not in humus's packages. On startup, Grafana updates its bundled Tempo plugin from grafana.com, and the update fails on the overlay's read-only root filesystem after the old plugin is already unloaded. Every overlay's Grafana is affected, not only the three whose tests check it, and Loki and Mimir will break the same way on their plugins' next release. #1275 is filed to fix it.

Is this a misuse of humus?

No. The overlays and their Grafana are the repository's own code. No end user is involved: example/*/deploy/kind provisions Grafana, and up.sh and the kind job run it. Nothing outside the repository can change what the overlay does with Grafana.

Is it a feature request?

No. #629, revision 14 promises each kind overlay has "Grafana, with all three as data sources". #1177, #948 and #949 promise the same for the wallet, the shoppingcart and billing, and their kind-only checks test it. Restoring that changes no API. Revision 13 also asks for "a non-root container with a read-only root filesystem". Every overlay gives Grafana that security context, and the bug comes from it.

Is it reproducible?

Yes. I ran the wallet's kind overlay at 08b44fa, with up.sh, on a local kind cluster: kind v0.22.0 on rootless Podman 5.8.9, Go 1.27.0, grafana/grafana:13.2.4, grafana/loki:3.6.18, grafana/tempo:3.1.0, grafana/mimir:3.2.2. Tempo's plugin on grafana.com was 13.3.0.

test	runs failed
a new test: restart Grafana, wait for its plugin installer to finish with tempo, then check each data source's health	4 of 4
the existing TestWallet_sends_its_telemetry_to_the_LGTM_stack_on_kind, as a baseline	1 of 1

A fresh Grafana pod loses its Tempo plugin. Each run restarted Grafana, so it started on an empty data directory. Grafana's own log, in order, from run 1:

1. logger=plugins.registration msg="Plugin registered" pluginId=tempo: the bundled plugin, 13.2.4, loads.
2. logger=plugin.backgroundinstaller msg="Installing plugin" pluginId=tempo, then logger=plugin.installer msg="Updating plugin" pluginId=tempo from=13.2.4 to=13.3.0.
3. logger=plugin.tempo msg="plugin process exited" plugin=/usr/share/grafana/data/plugins-bundled/tempo/gpx_grafana-tempo-datasource_linux_amd64: the old plugin is unloaded.
4. logger=plugin.backgroundinstaller level=error msg="Failed to install plugin" pluginId=tempo error="unlinkat /usr/share/grafana/data/plugins-bundled/tempo: read-only file system".
5. For the next 2 minutes, GET /api/datasources/uid/tempo/health answers 404 {"statusCode":404,"messageId":"plugin.notRegistered","message":"Plugin not registered"}. Mimir answers "Successfully queried the Prometheus API." and Loki "Data source successfully connected.", both "status":"OK".

The installer reached step 4 within 2 seconds of the rollout in every run. The same log also shows the update failing, the same way, for elasticsearch, grafana-pyroscope-datasource, opentsdb, stackdriver and zipkin, which grafana.com also lists newer versions of.

The baseline failed with the message in the report, Mimir, Loki and Tempo as Grafana's data sources: context deadline exceeded (last error: <nil>).

I also ran a plain Grafana 13.2.4 container with the wallet's data sources file. On a writable root filesystem, the update went through: Downloaded and extracted tempo v13.3.0 zip successfully to /var/lib/grafana/plugins/tempo. Run read-only, with the data directory and /tmp writable as in the overlay, it failed as in step 4 and Tempo was plugin.notRegistered. So the trigger is the read-only root filesystem together with a newer plugin on grafana.com.

This corrects two points of the report:

• Mimir and Loki are registered. Their health is OK once Tempo's failure is out of the way. The billing's loki and mimir subtests fail only because the tempo subtest used up their shared deadline. The wallet's check names Mimir in its last error for the same reason. Reproduced.
• The wallet's and the shoppingcart's checks hide Grafana's answer. Their getJSON treats any 404 as "not there yet" (wallet), so plugin.notRegistered turns into last error: <nil>. Reasoned from code, and the baseline's message matches.

Why

Grafana updates the plugins it bundles on every start. Grafana 13.2.4 preinstalls a default list of plugins, which includes prometheus, loki and tempo. With preinstall_auto_update = true, the default, the background installer asks grafana.com for each plugin's newest compatible version, and installs it when it is newer than the one loaded. The overlays leave both settings at their defaults. Reasoned from code, and the log lines above show it.

An update removes the old plugin before it has the new one. updateFromCatalog calls Remove first. Remove unloads the plugin, then deletes its directory. The bundled plugins live under /usr/share/grafana/data/plugins-bundled, in the image's root filesystem. The overlays mount it read-only, so the delete fails, Remove returns the error, and nothing loads the plugin again. The installer logs the error and goes on. Reasoned from code, and reproduced.

What changed outside the repository. grafana.com lists Tempo's plugin 13.3.0, last updated at 2026-10-22T00:17:51Z. That is after main's passing run at 00:02 and before the first failing run at 02:09. Until then, 13.2.4 was the newest version, the installer had nothing to update, and the bundled plugin stayed loaded. The image's tag and digest didn't change. That fits the report's timeline; I didn't rerun the old runs.

Wider impact. All six kind overlays run Grafana 13.2.4 with readOnlyRootFilesystem: true: wallet, shoppingcart, billing, order status, databases and mailer. So the order status's, the databases' and the mailer's Grafana have no Tempo data source either. Their jobs pass only because their tests don't ask Grafana. Grafana 13.2.4 bundles loki 13.2.2 and prometheus 13.2.3, the newest versions on grafana.com today. When either gets a newer release, the Loki or Mimir data source drops out the same way, so those data sources' health also depends on grafana.com on the day a pod starts. I got that from Grafana's code and the bundled plugin.json files, and didn't reproduce it for loki or prometheus.

#1275 is filed to fix it:

• every overlay's Grafana serves Mimir, Loki and Tempo with health OK after a fresh start, whatever grafana.com lists, and keeps its least-privilege security context;
• the wallet's functional tests hold this reproduction, and the kind jobs are what show it;
• the wallet's and the shoppingcart's checks name Grafana's answer when they fail;
• example: the wallet, shoppingcart and billing walkthroughs fail on kind on every run since 2026-10-22, waiting for Grafana's data sources, so nothing can merge #1274 is closed as Resolved once the fix merges.

[Zaba505]

Fixed by #1276 (story #1275), merged as 0c0276b.

All six kind overlays' Grafana (wallet, shoppingcart, billing, order status, databases, mailer) now set GF_PLUGINS_PREINSTALL_DISABLED=true. Grafana's background plugin installer no longer runs, so it keeps the prometheus, loki and tempo plugins its image bundles, whatever grafana.com lists, and the read-only root filesystem and the rest of its security context stay as they were. The wallet's functional tests now hold the triage's reproduction: restart Grafana, hold until it is done with tempo, then require Mimir, Loki and Tempo to report health OK. The wallet's and the shoppingcart's Grafana checks name the data source and Grafana's answer when they fail, instead of last error: <nil>. All six kind jobs passed on the pull request.