example/databases: a Secret deleted while a reconcile is in flight comes back with the old password, so the rotation is lost

[Zaba505]

What happens

The shared Rotation case of the Databases walkthrough fails intermittently, on main, in the merge queue and on PRs. It fails under both Databases walkthrough (testcontainers) and Databases walkthrough (kind), e.g. run 39619448817 on main:

--- FAIL: TestInit_passes_the_shared_cases_against_k3s_and_PostgreSQL/in_parallel/Rotation (15.49s)
    cases.go:102:
        Error Trace:  example/databases/cases/cases.go:102
                      example/databases/app/app_testcontainers_test.go:250
        Error:        Should not be: "PQIYQB7QXGQ3ETHMZBMI5E4GSV"

It has failed this way 10 times since 2026-10-20, out of about 200 CI runs, about 1 in 20:

Run	Where	Job
39388749963	merge queue (#1161)	testcontainers
39395546931	main	testcontainers
39397716138	merge queue (#1164)	testcontainers
39397931601	main	testcontainers
39412690207	merge queue (#1172)	kind
39455011674	merge queue (#1173)	kind
39487562056	main	testcontainers
39509752786	merge queue (#1228)	testcontainers
39524549052	Renovate PR	kind
39619448817	main	testcontainers

The case deletes the Database's Secret and waits for a Secret with a new UID. One comes, but it has the old password, so cases.go:102 fails. The rotation is lost for good: every later reconcile finds the new Secret and keeps the password in it.

The Rotation section says deleting the Secret rotates the password.

Why

This hasn't been triaged. What follows comes from the code and a reproduction.

Reconcile reads the password from the Secret with a Get, provisions the role with it, then applies the Secret with server-side apply. If the Secret is deleted after the Get and before the apply, the apply creates a new Secret, with a new UID, holding the password the Get read. The deletion's reconcile then finds that Secret and keeps its password.

A reconcile is likely in flight when the case deletes the Secret: the Controller owns Secrets, so the first reconcile's own Secret apply and status update each queue another reconcile of the Database, and under CI's load, with the eight cases in parallel under -race, one of them can still be running.

Reproduction. This test, added to example/databases/controller at 02f0286, has the delete land right after the reconcile's Get. It fails every time (-count=3), with the Secret holding the old password:

func TestRepro_a_Secret_deleted_during_a_reconcile_comes_back_with_the_old_password(t *testing.T) {
	kc := kubefake.NewClientset()
	admin := &fakeAdmin{}
	r := newReconciler(kc, admin)
	db := newDatabase()
	first, err := r.Reconcile(t.Context(), db)
	require.NoError(t, err)
	db.Status = *first.Status
	old := secret(t, kc)

	kc.PrependReactor("get", "secrets", func(a clienttesting.Action) (bool, runtime.Object, error) {
		kc.ReactionChain = kc.ReactionChain[1:]
		gvr := corev1.SchemeGroupVersion.WithResource("secrets")
		obj, err := kc.Tracker().Get(gvr, "shop", "orders-postgres")
		require.NoError(t, err)
		require.NoError(t, kc.Tracker().Delete(gvr, "shop", "orders-postgres"))
		return true, obj, nil
	})
	_, err = r.Reconcile(t.Context(), db)
	require.NoError(t, err)

	_, err = r.Reconcile(t.Context(), db)
	require.NoError(t, err)

	require.NotEqual(t, old.Data["password"], secret(t, kc).Data["password"], "the rotation was lost")
}

--- FAIL: TestRepro_a_Secret_deleted_during_a_reconcile_comes_back_with_the_old_password (0.00s)
    Error:      Should not be: []byte{0x4b, 0x34, 0x33, ...}
    Messages:   the rotation was lost

Expected

Deleting a Database's Secret always rotates its password, whenever the delete lands. The Rotation case passes whenever the controller works.

References

• run 39619448817, the run that prompted this
• Reconciler.password and applySecret
• the Rotation case

[Zaba505]

Triage

Verdict: a reproducible bug in the Databases walkthrough's Reconciler, not in kube. A Secret deleted while a reconcile is in flight comes back with the old password, and the rotation is lost for good, as reported. The window is wider than the report says: it runs from the reconcile's read of the Secret to its apply, so the whole of Provision is in it, and three reconciles followed the first Ready in every container run. #1251 is filed to fix it.

Is this a misuse of humus?

No. The behaviour is the walkthrough's own code: its Reconcile, password and applySecret. The reporter only deleted a Secret, which the Rotation section tells the reader to do.

kube keeps its promises here. Its docs say reconciles are level-triggered, and "one object is never reconciled twice at once". The deletion did queue a reconcile, and it ran right after the in-flight one, as client-go's queue does with a key added while it is processed (Add, Done). That reconcile was handed the Database as it is now, and found a Secret, because the in-flight one had made it again.

Is it a feature request?

No. Three promises cover it:

• The Rotation section: "Deleting the Secret rotates the password."
• #923: "Deleting the Secret makes a new password, which Owns brings about within a reconcile. That doubles as the walkthrough's rotation."
• story(example): run the controller's shared cases in process, against k3s and PostgreSQL in containers #1132's Rotation case: "deleting the Secret brings a new one, whose password connects, and the old password doesn't."

A fix changes the walkthrough's controller package and its Reconciling page. It changes no exported humus name, and the walkthrough isn't released.

Is it reproducible?

Yes. At 02f0286, with Go 1.27.0, client-go v0.37.1, rancher/k3s:v1.37.1-k3s1, postgres:18.0 and testcontainers-go v0.44.0 on Podman 5.8.9 (crun 1.28). Each test ran 4 times under -race.

Scenario	Harness	Runs failed
Deleted after the in-flight reconcile read the Secret	unit, kubefake	4 of 4
Deleted while the in-flight reconcile provisions the role	unit, kubefake	4 of 4
Deleted 1s after Ready, while the next reconcile waits to provision	k3s and PostgreSQL	4 of 4
Deleted once the reconciles have settled	k3s and PostgreSQL	0 of 4
The shared Rotation case, alone (baseline)	k3s and PostgreSQL	0 of 4

In both unit tests, all three Provision calls got the first password, and the Secret held it at the end.

Deleted while the next reconcile waits to provision: the Secret comes back with the old password. The Database carries a scenario that delays postgresql:provision by 3s, so each reconcile holds the Secret it read for 3s. The test deletes the Secret 1s after it sees Ready. From run 1, the test's watches on the Secret and the Database, and the reconcile spans:

1. The first reconcile applies the Secret, then the status: 19.124 Secret ADDED uid d68338d4, 19.128 Database MODIFIED … last written by databases-controller Apply status.
2. Those events queue the next reconcile, which reads the Secret at once: reconcile 19.128 to 22.146, GET the Secret at 19.128.
3. The test deletes the Secret: 20.343 Secret DELETED uid d68338d4.
4. The in-flight reconcile sets the old password on the role, then applies the Secret, which creates it again: ALTER ROLE at 22.137, PATCH the Secret at 22.140, 22.143 Secret ADDED uid 99d4f288, last written by databases-controller Apply.
5. The deletion's reconcile runs only after that, and finds the new Secret: reconcile 22.146 to 25.163, GET the Secret at 22.147. It keeps the old password, and so does the one after it.
6. Nine seconds later the Secret is still uid 99d4f288, with the old password.

All four runs showed the same order.

Deleted once the reconciles have settled: the password rotates. The same scenario, with the delete 9s after Ready. The deletion's reconcile read no Secret and made a new password, in all four runs. So the trigger is the in-flight reconcile, not the delay.

The report is right about the mechanism, and its reproduction is the first unit scenario. Three things it didn't say:

• The role is rotated back too. The in-flight reconcile sets the old password on the role before it applies the Secret (step 4). So the role and the Secret agree, and nothing fails to connect. The rotation is lost silently, behind a Secret with a new UID. Reproduced.
• The window is the whole reconcile, not only its tail. Deleting the Secret while the role is provisioned loses the rotation too. Reproduced in the unit test. Against PostgreSQL, Provision is in the window, so a slow PostgreSQL widens it.
• More reconciles follow Ready than the report counts. In the settled scenario, all four runs showed four reconciles after the finalizer's, where kube's docs count two, and the second wrote the status again (47.898 Database MODIFIED … Apply status in run 1). Each one holds the Secret it read. I didn't look into why the status was written twice: it isn't this bug's cause.

Why

The Secret is the password's source of truth, and Reconcile reads it once, then writes back what it read:

1. password gets the Secret from the API server, and keeps the password in it. Reasoned from code, and reproduced.
2. applySecret applies the Secret with server-side apply, forced. An apply creates an object that doesn't exist. So when the Secret was deleted after step 1, the apply makes a new one, with a new UID and the old password. Reproduced: Secret ADDED uid 99d4f288, last written by databases-controller Apply.
3. Init has the Controller own Secrets, so the deletion queues a reconcile. But the key is being processed, so client-go's queue holds it until the in-flight reconcile is Done (Add, Done). Reasoned from code, and reproduced by the spans: the deletion's reconcile started at 22.146, the moment the in-flight one ended.
4. That reconcile finds the Secret from step 2, so step 1 keeps its password. Every later reconcile does the same. Reproduced.

In CI, the cases run in parallel under -race, and the reconciles that follow Ready read the Secret within milliseconds of the case's delete. That explains the 1 in 20, but I got it from the timings above and didn't reproduce CI's load. Running the Rotation case alone passed 4 of 4.

#1251 is filed to fix it:

• deleting a Database's Secret rotates its password whenever the delete lands, and a reconcile never brings back a deleted Secret's password;
• the triage's unit tests, and its container tests against k3s and PostgreSQL, show it;
• the shared Rotation case passes under the testcontainers and kind jobs;
• the Reconciling page describes the fixed steps.

[Zaba505]

Fixed by #1251, in #1252 (merged as 51986b1).

The Reconciler's apply of the Secret now names the UID of the Secret it read. The API server creates a Secret an apply names that doesn't exist, but not one whose UID it names. So once that Secret is deleted, the apply fails with a conflict instead of bringing it back with the old password. On that conflict the reconcile rotates there and then: a new password, set on the role, then a new Secret. The deletion's own reconcile then keeps the new password. The triage's unit and k3s reproductions are in controller/reconciler_test.go and app/app_testcontainers_test.go, and the Reconciling page describes the new steps.