feat(CC-0102): test(e2e): Cover failure & recovery paths for external depen

[berendt]

GitHub Issue #315: test(e2e): Cover failure & recovery paths for external dependencies (#277 Phase 1)

Context

Parent: #277, Phase 1 → Failure / recovery paths.

The existing failure-path coverage is limited:

• tests/e2e/keystone/missing-secret/ (CC-0016) covers absence of the referenced Secret refs but not transient backend outages.
• tests/e2e-chaos/ covers pod-kill and network-partition for MariaDB, Memcached and the operator, but does not exercise OpenBao seal/policy or ESO downtime.
• tests/e2e/keystone/release-upgrade/ covers the happy-path 2025.2 → 2026.1 upgrade but does not assert the failure branch in reconcileDatabase (reconcile_database.go:563/566, conditionReasonExpandFailed).

This sub-issue closes those four gaps.

Scope — new test scenarios

Each row describes one Chainsaw test directory. Splitting per row keeps parallel: 4 execution
balanced and keeps catch-blocks scoped.

#	Test directory	Scenario	Expected status	Recovery step
1	tests/e2e/keystone/openbao-sealed/	Apply CR → Ready=True. Then seal all three OpenBao replicas (kubectl exec openbao-{0,1,2} -n openbao-system -- bao operator seal). Within RequeueSecretPolling, the ClusterSecretStore flips to Ready=False and reconcileSecrets (reconcile_secrets.go:66) sets SecretsReady=False, reason=SecretStoreNotReady	(conditions[?type=='SecretsReady']).status == \"False\" && .reason == \"SecretStoreNotReady\"; aggregate Ready=False, reason=NotAllReady	Unseal all three pods using the three threshold keys from Secret openbao-system/openbao-init-keys (mirroring deploy/openbao/bootstrap/init-unseal.sh:135-140); assert Ready=True, reason=AllReady returns within 2 min
2	tests/e2e/keystone/openbao-policy-revoked/	Apply CR → Ready=True. Revoke the keystone-write policy in OpenBao via bao policy delete keystone-write. Wait for the next Fernet rotation tick (forced via kubectl create job --from=cronjob/<name>-fernet-rotation manual-rotation); the staging-Secret push is rejected by OpenBao with HTTP 403	The PushSecret <name>-fernet-keys-backup reports a Failed condition; the operator emits a Warning event with reason matching PushSecretError or surfaces 403 in the message; the CR's SecretsReady stays True (cluster store is healthy) but the rotation-side condition flips	Re-write the policy via bao policy write keystone-write - (verbatim copy from bootstrap script); assert PushSecret Ready=True returns
3	tests/e2e/keystone/eso-down/	Scale external-secrets Deployment in external-secrets namespace to 0 replicas. Apply a fresh Keystone CR (no DB/admin Secrets pre-materialized in the test namespace). Within RequeueSecretPolling, reconcileSecrets sees WaitForExternalSecret == false and sets SecretsReady=False, reason=WaitingForDBCredentials	(conditions[?type=='Ready']).status == \"False\" for at least 60 s of sustained polling — proves the operator does not flip to Ready spuriously while the upstream syncer is absent	Scale ESO back to 1 replica; assert Ready=True within 3 min
4	tests/e2e/keystone/db-expand-failure/	Apply CR → Ready=True. Patch spec.image.tag to a tag whose keystone-manage db_expand is guaranteed to fail in this kind cluster — concretely a tag that does not exist in the test image set (e.g. bogus-tag). The Deployment will not roll because the bootstrap chain blocks on the expand Job. The expand Job goes Failed; reconcile_database.go:563-566 records db-expand as failed via recordDBJobTerminalState and emits Warning ExpandFailed. The status reports DatabaseReady=False, reason=ExpandFailed	DatabaseReady=False, reason=ExpandFailed; aggregate Ready=False; a Warning event with reason ExpandFailed exists	Patch spec.image.tag back to the original (pre-failure) tag. The operator's idempotent expand-Job naming (<name>-db-expand) means the failed Job needs an explicit kubectl delete job <name>-db-expand before the next reconcile re-creates it. Assert Ready=True, reason=AllReady returns

Out of scope (prerequisites, separate tickets)

None — every cited code path exists and is exercised by unit/integration tests today
(reconcile_secrets_test.go, reconcile_database_test.go:1098-1180,
keystone_controller_test.go cluster-store NotReady cases). The four scenarios are pure E2E
gaps.

Test design

Each row gets its own directory with the three-file convention used elsewhere in the suite:

tests/e2e/keystone/openbao-sealed/
├── 00-keystone-cr.yaml      (basic-deployment fixture)
├── chainsaw-test.yaml       (apply, seal, assert False, unseal, assert True)
└── seal-all.sh / unseal-all.sh   (script files referenced via try.script.content)

Pattern decisions:

• Sealing only one of three pods is insufficient — Raft elects a new leader, and the CSS stays Ready. The test must seal all three replicas. The seal-all.sh helper iterates openbao-{0,1,2} and runs bao operator seal (no key needed; root-token-authenticated command).
• Unseal keys: read from Secret openbao-system/openbao-init-keys, key init-output, parsed via jq -r '.unseal_keys_b64[0..2]' — reuse the exact path from init-unseal.sh:131.
• Policy file: dump the current keystone-write policy via bao policy read keystone-write in a setup step before deletion, restore it from that captured payload. Avoids drift between test fixture and production policy.
• db-expand test: the cleanup step kubectl delete job <name>-db-expand is deliberate, not a test-quality compromise — it mirrors the operator's "Job-as-side-effect" semantics. The release-upgrade test (tests/e2e/keystone/release-upgrade/chainsaw-test.yaml) already uses this idiom for the success path.
• catch blocks: each test ships the same catch: block as events/chainsaw-test.yaml (pod logs, job logs, events) so CI failure diagnostics stay homogeneous.

All four tests honour tests/e2e/chainsaw-config.yaml: 30 s apply / 5 m assert / 30 s delete /
3 m cleanup. parallel: 4 is fine because each test owns its own namespace and the openbao
seal/unseal ops are idempotent under the post-test recovery step (the test always leaves the
cluster in the same state it found it).

Definition of Done

• All four chainsaw test ... directories green locally against make kind-up
• make e2e green; two consecutive parallel: 4 CI runs without new flakes
• OpenBao state is restored to "all unsealed, policy intact" at end of every test (cleanup block); a soak run with chainsaw test --selector phase1-failure repeated 3× must keep all later tests green
• No new infra in deploy/kind/; no chart changes; no operator code changes
• Parent Meta: Expand E2E test coverage for Keystone operator #277: all four covered checkboxes ticked
• Commit prefix test(CC-xxxx) once a feature code is assigned

Risks / open questions

• Test isolation under parallel: 4: sealing OpenBao globally affects every other Keystone CR running in parallel. The openbao-sealed test must run with a Chainsaw label that excludes it from the parallel pool (e.g. serial: openbao-global), or every Phase 1 failure test must carry a synchronisation label. Recommend modelling after the existing chaos-mesh tests in tests/e2e-chaos/ which serialise via concurrent: false. Please confirm the right knob in review.
• Image tag for db-expand failure: a bogus tag triggers ImagePullBackOff before the expand Job runs, which surfaces a different condition reason. The cleaner approach is to patch the Job spec to inject a failing command; needs validation. Alternative: use a real but incompatible tag (downgrade from 2026.1 to 2025.1) which is rejected by the upgrade matrix. Pick during implementation.
• PushSecretError reason name: reconcile_secrets.go does not emit a typed event for PushSecret-side failures today; the test asserts the symptom (PushSecret status Failed plus message containing 403) rather than a specific reason string. If review prefers a typed event, that becomes a small feat(CC-xxxx) follow-up — out of scope for this E2E sub-issue.

---

Source: #315