Skip to content

test(e2e): full seal suite vs the dev backend (backend#1184 fast-follow) - #598

Merged
LukasWodka merged 3 commits into
developfrom
feat/1184-full-seal-dev-harness
Aug 5, 2026
Merged

test(e2e): full seal suite vs the dev backend (backend#1184 fast-follow)#598
LukasWodka merged 3 commits into
developfrom
feat/1184-full-seal-dev-harness

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What (the backend#1184 deferred fast-follow)

backend#1184 closed accepting CI-substrate coverage as v1-sufficient, with one recorded residual: "the FULL suite (incl. backend-reachability + bound-PVC storage-assertions) on the dev-env e2e-test-agent with real credentials." This is that harness.

scripts/tests/e2e-full-seal.sh (new)

Real k3d cluster → install the working-tree chart as the dedicated dev e2e-test-agent client (real creds, CLIENT_ENV=dev, lockdown engaged, same base profile as the sibling) → crisp pre-asserts (every release PVC Bound within 300s — separates "slow to bind" from "assertions wrong"; mysql + jobs-manager rolled out, i.e. a real backend session) → shared egress positive control → hook-presence guard for all three checks (an unfiltered helm test "passes" a release whose hooks silently stopped rendering — same trap the sibling guards for its --filter) → helm test unfiltered: egress-enforcement + backend-reachability + storage-assertions in one release. On failure, every seal-check pod log dumps via the §8.2 enumeration label.

full-seal-e2e job (helm-ci)

  • push / workflow_dispatch only, never PRs — fork PRs can't read secrets, and a real dev-backend login per PR is platform churn for no extra signal (the secret-free enforcement probe keeps covering PRs).
  • Skips green with a ::notice until provisioned — the job is live the moment the secrets exist, no second PR needed.
  • Job-level concurrency (one dev-agent session at a time — two clusters on one client id reads as a capacity anomaly).
  • workflow_dispatch added to helm-ci for on-demand runs (e.g. right after provisioning).

Refactor + docs

  • The egress positive control moved verbatim from e2e-seal-check.sh into lib/e2e-common.sh (provenance + test(seal-check): run the egress-enforcement probe live in k3d CI (#1184) #541 review rationale documented at the function) — one copy, two callers, no drift.
  • SEAL-CHECK.md: new CI-coverage map (what runs where, activation contract, "never a person's shared dev identity — the backend#1180 token-churn failure class"), and the follow-ups list drops three items that shipped since it was written (test(seal-check): run the egress-enforcement probe live in k3d CI (#1184) #541 live probe, cli#393 CLI verdict, cli#449 §8.3 matrix).
  • Both installer-tests.yaml shellcheck gates now include the new script; helm-ci path filters gain the new script + lib/e2e-common.sh (which was already load-bearing for the seal job but absent from the filter).

⚠️ Activation — the one human step (not in this PR)

Provision a dedicated dev-platform client (e2e-test-agent) and add two repo Actions secrets: TB_E2E_CLIENT_ID, TB_E2E_CLIENT_PASSWORD. Until then the job skips green by design. (Public repo: fork PRs can't read the secrets; the job never runs on PRs anyway.)

Test plan

  • bash -n + shellcheck --severity=error green on all three touched scripts (both CI shellcheck lists updated).
  • Workflow YAML parse-validated; R8 manifest untouched by design (scripts/tests/ isn't manifest-covered); style guard clean.
  • The refactored sibling (seal-check-e2e) runs in THIS PR's CI — proving the factored positive control live on k3d.
  • The new job's full path can't run pre-provisioning by construction; first green run gets recorded in SEAL-CHECK.md per the doc.

Epic: tracebloc/backend#1151 (workstream G residual) · closes the backend#1184 fast-follow · RFC-CLI-0003 D12 §8.2

🤖 Generated with Claude Code


Note

Low Risk
Changes are CI/e2e harness and documentation only; no chart runtime or production install path. Real credentials stay in Actions secrets and a dedicated dev test client, with PRs excluded from the credentialed job.

Overview
Adds e2e-full-seal.sh and a helm-ci full-seal-e2e job to run the complete chart seal suite (egress-enforcement, backend-reachability, storage-assertions) on k3d against the dev backend with dedicated e2e-test-agent credentials—not just the secret-free egress probe PRs already get.

The new script installs the working-tree chart with real creds in a mode-0600 values file, waits for PVCs to bind and for jobs-manager rollout, verifies all three test hooks exist (guards against silent empty helm test), then runs unfiltered helm test with failure log dumps via the seal-check label.

CI wiring: workflow_dispatch on helm-ci; path filters include the new script and e2e-common.sh; installer shellcheck lists updated. The job runs only on push/dispatch (not PRs), uses 45m timeout and concurrency so one dev-agent session runs at a time, and skips green with a notice until TB_E2E_CLIENT_ID / TB_E2E_CLIENT_PASSWORD secrets exist.

Refactor: egress positive-control logic moves from e2e-seal-check.sh into e2e_egress_positive_control in e2e-common.sh for both seal e2e scripts.

Docs: SEAL-CHECK.md gains a CI coverage map and trims follow-ups that shipped elsewhere.

Reviewed by Cursor Bugbot for commit d639aca. Bugbot is set up for automated code reviews on this repo. Configure here.

… fast-follow

New e2e-full-seal.sh installs the working-tree chart on real k3d as the
dedicated dev e2e-test-agent (real credentials, CLIENT_ENV=dev), waits for
every release PVC to Bind and jobs-manager to hold a real backend session,
then runs helm test UNFILTERED — egress-enforcement + backend-reachability
+ storage-assertions in one release, hook-presence-guarded so a regated
check can't vanish silently. New helm-ci job full-seal-e2e runs it on
push/dispatch only and skips green with a notice until the e2e-test-agent
secrets (TB_E2E_CLIENT_ID / TB_E2E_CLIENT_PASSWORD) are provisioned.

The egress positive control moves verbatim into e2e-common.sh (one copy,
shared with e2e-seal-check.sh). SEAL-CHECK.md gains the CI-coverage map and
drops three follow-ups that shipped since (#541 live probe, cli#393
verdict, cli#449 matrix).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka LukasWodka self-assigned this Aug 5, 2026
@LukasWodka

Copy link
Copy Markdown
Contributor Author

bugbot run

Comment thread scripts/tests/e2e-full-seal.sh
Comment thread scripts/tests/e2e-full-seal.sh Outdated
Comment thread .github/workflows/helm-ci.yaml Outdated
- credentials travel in a mode-0600 temp values file, never on argv
  (process-list exposure on a shared runner + helm --set comma/brace
  mangling); removed on every exit path — installer parity
- the PVC wait does one guarded fetch per iteration, so a transient
  kubectl failure retries until the deadline instead of aborting under
  set -euo pipefail
- full-seal-e2e job timeout 30m -> 45m: the script stacks a 300s PVC wait,
  two 300s rollouts and a 600s unfiltered helm test on create_cluster's
  15m bound — GHA must not kill a slow-but-healthy run

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

All three findings addressed: creds now travel in a 0600 temp values file (never argv — and immune to helm --set's comma/brace mangling), the PVC wait retries through kubectl blips instead of aborting under set -euo pipefail, and the job cap is 45m with the summed script bounds documented.

bugbot run

Comment thread scripts/tests/e2e-full-seal.sh
…ull-seal install (Bugbot)

The chart's PVC default is ReadWriteMany; rancher.io/local-path never
provisions it, so every claim sat Pending and both the Bound pre-wait and
storage-assertions were guaranteed to fail once the job activates. The
installer writes pvcAccessMode: ReadWriteOnce for this exact storage path
— the harness now installs with the same value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

Second-round finding fixed: the install now pins pvcAccessMode=ReadWriteOnce (chart default is RWX, which local-path never provisions — the Bound pre-wait and storage-assertions were structurally unable to pass; installer parity for this storage path).

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d639aca. Configure here.

Comment thread scripts/tests/e2e-full-seal.sh
@LukasWodka
LukasWodka merged commit c6c7251 into develop Aug 5, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants