Skip to content

docs(seal-check): record k3d/k3s NetworkPolicy substrate verification (backend#1184) - #504

Merged
LukasWodka merged 2 commits into
developfrom
docs/1184-record-k3d-netpol-verify
Jul 31, 2026
Merged

docs(seal-check): record k3d/k3s NetworkPolicy substrate verification (backend#1184)#504
LukasWodka merged 2 commits into
developfrom
docs/1184-record-k3d-netpol-verify

Conversation

@LukasWodka

@LukasWodka LukasWodka commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Records the k3d/k3s NetworkPolicy substrate verification that RFC-0003 §8.4 explicitly demands ("do not assume k3d enforces NetworkPolicy"). Docs-only — one file.

What was run

On a throwaway k3d v5.8.3 cluster (k3s v1.33.6+k3s1), a standalone deny-egress NetworkPolicy (podSelector on a probe pod, policyTypes: [Egress], empty egress:) was applied and a curl from the pod to 1.1.1.1:443 was measured across three states:

state result
no policy (baseline) reachable — HTTP 301
deny-egress policy applied BLOCKED — connect failure (egress dropped)
policy removed (control) reachable again — HTTP 301

The reachable → blocked → reachable transition makes the block attributable to the policy, not a fluke or an image-pull artifact. So k3s's embedded (kube-router) controller does enforce egress NetworkPolicy on this k3d version — flannel handles pod networking, but the netpol controller is active and enforcing.

Honest scope (deliberately precise)

This verifies the substrate primitive — the specific doubt §8.4 raises. It is not the full-chart egress-enforcement probe run against a deployed release; that end-to-end run stays open under backend#1184, and I've kept the §8.3 k3d cell reading "substrate verified; full-probe run pending" rather than overclaiming "sealed".

Why it matters

The chart's egress-enforcement seal-check relies on the CNI actually dropping egress. That reliance was documented-but-unverified ("expected is not verified"). It's now verified for the k3d/k3s substrate, so the local-dev "nothing gets out" guarantee rests on measured behavior, not an assumption.

Epic: backend#1151 · Ticket: backend#1184 (D12)

🤖 Generated with Claude Code


Note

Low Risk
Documentation-only change to seal-check status and runbook text; no runtime, chart, or security behavior changes.

Overview
docs/SEAL-CHECK.md now records a completed k3s egress NetworkPolicy substrate check on k3d v5.8.3 / k3s v1.33.6+k3s1 (2026-07-30): deny-egress policy produced reachable → blocked → reachable again on curl to 1.1.1.1:443, with that evidence centralized in the §8.4 Status note.

The §8.3 k3d matrix cell and follow-ups are updated to substrate verified; full-chart egress-enforcement probe still pending, so k3d remains unsealed for the egress guarantee until the deployed-release runbook (steps 1–4) is recorded—without overclaiming sealed status.

Reviewed by Cursor Bugbot for commit b30bdf9. Bugbot is set up for automated code reviews on this repo. Configure here.

… (backend#1184)

RFC-0003 §8.4 says 'do not assume k3d enforces NetworkPolicy'. Ran the
substrate check on a throwaway k3d v5.8.3 / k3s v1.33.6+k3s1 cluster: a
standalone deny-egress NetworkPolicy on a probe pod took a curl to
1.1.1.1:443 reachable -> BLOCKED -> reachable-again-after-removal
(HTTP 301 -> connect failure -> HTTP 301). The block is attributable to
the policy, so k3s's embedded kube-router controller does enforce egress
NetworkPolicy on this k3d version — the substrate doubt is resolved.

Records the result in the §8.4 runbook + the follow-ups list; the §8.3 k3d
cell now reads 'substrate verified; full-probe run pending'. Recording the
full-chart egress-enforcement probe run against a deployed release stays
open (still backend#1184). Docs-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

👋 Heads-up — Code review queue is at 33 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

@saqlainsyed007 saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs-only review of docs/SEAL-CHECK.md. The experiment itself is sound and honestly scoped (curl to an IP so no DNS is needed; reachable→blocked→reachable proves attribution). But the change asserts an edit to the §8.3 matrix cell that it never actually makes, leaving the document self-contradictory. Details inline.

🤖 Generated with Claude Code

Comment thread docs/SEAL-CHECK.md Outdated
Comment thread docs/SEAL-CHECK.md Outdated
Comment thread docs/SEAL-CHECK.md
…ain #504)

- Update the §8.3 k3d matrix cell (was 'verification run pending; do not
  assume') to 'Substrate verified; full-probe run pending', matching the
  prose that cross-references it — the concrete edit the PR promised but
  never made (Saqlain #i).
- Status note now LEADS with 'still UNSEALED for the egress guarantee on k3d
  until the full-chart probe is recorded', honoring this doc's
  unsealed-never-silently-sealed philosophy — a skimmer can no longer read
  the bold VERIFIED as 'sealed' (Saqlain #ii).
- State the run evidence (versions + reachable->blocked->reachable) ONCE in
  the status note; the post-runbook paragraph, the follow-ups bullet, and the
  matrix cell now reference it instead of restating — which is how the cell
  drifted out of sync in the first place (Saqlain #iii).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka

Copy link
Copy Markdown
Contributor Author

Thanks — all three addressed in b30bdf9:

  • #i — updated the §8.3 k3d matrix cell to "Substrate verified; full-probe run pending", matching the prose that cross-references it. That was the concrete edit the PR promised but never made — the document is no longer self-contradictory.
  • #ii — the Status note now leads with "still UNSEALED for the egress guarantee on k3d until the full-chart probe is recorded", so the bold VERIFIED can't be skimmed as "sealed". The distinction (probe-pod netpol, not the chart's training-labelled selector) is stated up front.
  • #iii — the run evidence (versions + reachable→blocked→reachable) is now stated once in the Status note; the post-runbook paragraph, the follow-ups bullet, and the matrix cell reference it — which is how the cell drifted in the first place.

@saqlainsyed007 saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three review findings addressed in b30bdf9:

  1. §8.3 k3d matrix cell now reads "substrate verified; full-probe run pending" — matches the prose that references it; self-contradiction resolved.
  2. Status block now leads with the explicit "still UNSEALED for the egress guarantee on k3d" framing, restoring the doc's "never silently sealed" philosophy.
  3. Evidence now lives in a single record (the §8.4 Status note); the post-runbook paragraph and follow-ups reference it instead of restating.

Clean, root-cause fixes.

🤖 Generated with Claude Code

@LukasWodka
LukasWodka merged commit 5267bac into develop Jul 31, 2026
10 checks passed
@LukasWodka

Copy link
Copy Markdown
Contributor Author

/fr-pass

LukasWodka added a commit that referenced this pull request Aug 3, 2026
…184) (#541)

* test(seal-check): run the egress-enforcement probe live in k3d CI (#1184)

RFC-0003 D12: the chart's enforcement probes ship as `helm.sh/hook: test`
Jobs, but `helm test` ran nowhere in CI — SEAL-CHECK §8.4 recorded the k3s
NetworkPolicy substrate as verified (#504) while the full-chart egress probe
run stayed "pending". This closes that gap.

- scripts/tests/e2e-seal-check.sh: install the local chart on a real k3d
  cluster with public images + the egress lockdown engaged
  (allowExternalHttps=false), then `helm test --filter` the
  egress-enforcement seal-check. Requires BOTH a zero exit AND the probe's
  `OK  egress lockdown verified` marker in the logs — guarding the
  helm-test-`--filter`-matches-nothing silent-pass trap.
- .github/workflows/helm-ci.yaml: a `seal-check-e2e` job mirroring
  upgrade-e2e (stock ubuntu runner, zero secrets — public curl vs 1.1.1.1).
- client/tests/egress_enforcement_check_test.yaml: pin the probe Job's
  metadata.name so the e2e --filter can never silently drift off it.

Local: shellcheck clean · helm template renders the probe Job · helm-unittest
27 suites / 320 tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(seal-check): install k3d/helm/kubectl before create_cluster (exit 127)

The sourced libs define install_kubectl/install_k3d/install_helm but do not
call them; create_cluster + helm need the binaries on PATH first, and a stock
runner has none preinstalled. Mirror e2e-auto-upgrade.sh's prerequisite block
+ the post-create node-ready wait.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(seal-check): drive helm test off exit code, not --logs (Job hook)

The egress-enforcement probe is a Job-type test hook with hook-delete-policy
hook-succeeded. `helm test --logs` looks up the pod by the Job's bare name,
but a Job's pod has a generated suffix ("pods not found"), and Helm deletes
the Job on success anyway — so --logs errored even though the probe passed.

Drive off the exit code instead (the probe exits 0 only when egress is
verified blocked), guard the --filter-silent-pass by asserting the hook is in
`helm get hooks` first, and dump the pod log via kubectl only on failure
(the Job persists when it fails).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(seal-check): address Bugbot review (autostart, paths, shellcheck)

Three Bugbot findings on the new e2e seal-check:

- Medium: set TRACEBLOC_NO_AUTOSTART=1 before create_cluster (like the sibling
  e2e-*.sh) so it never mutates the host's Docker restart policy / runs
  `systemctl enable docker`.
- Medium: add scripts/tests/e2e-seal-check.sh to helm-ci.yaml on.push/
  on.pull_request paths so script-only edits re-trigger the k3d job.
- Low: enumerate the script in the installer-tests ShellCheck gate (both the
  error and warning passes), matching the other e2e entrypoints.

The High finding ("Job logs never reach marker check") was already resolved in
the prior commit — the script drives off `helm test`'s exit code and no longer
greps --logs output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(seal-check): avoid SIGPIPE/pipefail false-fail in hook guard (Bugbot)

`helm get hooks | grep -q` lets grep close the pipe on first match, SIGPIPE-ing
helm mid-write; under set -o pipefail that false-fails the guard even when the
hook exists. Capture helm output to a var, then grep a here-string — no pipe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci(seal-check): bump job timeout 20m->30m (Bugbot)

The script's own bounds (create_cluster up to 15m + helm test 360s + tool
install + helm install) can exceed a 20m GHA cap on a slow cluster bring-up,
false-failing even while each component is inside its own timeout. Match the
sibling k3d job (upgrade-e2e = 30m).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(seal-check): positive control + probe timeout + lib paths + NS (Saqlain)

- Positive control (blocking): before trusting a BLOCKED probe, prove the
  cluster can REACH the host from a non-policied pod (in `default`, ungoverned
  by the namespace-scoped training-egress policy). Without it, egress failing
  for an unrelated reason (runner firewall / target outage / rate-limit) would
  make the probe print OK and pass green while the NetworkPolicy did nothing.
  A reachable positive + a blocked training pod = the block is attributable to
  the policy. Positive-control failure now fails the seal-check as inconclusive.
- Bump enforcementProbeTimeoutSeconds 60s->240s (blocking): on a cold GHA
  runner k3s can take >60s to program the pod iptables while the chart installs;
  the probe is single-shot (backoffLimit 0), so 60s false-fails. 240s is well
  inside the 360s helm-test budget.
- Add scripts/lib/** to helm-ci paths (blocking): the script sources
  scripts/lib/{common,setup-linux,cluster,preflight}.sh, so a lib-only edit must
  re-trigger seal-check-e2e + upgrade-e2e (both depend on it).
- Derive NS from CLUSTER_NAME so a CLUSTER_NAME override isolates a run under
  one name instead of desyncing cluster vs release/namespace.

shellcheck clean; workflow parses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(seal-check): --tlsv1.2 (house rule) + SA wait + required Lint list (Bugbot)

- Add --tlsv1.2 to the positive-control curl — the curl-tls house rule (the
  required quality/house-rules gate) rejects a curl that could negotiate a
  downgraded TLS version.
- Wait for the default ServiceAccount before the positive-control kubectl run:
  a fast runner can schedule the pod before the SA exists, aborting under set
  -e with 'serviceaccount default not found' before the attribution message.
- Add e2e-seal-check.sh to the standard-checks.yml Lint shellcheck list (the
  REQUIRED branch-protection gate) — installer-tests had it, the required Lint
  did not, so a shellcheck regression in the new script could miss the gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants