fix(cilium): make Ceph host policy select host endpoints

[Aviator-Coding]

Intent

Fix a silent-fail defect in the just-merged stage-1 Ceph LAN isolation host policy (PR #1417), found while running the step-4 post-merge verification.

CONTEXT: stage 1 enabled Cilium hostFirewall with global policyAuditMode (nothing enforces) plus a deny-only CiliumClusterwideNetworkPolicy 'ceph-lan-isolation' covering the LAN-exposed host-network Ceph daemon ports. The staged plan is: merge stage 1 in audit mode, observe audit verdicts over a real workload window, then a separate stage-2 PR flips audit off to enforce. Stage 2 is NOT part of this change and must stay blocked.

THE DEFECT: the policy shipped with nodeSelector 'kubernetes.io/os: linux'. Cilium's default label filter strips that label from the HOST ENDPOINT's label set (which is not the same as the node's label set), so the selector matched nothing and the policy was inert on all three nodes - the Ceph daemons remained LAN-exposed. This fails silently AND open: the CCNP reports VALID:True, the agent loads the rule, but the host endpoint reports policy-enabled:none and 'cilium-dbg bpf policy get' shows zero Deny rows. Everything looks healthy except the thing that matters. Had this not been caught, a stage-2 enforce PR would have shipped an inert policy while everyone believed Ceph was protected.

MEASURED EVIDENCE (talos-1/2/3, identical): node labels DROPPED from the host endpoint are beta.kubernetes.io/arch, beta.kubernetes.io/os, kubernetes.io/arch, kubernetes.io/hostname, kubernetes.io/os. Labels that SURVIVE: node-role.kubernetes.io/control-plane, topology.kubernetes.io/region, topology.kubernetes.io/zone, extensions.talos.dev/*, intel.feature.node.kubernetes.io/gpu. Note the upstream Cilium host-policy docs example uses kubernetes.io/hostname, which is subject to the same filter.

THE FIX: nodeSelector now uses matchExpressions with key node-role.kubernetes.io/control-plane operator Exists. Chosen because it is empirically present on all three host endpoints. I deliberately did NOT use an empty nodeSelector: an empty selector is undocumented for host policies (the upstream docs only show explicit custom labels) and I would not rely on undocumented behavior for a safety-critical control. The known tradeoff, called out in a comment: a future non-control-plane node would not be covered and its Ceph daemons would stay exposed. That is an accepted, documented coverage assumption for this 3-node all-control-plane cluster, not an oversight.

PROOF THE FIX WORKS (this is the point - the previous selector was shipped on reasoning alone and was wrong): I applied a temporary probe CCNP carrying the new selector with a deny on port 7000 only. Port 7000 has no listener (ceph mgr services lists only prometheus; the dashboard is disabled) and it sits below the ephemeral range, so the probe provably could not drop real traffic, and global audit mode was active besides. Result on all three nodes: 'Deny Ingress reserved:world 7000/TCP' installed (fromEntities:world expands to reserved:world, reserved:world-ipv4, reserved:world-ipv6) AND the catch-all 'Allow Ingress ANY' row survived - live confirmation that enableDefaultDeny:{ingress:false,egress:false} keeps the policy subtractive and cannot lock the nodes out. The probe was deleted immediately and the cluster verified back to baseline (only the Flux-managed policy remains, no Deny rows).

ALSO IN THIS CHANGE: audit plan section 2c is upgraded from a one-way check to an explicit two-way gate - Deny rows MUST be present (their absence means an inert policy) and the Allow Ingress ANY catch-all MUST survive (its absence means default-deny took hold and is a revert-immediately condition). An AGENTS.md ANTI-PATTERNS entry records the host-endpoint label filter trap so a future session does not repeat it.

STAGE-1 ROLLOUT VERIFICATION ALREADY PASSED and is unchanged by this PR: Flux reconciled to the merge commit, cilium-config shows enable-host-firewall=true and policy-audit-mode=true, all three agents rolled one node at a time under maxUnavailable:1 with 0 restarts and ready=3, and no BPF verifier or 'program too large' errors (the cilium/cilium#38967 risk did not materialise on 1.18.6).

SCOPE: this PR only makes the policy actually select the host endpoint. It does not enforce anything - global audit mode remains on, so nothing can drop a packet. The real audit observation window can only begin once this merges, because until now there was nothing to observe. Do not treat the absence of captured verdict evidence or of the stage-2 enforce flip as incompleteness; both are deliberately deferred and gated on captain review.

What Changed

• Switched the ceph-lan-isolation CCNP nodeSelector from kubernetes.io/os: linux (stripped from host endpoints by Cilium’s default label filter, so the policy matched nothing) to node-role.kubernetes.io/control-plane Exists, with measured host-endpoint label sets and the all-control-plane coverage assumption documented on the policy.
• Upgraded audit-plan §2c to a two-way post-merge gate (Deny rows must be present and Allow Ingress ANY must survive) and recorded the host-endpoint label-filter trap in AGENTS.md.
• Added scripts/ci/hostpolicy-ceph-selector-test.py to offline-assert the shipped selector matches measured host-endpoint labels, the broken selector does not, and the subtractive/audit-mode safety contract still holds.

Risk Assessment

✅ Low: Minimal nodeSelector fix with enableDefaultDeny left false and global policyAuditMode still true, so the change cannot drop traffic while making the host policy actually attach under the documented all-control-plane coverage assumption.

Testing

No live kubeconfig in this worktree, so cilium-dbg BPF Deny-row verification could not be re-run here (author probe + deferred §2c window per intent). Added and ran an offline host-endpoint label-selector regression that reproduces the inert baseline selector, passes the fixed control-plane Exists selector on all three measured host endpoints, keeps subtractive defaults and audit mode on, and confirms audit-plan §2c is a two-way gate; overall PASS.

Evidence: Selector regression transcript (baseline inert vs fixed match)

Ceph LAN isolation host-policy selector fix — offline regression evidence
========================================================================

Environment: no kubeconfig / no live cluster in this worktree.
Live BPF probe (Deny 7000 + Allow ANY) was author-run pre-PR; intent
explicitly defers post-merge §2c observation. This run proves the
silent-open selector class of failure offline via host-endpoint label
matching (the mechanism of the defect).

1) BASELINE (commit a0bb6df9) — reproduces inert policy
   nodeSelector: {"matchLabels": {"kubernetes.io/os": "linux"}}
   uses stripped labels: ['kubernetes.io/os']
   matches nodes:          ['talos-1', 'talos-2', 'talos-3']
   matches host endpoints: []  ← inert
   result: FAIL_AS_EXPECTED

2) FIXED (kustomize-built ceph-lan-isolation)
   nodeSelector: {"matchExpressions": [{"key": "node-role.kubernetes.io/control-plane", "operator": "Exists"}]}
   enableDefaultDeny: {'egress': False, 'ingress': False}
   fromEntities: ['world']
   matches host endpoints: ['talos-1', 'talos-2', 'talos-3']
   broken selector still matches 0 host endpoints: []
   result: PASS

3) NEGATIVE — inject kubernetes.io/os=linux into fixed tree
   failures: ["nodeSelector uses host-endpoint-stripped label(s): ['kubernetes.io/os']", 'shipped selector missed host endpoints: matched []']
   result: FAIL_AS_EXPECTED

4) STAGE-1 CONTRACT (must remain non-enforcing)
   policyAuditMode: True
   hostFirewall: {'enabled': True}

5) AUDIT PLAN §2c TWO-WAY GATE
   {
  "has_two_way_gate_language": true,
  "requires_deny_rows_present": true,
  "requires_allow_any_survive": true,
  "documents_inert_failure_mode": true,
  "documents_control_plane_selector": true,
  "section_chars": 2380,
  "result": "PASS"
}

Per-node match matrix (fixed policy vs measured host-endpoint labels):
   talos-1: host_ep=True node=True broken_host_ep=False
   talos-2: host_ep=True node=True broken_host_ep=False
   talos-3: host_ep=True node=True broken_host_ep=False

OVERALL: PASS — selector fix selects host endpoints; baseline inert;
subtractive defaults intact; audit mode still on; stage-2 not present.

Evidence: Fixed policy semantic test JSON

{
  "brokenSelectorHostMatches": [],
  "brokenSelectorNodeMatches": [
    "talos-1",
    "talos-2",
    "talos-3"
  ],
  "denyPorts": [
    {
      "endPort": null,
      "port": "3300"
    },
    {
      "endPort": null,
      "port": "6789"
    },
    {
      "endPort": 7568,
      "port": "6800"
    },
    {
      "endPort": null,
      "port": "7000"
    },
    {
      "endPort": null,
      "port": "80"
    },
    {
      "endPort": null,
      "port": "8003"
    },
    {
      "endPort": null,
      "port": "8443"
    },
    {
      "endPort": null,
      "port": "9283"
    },
    {
      "endPort": null,
      "port": "9926"
    }
  ],
  "enableDefaultDeny": {
    "egress": false,
    "ingress": false
  },
  "fromEntities": [
    "world"
  ],
  "hostFirewall.enabled": true,
  "nodeSelector": {
    "matchExpressions": [
      {
        "key": "node-role.kubernetes.io/control-plane",
        "operator": "Exists"
      }
    ]
  },
  "nodes": {
    "talos-1": {
      "broken_host_endpoint_match": false,
      "broken_node_match": true,
      "host_endpoint_match": true,
      "node_match": true
    },
    "talos-2": {
      "broken_host_endpoint_match": false,
      "broken_node_match": true,
      "host_endpoint_match": true,
      "node_match": true
    },
    "talos-3": {
      "broken_host_endpoint_match": false,
      "broken_node_match": true,
      "host_endpoint_match": true,
      "node_match": true
    }
  },
  "policy": "ceph-lan-isolation",
  "policyAuditMode": true,
  "result": "PASS",
  "shippedSelectorHostMatches": [
    "talos-1",
    "talos-2",
    "talos-3"
  ]
}

Evidence: Baseline inert-selector reproduction

{
  "commit": "a0bb6df98312f23a7a98b24fe1750494a15f8291",
  "enableDefaultDeny": {
    "egress": false,
    "ingress": false
  },
  "hostEndpointMatches": [],
  "inert_on_host_endpoints": true,
  "nodeMatches": [
    "talos-1",
    "talos-2",
    "talos-3"
  ],
  "nodeSelector": {
    "matchLabels": {
      "kubernetes.io/os": "linux"
    }
  },
  "result": "FAIL_AS_EXPECTED",
  "selectorKeys": [
    "kubernetes.io/os"
  ],
  "usesStrippedLabels": [
    "kubernetes.io/os"
  ]
}

Evidence: Negative injected broken selector

{
  "injectedSelector": {
    "matchLabels": {
      "kubernetes.io/os": "linux"
    }
  },
  "strippedLabels": [
    "kubernetes.io/os"
  ],
  "hostEndpointMatches": [],
  "failures": [
    "nodeSelector uses host-endpoint-stripped label(s): ['kubernetes.io/os']",
    "shipped selector missed host endpoints: matched []"
  ],
  "result": "FAIL_AS_EXPECTED"
}

Evidence: Kustomize-built ceph-lan-isolation spec

{
  "apiVersion": "cilium.io/v2",
  "kind": "CiliumClusterwideNetworkPolicy",
  "metadata": {
    "name": "ceph-lan-isolation"
  },
  "spec": {
    "description": "Deny non-cluster (world) access to host-network Ceph daemon ports. Subtractive only - enableDefaultDeny is false in both directions so the host endpoint never enters default-deny and management access cannot break.",
    "enableDefaultDeny": {
      "egress": false,
      "ingress": false
    },
    "ingressDeny": [
      {
        "fromEntities": [
          "world"
        ],
        "toPorts": [
          {
            "ports": [
              {
                "port": "3300",
                "protocol": "TCP"
              },
              {
                "port": "6789",
                "protocol": "TCP"
              },
              {
                "port": "80",
                "protocol": "TCP"
              },
              {
                "port": "9283",
                "protocol": "TCP"
              },
              {
                "port": "9926",
                "protocol": "TCP"
              },
              {
                "port": "8003",
                "protocol": "TCP"
              },
              {
                "port": "7000",
                "protocol": "TCP"
              },
              {
                "port": "8443",
                "protocol": "TCP"
              },
              {
                "endPort": 7568,
                "port": "6800",
                "protocol": "TCP"
              }
            ]
          }
        ]
      }
    ],
    "nodeSelector": {
      "matchExpressions": [
        {
          "key": "node-role.kubernetes.io/control-plane",
          "operator": "Exists"
        }
      ]
    }
  }
}

Evidence: Stage-1 audit mode / hostFirewall contract

{
  "policyAuditMode": true,
  "hostFirewall": {
    "enabled": true
  },
  "note": "Stage 2 enforce flip is deliberately NOT in this PR"
}

Evidence: Audit plan §2c two-way gate check

{
  "has_two_way_gate_language": true,
  "requires_deny_rows_present": true,
  "requires_allow_any_survive": true,
  "documents_inert_failure_mode": true,
  "documents_control_plane_selector": true,
  "section_chars": 2380,
  "result": "PASS"
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

• python3 scripts/ci/hostpolicy-ceph-selector-test.py against kustomize-built ceph-lan-isolation (PASS: host-endpoint match on talos-1/2/3, enableDefaultDeny false/false, policyAuditMode true)
• Baseline reproduction via git show a0bb6df9:.../hostpolicy-ceph.yaml selector matched against measured host-endpoint label sets (FAIL_AS_EXPECTED: 0 host-endpoint matches, stripped kubernetes.io/os)
• Negative injection of matchLabels: kubernetes.io/os=linux into the fixed policy (FAIL_AS_EXPECTED)
• kubectl kustomize kubernetes/apps/kube-system/cilium/app structural check of shipped CCNP spec
• HelmRelease stage-1 contract check (policyAuditMode: true, hostFirewall.enabled: true)
• Audit plan §2c two-way gate structure check (Deny rows required + Allow Ingress ANY must survive + inert failure documented)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[mortyops]

--- kubernetes/apps/kube-system/cilium/app Kustomization: kube-system/cilium CiliumClusterwideNetworkPolicy: kube-system/ceph-lan-isolation

+++ kubernetes/apps/kube-system/cilium/app Kustomization: kube-system/cilium CiliumClusterwideNetworkPolicy: kube-system/ceph-lan-isolation

@@ -37,9 +37,10 @@

       - port: '8443'
         protocol: TCP
       - endPort: 7568
         port: '6800'
         protocol: TCP
   nodeSelector:
-    matchLabels:
-      kubernetes.io/os: linux
+    matchExpressions:
+    - key: node-role.kubernetes.io/control-plane
+      operator: Exists