Skip to content

Fix destroy: delete profiles before networks to avoid "in use" error#1

Closed
jpaussa wants to merge 0 commit intopsaab:masterfrom
jpaussa:master
Closed

Fix destroy: delete profiles before networks to avoid "in use" error#1
jpaussa wants to merge 0 commit intopsaab:masterfrom
jpaussa:master

Conversation

@jpaussa
Copy link
Copy Markdown

@jpaussa jpaussa commented Feb 11, 2026

Profiles reference the networks, so Incus refuses to delete a network while a profile still holds a device entry for it.
before fix:

jdp@coi:~/bpfrx$ ./test/incus/setup.sh destroy
==> Instance bpfrx-fw does not exist
Also remove networks and profiles? [y/N] y
==> Deleting network bpfrx-trust
Error: The network is currently in use

after fix:

jdp@coi:~/bpfrx$ ./test/incus/setup.sh destroy
==> Instance bpfrx-fw does not exist
Also remove networks and profiles? [y/N] y
==> Deleting profile bpfrx-vm
Profile bpfrx-vm deleted
==> Deleting profile bpfrx-container
Profile bpfrx-container deleted
==> Deleting network bpfrx-trust
Network bpfrx-trust deleted
==> Deleting network bpfrx-untrust
Network bpfrx-untrust deleted
==> Deleting network bpfrx-dmz
Network bpfrx-dmz deleted
==> Deleting network bpfrx-tunnel
Network bpfrx-tunnel deleted
==> Destroy complete.

@jpaussa
Copy link
Copy Markdown
Author

jpaussa commented Feb 11, 2026

Urgh, it also added this to the commit, claude said there was no reason to pin to version 21 so I just got it to remove the pin.
Have told claude to use branches:

jdp@coi:~/bpfrx$ make generate
go generate ./pkg/dataplane/...
go: downloading go1.24.9 (linux/amd64)
go: downloading github.com/vishvananda/netlink v1.3.1
go: downloading github.com/cilium/ebpf v0.20.0
go: downloading golang.org/x/sys v0.40.0
go: downloading github.com/vishvananda/netns v0.0.5
Error: exec: "llvm-strip-21": executable file not found in $PATH
exit status 1
pkg/dataplane/loader.go:14: running "go": exit status 1
make: *** [Makefile:15: generate] Error 1
jdp@coi:~/bpfrx$ make generate
go generate ./pkg/dataplane/...
In file included from /home/jdp/bpfrx/bpf/xdp/xdp_main.c:12:
/home/jdp/bpfrx/bpf/xdp/../headers/bpfrx_helpers.h:606:2: warning: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering [-Wpass-failed=transform-warning]
  606 |         for (__u32 i = 0; i < MAX_FILTER_RULES_PER_FILTER; i++) {
      |         ^
1 warning generated.
/home/jdp/bpfrx/bpf/xdp/xdp_zone.c:96:4: warning: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering [-Wpass-failed=transform-warning]
   96 |                         for (__u32 ni = 0; ni < MAX_NAT64_PREFIXES; ni++) {
      |                         ^
1 warning generated.
/home/jdp/bpfrx/bpf/xdp/xdp_policy.c:847:7: warning: loop not unrolled: the optimizer was unable to perform the requested transformation; the transformation might be disabled or specified as part of an unsupported transformation ordering [-Wpass-failed=transform-warning]
  847 |                                                 for (__u32 ni = 0; ni < MAX_NAT64_PREFIXES; ni++) {
      |                                                 ^
1 warning generated.

@jpaussa jpaussa closed this Feb 14, 2026
psaab added a commit that referenced this pull request Apr 2, 2026
Both /failover-test and /ha-failover-test skills now require
build+deploy as the first step before any tests. Stale binaries
were the #1 source of false test failures throughout the HA
debugging sessions. Includes checksum verification and force-push
instructions for when the deploy script fails silently.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 2, 2026
- Fix extra whitespace before comments in election_test.go (lines 788, 846)
- Improve weight preservation test comment to clarify what's being
  verified (weight not zeroed, stays at monitor-derived value)

Copilot suggestion #1 (non-default weight) not applied — SetMonitorWeight
requires interface-down state, making the test unnecessarily complex for
verifying weight != 0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 19, 2026
Four changes addressing the review findings:

HIGH — bounded ingest → drain_shaped_tx loop:
  Single re-ingest (previous commit) only closes the race window
  up to its call site. Peers can still push to the MPSC inbox
  during the prepared/backup drain that follows. Replace the
  single re-ingest with a bounded loop (budget 4) of
  ingest → drain_shaped_tx that widens the window until either
  all three pending buffers quiesce or the budget is exhausted.
  Budget sized to absorb a 3× burst without risking starvation
  of the outer worker loop.

HIGH — belt-and-suspenders drop filter at backup paths:
  Add drop_cos_bound_prepared_leftovers + drop_cos_bound_local_leftovers.
  Before the post-CoS backup transmit, scan pending_tx_prepared
  and pending_tx_local; any item whose cos_queue_id.is_some() is
  dropped (UMEM slot recycled via recycle_prepared_immediately
  for prepared items; Vec<u8> freed for local items) instead of
  transmitted unshaped. Makes the CoS cap a hard guarantee
  regardless of how many race windows remain open upstream.

MEDIUM — suppress owner_pps/peer_pps attribution on re-ingest:
  Items left in pending_tx_local after the first pass get
  re-classified as owner-local by the second ingest's
  owner_local_count / peer_count split; that corrupts the
  per-binding pps provenance telemetry (inflates owner_pps,
  deflates peer_pps). Split ingest into
  ingest_cos_pending_tx_with_provenance(count_pps: bool); the
  first pass counts, re-ingests do not.

LOW #1 — cacheline isolation:
  Move drain_sent_bytes_shaped_unconditional and
  post_drain_backup_bytes from BindingLiveState (which shares a
  line with multi-writer redirect_inbox_overflow_drops) onto the
  already-isolated #[repr(align(64))] OwnerProfileOwnerWrites
  block. Owner-only writes no longer ping-pong with peer overflow
  writers during overload. Also move the two new drop-filter
  counters there.

LOW #2 — document read skew:
  drain_sent_bytes_shaped_unconditional, post_drain_backup_bytes,
  and tx_bytes are written at separate sites and read at separate
  scrape instants. Doc-comment clarifies these are
  delta-over-window diagnostics, not point-in-time equalities.

Also adds BindingStatus fields for post_drain_backup_cos_drops
and post_drain_backup_cos_drop_bytes so operators can observe
the drop filter firing via the JSON snapshot.

All 703 Rust tests + Go dataplane/userspace tests pass.

Refs #760, #772, #773

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 20, 2026
Address both adversarial reviews on the D3 RSS indirection PR —
Codex HIGH #1/#2/MEDIUM and Go MEDIUM #1/#2/#3 + LOWs.

Kill switch (Codex MEDIUM, rollback):
  New `set system dataplane rss-indirection enable|disable` knob.
  Default enabled; `disable` short-circuits applyRSSIndirection so the
  kernel's default hashing stays intact. Compiled into a bool on
  UserspaceConfig, gated at both the startup and reconcile call sites.

Reconcile on config change (Codex HIGH #2):
  applyConfig now calls reapplyRSSIndirection() after cluster transport
  reconcile. Worker-count changes (`set system dataplane workers N`) and
  kill-switch toggles take effect on commit, not only first boot.
  Idempotent: matching tables skip the write.

Explicit per-interface mlx5 guard at the call site (Codex HIGH #1):
  applyRSSIndirection now driver-filters before invoking ethtool. Any
  non-mlx5 netdev logs at slog.Debug and is skipped without shelling
  out. The existing per-interface check inside applyRSSIndirectionOne
  is retained as defense in depth.

Injectable sysfs scan (Go MEDIUM #3):
  rssExecutor grows a listInterfaces() method; tests no longer leak to
  real /sys/class/net. This also unblocks Go MEDIUM #2.

Real non-mlx5 skip test (Go MEDIUM #2):
  TestApplyRSSIndirection_NonMlx_Skips now calls applyRSSIndirection
  with a virtio_net + iavf fixture and asserts zero ethtool calls.
  Adds TestApplyRSSIndirection_MixedDrivers_OnlyMlxTouched to cover
  the mixed-driver host case from HIGH #1.

Stable errors.Is for ethtool-not-found (Go MEDIUM #1):
  isExecNotFound swapped to errors.Is(err, exec.ErrNotFound). Drops
  the brittle substring match and its stale comment. Pinned by a new
  TestIsExecNotFound_DetectsSentinel.

Two-call idempotency test (Go LOW #3):
  TestApplyRSSIndirection_TwiceIsIdempotent asserts the second call
  after a matching-table first call adds exactly one probe ethtool
  invocation and zero writes.

Fixture cleanup (Go LOW #1):
  Dead `ethtoolXFailNotFound` field removed from fakeRSSExecutor.

Boot-time log demotion (Go LOW #2):
  `workers <= 0` (non-userspace deploys) downgraded to slog.Debug.
  `workers == 1` (explicit operator choice) stays at slog.Info.

Config schema + compiler:
  `rss-indirection` appears under `system dataplane` with args=1 and
  compiles the string `disable` to UserspaceConfig.RSSIndirectionDisabled.
  TestUserspaceDataplaneRSSIndirectionDisable pins the default-off,
  disable-flips-on, enable-is-no-op behaviour.

No functional regression when D3 is enabled (default): the weight
vector, idempotency logic, and ordering invariants are unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 20, 2026
…ning

Address PR #796 review findings from Codex (HIGH + MEDIUM + LOW)
and Rust reviewer (3x MEDIUM).

== Codex HIGH — queue_vtime round-trip neutrality ==

`cos_queue_pop_front()` advanced `queue_vtime` by the drained
packet's bytes BEFORE TX was attempted. On TX-ring-full retry the
popped item bounced back via `cos_queue_push_front()`, but
`queue_vtime` stayed advanced — so a newly-active flow arriving
after the retry inherited an inflated vtime anchor and was pushed
behind established traffic even though zero bytes had actually
drained. This broke the byte-rate fairness contract under
TX-ring-full conditions.

Fix: per-item symmetric rewind. `cos_queue_push_front` subtracts
the pushed item's bytes from `queue_vtime` (mirroring pop_front's
add). Per-item (not a single snapshot) is the right granularity
because TX-ring-full settle paths pop N items into a scratch Vec,
submit M, and push_front the remaining N-M in LIFO order — a
single snapshot would only work for full rollback; symmetric
rewind handles partial commit too.

== Rust MEDIUM #1 — drained-bucket round-trip neutrality ==

Closely related to the Codex HIGH: when the popped item was the
sole packet in its bucket, the bucket drained, head/tail reset
to 0, and a push_front re-anchored to `max(0, vtime) + bytes` —
one packet past the pre-pop head. Fix: new
`CoSQueuePopSnapshot` captures pre-pop bucket + head-finish +
tail-finish on every pop; push_front on a matching bucket
restores head/tail exactly. Snapshot invalidated on any
`cos_queue_push_back` (new enqueue changes bucket state
underneath). Snapshot is per-queue, size-bounded (16 bytes +
option discriminant), no hot-path allocation.

== Codex MEDIUM — brief-idle re-entry pin ==

New pin `mqfq_brief_idle_reentry_exercises_both_max_arms` —
bucket drains, another bucket drains advancing vtime modestly,
first bucket re-enqueues. Exercises BOTH arms of the
`max(tail_finish, queue_vtime) + bytes` re-anchor formula:
first re-enqueue (tail=0 < vtime, picks vtime) and second
enqueue (tail > vtime, picks tail). Prior pins only covered
large-idle cases.

== Rust MEDIUM #2 — real-field headroom pin ==

Rewrote `mqfq_finish_time_u64_has_decades_of_headroom` from a
calculator (recomputes wrap math in Rust, never reads the real
field) into an actual pin: drives `queue_vtime` to
`u64::MAX - 10_000`, enqueues 9000-byte packets via
`cos_queue_push_back`, asserts the bucket's head/tail did not
wrap AND that the second enqueue saturates at `u64::MAX`. A
regression to u32 or non-saturating add would now fail the test.
Kept the original 40-year calculator as a supplementary sanity
check.

== Rust MEDIUM #3 — golden-vector table ==

New pin `mqfq_golden_vector_pop_order_vs_drr` — a small table of
(packet_sizes, expected_mqfq_order, reference_drr_order) rows,
asserting MQFQ pop order per row AND that the table includes at
least one row where MQFQ diverges from DRR. Documents the
tie-break rule in `cos_queue_min_finish_bucket` and locks the
MQFQ-vs-DRR divergence into the test surface for regression
protection.

== Codex round-2 LOW #2 / Rust LOW — idle-return anchor pin ==

New pin `mqfq_idle_flow_reanchors_at_frontier_not_zero` — drains
flow A for 4500 bytes, idle-returning flow B's first packet must
anchor at `queue_vtime + bytes = 5700`, not at `0 + bytes = 1200`.
Complements `mqfq_queue_vtime_advances_by_drained_bytes` and
`mqfq_bucket_drain_resets_finish_time` by asserting the
CONSEQUENCE of those invariants.

== Rust LOW — back-reference in field doc ==

Added `Read by cos_queue_min_finish_bucket` back-reference on
`flow_bucket_head_finish_bytes` so a reader starting from the
field learns the consuming function.

== Existing pin updated ==

`mqfq_push_front_is_finish_time_neutral_on_active_bucket` now
also captures `pre_pop_vtime` and asserts it equals post-restore
`queue_vtime`. Prior version missed the Codex HIGH (test was
green while the bug shipped).

== Tests ==

All 11 MQFQ-tagged tests pass. All 9 flow_fair-tagged tests
pass. Full test suite: 731 passed, 0 failed, 1 ignored.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 20, 2026
Append round-2 verification section to
`docs/785-phase3-pr-review-rust.md` covering the fixes in
`4416eb27`.

All 3 round-1 MEDIUMs CLOSED:
- MED #1 (drained-bucket round-trip) — `CoSQueuePopSnapshot` +
  `last_pop_snapshot: Option<...>` on `CoSQueueRuntime`, taken in
  `cos_queue_push_front`, invalidated in `cos_queue_push_back`.
  Allocation-free, leak-free, composes with Codex HIGH's
  symmetric vtime rewind.
- MED #2 (headroom calculator) — rewritten pin now drives
  `queue_vtime` to `u64::MAX - 10_000` and calls real
  `cos_queue_push_back` twice, asserting first enqueue anchors
  exactly at `near_wrap + 9000` and second saturates at
  `u64::MAX`.
- MED #3 (mixed-size golden vector) — 3-row table with hard-coded
  MQFQ pop order; DRR reference column documented-not-executed;
  `any_divergent` meta-assert guards against accidental
  collapse-to-DRR.

New-angle audits clean:
- No new production unsafe/unwrap/expect/eprintln/panic (two new
  `.expect()` are test-only).
- New pins assert full state (head, tail, bytes, vtime, active
  count, items.len()).
- Snapshot cost: 24-byte Option on queue struct, written on every
  pop, cleared on every push_back — no allocation, in cache line.

11 mqfq_* tests pass (`cargo test --release ... mqfq`).

Merge-ready from Rust-quality angle.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 20, 2026
Incorporates two adversarial reviews into the plan:

Codex review (docs/line-rate-investigation-plan-review.md)
- HIGH #1 (counter ordering for H-FWD-1): Steps 1 and 2 swapped —
  userspace-dp counters (dbg_tx_ring_full, pending_tx_local_
  overflow_drops, outstanding_tx, flow_steer_snapshot) now captured
  BEFORE NIC counters. See "Step 1: capture userspace-dp counters".
- HIGH #2 (fairness rollback brittleness): fixed-5pp gate replaced
  with matched 5-run pre/post statistical protocol, rollback if
  mean-CoV delta > 2 × pre-stddev, retr > +100, or p99 regression.
  See "Statistical protocol".
- MEDIUM (TCP CC uncontrolled): new Step 0.3 pins tcp_congestion_
  control on client + server, verifies via ss -ti during runs.
- MEDIUM (XSK bind mode / fallback): new Step 0.6 captures
  xsk_bind_mode, zero-copy, XDP_FALLBACK per binding.
- MEDIUM (H-REV-3 observables): hypothesis restated against
  TX-ring-full / overflow / restore-helper observables.
- MEDIUM/LOW (success criteria, rollback mechanics): single success
  definition + explicit deferred-findings section.

Systems review (docs/line-rate-investigation-plan-review-systems.md)
- HIGH S-1 (IRQ ↔ worker CPU alignment): new Step 0.1 audits
  /proc/irq/*/smp_affinity_list for mlx5 queues; misalignment is a
  zero-code fix applied before any dataplane change.
- HIGH S-2 (no latency metric): p50/p99 via concurrent ping -i 0.01
  + ss -ti added to Step 3 and to the Validation section; latency
  non-regression gate added to rollback protocol.
- HIGH S-3 (NAPI/coalescence/busy-poll unaudited): new Step 0.2
  audits ethtool -c, netdev_budget, gro_flush_timeout, SO_BUSY_POLL.
  H-FWD-5 added. mpstat now breaks out %usr/%sys/%soft/%irq.
- MEDIUM S-4 (ring quadruple): Step 0.4 + renamed Step 5 audit all
  four rings (RX/TX/fill/completion), distinguishing TX-produce
  from completion-reap starvation.
- MEDIUM S-5 (L1d pressure of 24 KB MQFQ state): perf stat -e
  L1-dcache-load-misses added to Step 3 during-test sampling.
- MEDIUM S-6 (MQFQ small-ACK pathology): new H-REV-6 hypothesis +
  new Step 7 jumbo-MSS diagnostic; ACK-bucket histogram capture.
- LOW S-7 (C-states / frequency): turbostat + cpupower folded as
  Step 0.5 + Step 3 diagnostic observable.

Hard stops expanded: any mqfq_* OR flow_steer_* regression triggers
rollback. Step 0 (zero-code audit) lands inside Phase B, runs
before Step 1. Phases A-E structure preserved.

Plan grew from 296 to 522 lines (+292 / -66).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 21, 2026
Closes the three HIGH-severity findings from
step1-plan-review.md without weakening any other invariant.

HIGH #1 (threshold X wrong statistic). §4.2 rewritten on the
actual multinomial flow-count distribution. Previous draft used
per-worker binomial stddev applied to max−min (wrong statistic)
giving 79 % false-positive rate on fair RSS. New predicate:
`max(n_w) ≥ 8 OR min(n_w) ≤ 0` on integer worker flow counts.
Monte Carlo (10⁶ trials) gives combined FP = 0.133. Threshold Y
(rate spread) derived from MTU-quantum noise floor and the
8matrix-findings.md empirical 1.25× ceiling; set at 1.40×. New
park-rate threshold B_park = 100 parks/s derived from the 1 ms
token-refill tick. Threshold Z derived from observed no-cos
baseline (mean 2.1 full-ring events / 60 s) at 50 events / s ≈
330σ above noise. Each threshold now names its own FP target.

HIGH #3 (dbg_cos_queue_overflow wrong signal). Codex verified
the counter increments on admission rejection (flow_share /
buffer_exceeded at tx.rs:5326-5412), NOT MQFQ token starvation.
Verdict B predicate moved to the park counters already exposed
via status.cos_interfaces[].queues[] — root_token_starvation_parks
(tx.rs:1500) and queue_token_starvation_parks (tx.rs:1516). B
now requires three clauses: park-rate ≥ 100/s, rate-spread ≥
1.40×, AND (not A, not C). dbg_cos_queue_overflow demoted to
corroborating signal. Appendix A updated to list the park
counters as B-primary. No new instrumentation PR required —
all signals present on master.

HIGH #6 (script doesn't exist). Script committed in the
companion commit at test/incus/step1-capture.sh with bash -n
syntax check and arg-validation smoke test. §9 updated to
match reality.

Also addresses:
- MEDIUM #5 (CoS-liveness check): §6 now verifies runtime
  cos_interfaces + filter_term_counters delta after each
  transition using the tight 1 G shaper on port 5201 as the
  discriminator.
- MEDIUM #7 (perf stat wrong thread selector): §2.2(c) now
  uses `ps -eLo tid,comm` against actual thread name
  `xpf-userspace-worker-*` (coordinator.rs:693-695) instead of
  `pgrep -f` on cmdline. Invariant I8 checks 4 TIDs present
  and daemon ActiveEnterTimestamp unchanged mid-cell.
- MEDIUM #8 (primary drift): I9 promotes fw0-is-primary to
  start-of-run invariant, not per-cell only.
- MEDIUM #10 (rescope rule): §7 now defines explicit 60/90-min
  rescope ladder.

Wall-clock budget unchanged: 60-90 min target, 120 min ceiling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 21, 2026
Round-2 review (HIGH #1, HIGH on Y): the prior plan revision
documented X=max>=8/min<=0 with FP=13.3% as "HFT-acceptable
floor" without defending the choice, and Y=1.40 with no data
backing.

Threshold X: tighten to max(n_w) >= 9 OR min(n_w) <= 0 per the
round-2 reviewer's explicit recommendation. Monte Carlo (committed
in test/incus/step1-rss-multinomial.py) confirms FP=0.064 (~6.4%).
Defended choice: tighter (max>=10) starts hiding real imbalance
(one worker at 62%+); 6.4% is the right knee. Adds the inline
sample output so the FP number is reproducible from the doc alone.

Threshold Y: set to 2.72 = mean + 2*stddev of the within-cell
max/trimmed_min ratio measured across the four forward shaped
cells in evidence-8matrix/ (computed by
test/incus/step1-rate-spread-analysis.py). Rewrites the
derivation paragraph to cite the actual JSON files and explain
why high in-worker variance (Cubic oscillation under tight rate
caps) is real physics, not a bug to threshold against.

Threshold-summary table updated: Verdict A FP <= 0.064 (was 0.13);
Verdict B uses 2.72 (was 1.40).

Also drives schema correctness through the plan body:
- request envelope is `{"type":"status"}` (not `request_type`)
- response keys live under `.status.*` (not top-level)
- I6 invariant calls out `_error: control_socket_timeout`
  placeholders as failures so timeouts don't silently shrink N
- §6 reconciliation jq calls now read `.status.cos_interfaces`
- new §6 subsection enumerates the three failure-mode error
  paths the capture script implements (CoS apply fail, failover
  detection, control-socket timeout) so the plan documents what
  the script actually does

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 21, 2026
HIGH #1 — submit-side amortization collapse on small batches. §3.1
revised: per-commit stamp (one monotonic_nanos() per writer.commit())
applied to the `inserted` descriptors only; retry-tail not stamped.
Why `now_ns` reuse is rejected (up to ~1 ms staleness across the
worker loop at worker.rs:619 + afxdp.rs:176-178).

HIGH #2 — relaxed-atomic cross-CPU visibility on ARM. §3.6.a decision:
keep Relaxed on both writer and reader, document bounded-skew
semantics (mirrors existing umem.rs:1322-1329 pattern), downgrade
invariants 6/7 and §8 hard-stop #4 to "|sum - count| / count ≤ 0.01"
rather than exact equality. Upgrade to Release/Acquire rejected on
~2 % ARM cost grounds.

HIGH #3 — sidecar false sharing across workers. §3.3 rewritten after
reading the code: each binding has its own UMEM via
`WorkerUmemPool::new` at worker.rs:445 with `shared_umem=false`,
wrapped in `Rc<WorkerUmemInner>` at umem.rs:16-18 (single-owner
thread). Cross-worker false-sharing is structurally impossible.
UMEM-headroom in-frame approach (c) rejected on blast-radius grounds.

HIGH #4 (Codex numbering) — overhead arithmetic. §3.4 rewritten with
three operating-point numbers (`inserted = 256 / 64 / 1`); honest
worst-case 45 ns/pkt = 9.4 % of per-queue 481 ns budget. Corrects
earlier 0.13 % figure that hid the partial-batch regime and divided
by workers instead of queue.

Also closes MED #5 (sentinel vs clock-0 collision), MED #6 (sidecar
size 192 KiB not 64 KiB; 3×ring_entries per bind.rs:37-44), MED #7
(wire-size growth ~8-10 KiB per status poll), MED #8 (Bonferroni
family corrected to 3 composite tests per cell × 12 cells = 36),
MED #9 (symbolic two-thread race test replaced by partial-batch,
retry-unwind, and bounded-skew tests). LOW #10 (no ambient `now_ns`
reuse), LOW #13 (named const asserts + boundary test).

Plan-ready: pending Codex round-2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab pushed a commit that referenced this pull request Apr 21, 2026
Codex round-2 HIGH #1 demanded real evidence that libc::clock_gettime
resolves through VDSO (user-space) on the deploy target, not a syscall —
otherwise seccomp-strict profiles would SIGKILL the daemon and the plan's
"no syscall per packet" invariant is indefensible.

Commit three probe artifacts under docs/pr/812-tx-latency-histogram/evidence:

- vdso_probe.c: strace -e clock_gettime on the build host captures
  zero clock_gettime(...) syscall lines despite 10 000 userspace calls.
  strace_host.txt holds the raw output.
- vdso_probe2.c: run on bpfrx-fw0 (target deploy VM), captures a non-zero
  AT_SYSINFO_EHDR and a [vdso] mapping in /proc/self/maps. vm_bpfrx_fw0.txt
  holds the raw output.
- vdso_evidence.md: the write-up that the plan cites from §3.4a.

Build host: Linux 6.18.5, Debian glibc 2.42-11+b1.
Target VM (bpfrx-fw0): Linux 6.18.15, Debian glibc 2.42-13.

Both environments meet the glibc + kernel pair that guarantees VDSO
resolution for clock_gettime(CLOCK_MONOTONIC) on x86_64.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab pushed a commit that referenced this pull request Apr 21, 2026
…wned-static-send, 5% stop, bucket-0)

Round-2 review left all three HIGH findings PARTIAL or OPEN, plus two MED
findings around silent acceptance-criteria regression and bucket-0
deferral. Each is now closed with an in-place plan change:

HIGH #1 (VDSO fast path, PARTIAL -> CLOSED). Add §3.4a grounding the
"NOT a syscall" claim on strace (host) + AT_SYSINFO_EHDR
(bpfrx-fw0 VM) evidence committed in the prior commit. Chose option
(a) + (c) from the findings — verify on the target AND document the
deployment dependency with a named remediation path (graceful degrade
via the existing sentinel rule, not panic). Invariant 1 in §4 now
cites the evidence and names the fallback.

HIGH #2 (1% skew tolerance untested, Bonferroni tighter than
tolerance, OPEN -> CLOSED). Derive K_skew = ceil(λ × W_read) = 1
completion per snapshot at 1 Mpps × 1 µs in §3.6 R2. Replace §11.3
Bonferroni with a cell-level block-permutation test (Fisher-Pitman
style) whose null distribution is constructed from the data itself —
within-block reshuffles absorb K_skew noise by construction, so the
0.0014 bound is neither needed nor applied. Add §6.1 test #7 that
computes K_skew from the harness's own measured write rate and
read-window, asserting |sum - count| <= K_skew + 2 (paranoia margin
for TSO/ARM). The 1% integration hard-stop in §8 is now defended as
the scheduler-preemption-robust bound, not a guess.

HIGH #3 (snapshot-thread crossing, PARTIAL -> CLOSED). Add §3.5a
pinning BindingCountersSnapshot to owned values via a compile-time
const assert that requires 'static + Send. The struct already derives
Clone + Serialize + Deserialize + Default on u32/u64/i32 scalars; the
#812 extension adds Vec<u64> + two u64s, all owned. Rust type-system
trick: 'static on a struct with no lifetime parameter mechanically
rejects any future &'a U field addition. Defense in depth: §6.1 test
#4 (JSON round-trip) also fails for any accidental reference field
via serde's DeserializeOwned default.

MED (overhead hard-stop silently widened from 1% to 5%/10%).
Defended in §8.1 (option (b) from findings): the 1% bound was proposed
before §3.4 re-derived actual costs. Typical case 3.1%, worst case
9.4% on the inserted==1 partial-batch regime documented at
tx.rs:5953-5961 / tx.rs:6164-6174. A 1% gate would hard-stop on the
instrumentation's own presence under partial-batch traffic; that is
the wrong failure mode. 5% steady-state + 10% small-batch soft-gate
is the cheapest correctness-preserving budget; alternatives (rdtsc,
sampled stamping, bucket-midpoint sum) all lose measurement fidelity.

MED (bucket-0 coarseness deferred). Resolved as out-of-scope in §3.2
and §12 item 8. The §11 classifier's pre-registered statistics
(D1/D2/D3) all read from buckets 3+ — no verdict depends on sub-µs
resolution. Bucket 0 is intentionally coarse because the
MQFQ-vs-shaper separation (verdict B) lives in tens-of-microseconds
(buckets 4-7), not sub-µs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 21, 2026
Add 9 Rust unit tests covering the plan §6.1 / §5 test surface:

Rust-side (umem.rs test module):

  1. tx_latency_hist_bucket_boundary_roundtrip — plan §6.1 #1 /
     §5.1. Drive record_tx_completions_with_stamp with
     deterministic T0 and T0+K for K ∈ {500, 1500, 10_000, 100_000,
     10_000_000}; assert exactly one count in the bucket predicted
     by bucket_index_for_ns and zero in every other bucket. Pairs
     with the existing bucket boundary test so a bucket-layout
     drift fails BOTH pins.
  2. tx_latency_hist_partial_batch_stamping_only_touches_accepted_prefix
     — plan §6.1 #2 / §3.1 R1. `inserted ∈ {1, 2, 32, 64, 256}`;
     assert only the first `inserted` sidecar slots hold the stamp,
     tail stays at TX_SIDECAR_UNSTAMPED. The Codex HIGH #1 small-
     batch regime contract.
  3. tx_latency_hist_retry_unwind_leaves_no_stamps — plan §6.1 #3.
     The `inserted == 0` retry-unwind path hands an empty iterator
     to stamp_submits; every sidecar slot stays at the sentinel.
  4. tx_latency_hist_sentinel_skip_for_unstamped_completion —
     plan §6.1 #5 / §5.4 / Codex MED #5. A completion against an
     unstamped slot (never stamped, or canonicalized 0 from a
     VDSO-failure stamp) drops the sample — no bucket increment,
     no count/sum increment.
  5. tx_latency_hist_single_thread_sum_equals_count — plan §6.1
     #6 / §5.2. Drive N = 10_000 synthetic stamps + completions
     in one thread; assert `sum(hist) == count` exactly AND
     `sum_ns == sum(deltas)` exactly (single-thread invariant 6
     per plan §4).
  6. tx_latency_hist_cross_thread_snapshot_skew_within_bound —
     plan §6.1 #7 / §3.6 R2. Writer thread fires 1M+ fetch_adds;
     reader thread snapshots ≥ 5000 times. For each snapshot
     compute K_skew_i = ceil(λ_obs × W_read_i) + 2 using
     Instant::now() bracketing, assert |sum - count| ≤ K_skew_i.
     The derivation-driven cross-thread bound, not a free
     parameter.
  7. tx_submit_ns_sidecar_single_writer_ownership_is_rc_not_arc —
     plan §6.1 #6. Compile-time pin via fn-pointer probes that
     `WorkerUmem::shares_allocation_with` (calls Rc::ptr_eq) and
     `WorkerUmem::allocation_ptr` (calls Rc::as_ptr) retain the
     Rc shape. A future Rc→Arc migration breaks both bodies —
     silent drift becomes a loud compile failure.
  8. (extension of binding_live_snapshot_propagates_709_owner_profile_counters)
     — pin that the snapshot() path copies all three new
     atomics (hist + count + sum_ns) into BindingLiveSnapshot.

main.rs tests (wire-contract side):

  9. tx_latency_hist_serialization_roundtrip — plan §6.1 #4.
     JSON encode/decode round-trip on a non-trivial histogram;
     assert field-equality including Vec<u64> contents.
 10. tx_latency_hist_backward_compat_old_payload_deserializes —
     plan §6.1 #4 (second half) / §7 PR #804 wire-contract
     break row. Pre-#812 JSON payload (fields absent)
     deserializes with empty Vec and zero u64 via
     `#[serde(default)]`. THE backward-compat contract for
     step1-capture.
 11. tx_latency_hist_binding_counters_snapshot_is_static_send —
     plan §6.1 #8. Runtime corollary of the named compile-time
     const-assert added in commit 5 — exercising the `'static
     + Send` bound at test time too. Defence-in-depth if the
     const-assert were ever silently removed.

Also extends existing
binding_counters_snapshot_serializes_with_expected_wire_keys
and binding_counters_snapshot_projects_ring_pressure_fields to
include the three new wire keys so a rename or misattribution
is caught.

Refactor: factored the per-offset reap fold out of
reap_tx_completions into a new shared helper
`record_tx_completions_with_stamp` so the unit pins exercise
the PRODUCTION algorithm, not a test-only fake. The live
reap_tx_completions now calls the helper; same semantics,
same atomics, same order.

Go-side (pkg/dataplane/userspace/protocol.go):

  Add tx_submit_latency_hist / _count / _sum_ns fields on
  BindingStatus and BindingCountersSnapshot. omitempty keeps
  forward-compat — a pre-#812 helper that lacks these fields
  decodes into empty slice / zero u64 without Unmarshal
  erroring. Needed by the Go decoder that the daemon's
  status-poll path uses for step1-capture.

Plan: docs/pr/812-tx-latency-histogram/plan.md §6.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 21, 2026
…all 6 sites

Codex round-1 HIGH #1 (docs/pr/812-tx-latency-histogram/codex-code-review.md):
the submit stamp was sampled BEFORE `writer.commit()` at every submit
site. If the scheduler preempts between `insert()` and the ring submit,
that preemption window inflates the observed submit→completion
latency — exactly the opposite of what the histogram should report.

Fix: reorder so every site calls `writer.commit(); drop(writer);` before
sampling `monotonic_nanos()` and handing it to `stamp_submits`. Only the
accepted prefix (`.take(inserted as usize)`) is stamped; retry-tail
descriptors return to `free_tx_frames` untouched (unchanged contract).

Applied per-site at the six call sites enumerated in plan §3.1:
  - service_exact_local_queue_direct (~tx.rs:1977)
  - service_exact_local_queue_direct_flow_fair (~tx.rs:2118)
  - service_exact_prepared_queue_direct (~tx.rs:2270)
  - service_exact_prepared_queue_direct_flow_fair (~tx.rs:2408)
  - post-CoS backup transmit_batch (~tx.rs:6122)
  - transmit_prepared_queue continuation (~tx.rs:6351)

`stamp_submits` doc-comment updated to record the invariant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 25, 2026
1. rss_indirection.go:242 — log "rss weight reshaping skipped"
   instead of "rss indirection skipped" so it's clear that the
   weight-vector write is what's skipped, not the whole RSS path.
   When workers>=queues>1 with a stale table, the next line(s)
   show the actual restore action.

2. rss_indirection_test.go:47 — comment for argvErr said
   "checked AFTER" but code checks BEFORE. Aligned to actual
   precedence.

3. rss_indirection_test.go:71 — Go map iteration is randomized,
   so two overlapping prefix keys would produce flaky results.
   Implemented longest-prefix-wins tie-breaking for determinism.

4. rss_indirection_test.go:846 — test #17 appeared before #16.
   Reordered to be sequential.

5. queues>1 vs queues>0 mismatch was already fixed in 744f8b2
   (Copilot saw the change but flagged a stale comment-vs-code
   diff — addressed via the comment update accompanying #1).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 25, 2026
* #805 D3 RSS refresh on workers↔queues transition — plan

Bug: when `system dataplane workers` is bumped from <queue_count
to >=queue_count (e.g. 4→6 on 6-queue mlx5),
applyRSSIndirectionOne early-returns because computeWeightVector
gives nil for workers>=queues. The previously-written
[1,1,1,1,0,0] table stays live; queues 4 and 5 starve.

Fix: when nil-because-workers>=queues, inspect the live table;
if not default round-robin, run `ethtool -X iface default`.

Plan covers: root cause, detection helper, implementation
(single-file rss_indirection.go change), tests (5 cases — 2
behavioral + 2 parser + 1 regression), acceptance (test
cluster live deploy + Codex review + Copilot + test-failover),
risks (rebalance race already mitigated by rssWriteMu, manual
table writes intentionally clobbered by daemon ownership).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 plan — address Codex round 1 (2 HIGH, 6 MED, 1 LOW)

R1 verdict was PLAN-READY NO. Critical correctness fixes:

HIGH:
- 1: claimed `rssWriteMu`, `applyRSSIndirectionLocked`, epoch
  bumps that came from #840 (REVERTED). Master state has no
  locking, void return. §3 added documenting actual master
  contract. Fix is purely additive within the current
  void-returning shape.
- 2: claimed bool return on `applyRSSIndirectionOne`. Master
  has void return. Plan now uses void `maybeRestoreDefault`
  helper consistent with master shape.

MED:
- 3: "uses every queue at least once" check accepts
  false-positive customs. §4 captured empirically on
  loss:xpf-userspace-fw0 — mlx5 default is exact round-robin
  `entry[i] = i mod queue_count`. §7 tightened to exact-match.
- 4: empirical default-table capture committed via §4 +
  fixture in test #8.
- 5: skip-reason was string-parsing. §5 uses structured
  `workers > 1 && workers >= queues` condition directly.
- 6: probe-error behavior unspecified. §6 mirrors existing
  apply-path's ErrNotFound + generic err handling.
- 7: missing tests for queue_count=0, workers>queues
  (not just ==). §9 tests #2 + #6 added.

LOW:
- 8: workers∈{0,1} regression guard added (§9 tests #4 + #5).
- 9: `make test-failover` demoted from merge blocker to
  optional defense-in-depth.

§13 review-response table maps each finding to resolution.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 plan — address Codex round 2 (1 MED real bug, 2 LOW spec, 1 LOW wording, 1 LOW test)

R2 verdict was PLAN-READY NO. Fixes:

MED:
- 2: indirectionTableIsDefault could return true vacuously on
  empty/unparseable ethtool -x output. Added sawAnyRow guard
  (mirrors existing indirectionTableMatches shape) + §9 test
  #11 to pin the empty-output case.
- 3: runtime queue-count-only changes (ethtool -L without config
  commit) explicitly listed out-of-scope in §12. Operators
  changing ringparam are expected to follow with a config
  commit. Netlink-watch loop for ringparam events is separate
  scope.

LOW:
- 1: §3 wrongly cited d.applySem as the startup-vs-reconcile
  serializer. Real mechanism is lifecycle ordering — startup
  runs from enumerateAndRenameInterfaces before API/CLI is
  wired up. Wording corrected so reviewers don't waste time
  chasing a non-existent semaphore.
- 4: §9 test #8 fixture location pinned: inline string literal,
  not a testdata/ file. Output is small enough to embed cleanly.
- 5: §9 test #12 added: BootSequence_4then6_RestoresDefault
  covers the operator workflow end-to-end (workers=4 writes
  constrained, workers=6 sees stale and restores default).

§13 R2 review-response table added (5 findings).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 plan — address Codex round 3 (1 LOW edge case)

R3 LOW: sawAnyRow flag was set when a row-index parsed but
BEFORE any queue token was parsed. Input like "0:\n" (row index
with empty value list) would set sawAnyRow=true, the inner
for-loop over bytes.Fields would not execute, and the function
would return true — vacuously-default, exactly the failure mode
R2#2 was supposed to prevent.

Fix: rename sawAnyRow → sawAnyEntry, set inside the inner field
loop only AFTER a queue value has been parsed AND matched the
expected round-robin position. Now requires at least one verified
queue entry before returning true.

§9 test #11 expanded to enumerate three failure cases:
1. empty []byte{}
2. non-row text (header only)
3. row index with no queue tokens (the R3 case)

§13 R3 review-response row added.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 D3 RSS indirection refresh on workers↔queues transition

When `system dataplane workers` is bumped from a value below the
NIC RX-queue count to a value at-or-above it (e.g. 4 → 6 on a
6-queue mlx5), the previously-written `[1,1,1,1,0,0]` indirection
table stays live. Queues 4 and 5 now host worker-bound AF_XDP
sockets but receive no RSS traffic, starving those workers.

Root cause: `applyRSSIndirectionOne` calls `computeWeightVector`,
which returns nil when `workers >= queues`. The caller treats
nil as "skip" and never touches the live table. Correct on a
fresh install (kernel default round-robin = what we want) but
wrong on the workers<queues → workers>=queues transition.

Fix: on the nil-weights skip path, when
`workers > 1 && workers >= queues > 0`, inspect the live
indirection table; if it isn't the kernel's default round-robin
shape, run `ethtool -X iface default` to restore it.

`indirectionTableIsDefault` is a strict round-robin parser:
`entry[i] == i mod queueCount` exactly. Verified empirically
against the live mlx5 default table on
`loss:xpf-userspace-fw0/ge-0-0-2`. Stricter than
`indirectionTableMatches` (which would accept any custom
table that uses every queue at least once); rejects empty,
unparseable, value-less, and non-round-robin inputs via
`sawAnyEntry` guard.

`maybeRestoreDefault` mirrors the existing apply-path's
ethtool-probe failure handling: ErrNotFound → log Warn, return;
generic err → log Warn with output, return; never attempts a
write on probe failure.

Other skip paths (workers <= 0, workers == 1) leave the table
alone — those bring-up / single-worker cases have no prior
workers<queues state to undo.

12 new tests in pkg/daemon/rss_indirection_test.go:
- 7 behavioral (workers transition cases, stale-table preserved
  on workers∈{0,1}, queueCount=0 short-circuit, probe-failure
  skip)
- 4 parser (round-robin true, concentrated false, every-queue-
  but-non-round-robin false, empty/unparseable/value-less false)
- 1 end-to-end (BootSequence_4then6 covers operator workflow:
  step 1 writes [1,1,1,1,0,0]; step 2 sees stale and restores)

Plan + 4 Codex review rounds documented at
docs/pr/805-rss-refresh/plan.md (PLAN-READY YES at R4).

Closes #805.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 address Codex code-review LOWs (3 fixes: doc + 4 tests)

Codex code review verdict: MERGE YES with 3 LOW (cosmetic doc +
missing test coverage). All applied:

LOW 1: Stale §M2 invariant comment in applyRSSIndirection's
docblock said "workers >= queue_count is skipped" but the new
maybeRestoreDefault path probes and may write. Updated to
reflect the new behavior.

LOW 2: Generic-error branches in maybeRestoreDefault (non-
ErrNotFound -x probe failure; non-ErrNotFound -X default write
failure) had no regression coverage.
- Added argvErr scripted-error mechanism to fakeRSSExecutor:
  argv-prefix-keyed (out, err) tuples that take precedence
  over the existing per-iface ethtoolX/ethtoolC paths.
- Test #13 RestoreEthtoolXProbeGenericError_LogAndSkip: -x
  returns generic error → no -X default invocation.
- Test #14 RestoreEthtoolXDefaultGenericError_LoggedAndSwallowed:
  -X default returns generic error → both probe and write
  recorded, function returns normally without propagating.

LOW 3: New branch's interaction with non-mlx5 / empty-driver
guard untested at applyRSSIndirectionOne entry-point level.
- Test #15 NonMlxDriver_WorkersEqualsQueues_NotTouched:
  virtio_net driver with workers=6,queues=6 stale table →
  zero ethtool calls (driver guard short-circuits before
  restore path).
- Test #16 EmptyDriver_WorkersEqualsQueues_NotTouched: empty
  driver string (sysfs unreadable) → same expectation.

All 16 #805 tests pass plus 880+ existing tests clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 address Copilot inline (queues>0 vs queues>1 mismatch)

Comment in applyRSSIndirectionOne said "workers >= queues > 1"
but the guard was `workers > 1 && workers >= queues && queues > 0`.
On a single-queue NIC (queues=1) the guard would let
maybeRestoreDefault run, even though there's no possible
concentration to undo: queueCount=1 means entry[i] = i mod 1 = 0
for every i, which is both the default and the only possible
layout.

Tighten the guard to `queues > 1` to match the comment's intent
and skip the unnecessary probe on single-queue NICs.

Test #17 QueueCountOne_NoOp pins the new behavior with
workers=6, queues=1, stale-looking table — expects zero
ethtool calls.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #805 address Copilot inline review (5 fixes — clarity + determinism)

1. rss_indirection.go:242 — log "rss weight reshaping skipped"
   instead of "rss indirection skipped" so it's clear that the
   weight-vector write is what's skipped, not the whole RSS path.
   When workers>=queues>1 with a stale table, the next line(s)
   show the actual restore action.

2. rss_indirection_test.go:47 — comment for argvErr said
   "checked AFTER" but code checks BEFORE. Aligned to actual
   precedence.

3. rss_indirection_test.go:71 — Go map iteration is randomized,
   so two overlapping prefix keys would produce flaky results.
   Implemented longest-prefix-wins tie-breaking for determinism.

4. rss_indirection_test.go:846 — test #17 appeared before #16.
   Reordered to be sequential.

5. queues>1 vs queues>0 mismatch was already fixed in 744f8b2
   (Copilot saw the change but flagged a stale comment-vs-code
   diff — addressed via the comment update accompanying #1).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 26, 2026
- mouse_latency_probe.py: drop unused `random` and `struct` imports
  (Copilot R2 #6).
- mouse_latency_probe.py: docstring now documents all three
  min-attempts floors (M=1: 500, 2≤M<10: 1000, M≥10: 5000) so the
  intermediate branch is no longer surprising (Copilot R2 #5).
- test-mouse-latency.sh: distinguish iperf3-settle pull failure
  from a real cwnd-not-settled, with `INVALID-iperf3-settle-pull-failed`
  attribution (Copilot R2 #1).
- test-mouse-latency.sh: explicit probe.json availability check —
  `INVALID-probe-pull-failed`/`probe-missing`/`probe-invalid-json`
  instead of letting the matrix wrapper silently lose attribution
  (Copilot R2 #2).
- test-mouse-latency.sh: journalctl stderr now lands in
  `${OUT_DIR}/jc-stderr-${FW}.txt` instead of a per-rep `/tmp/`
  file on the caller, so it's captured alongside the other rep
  artifacts (Copilot R2 #3).
- test-mouse-latency-matrix.sh: preflight JSON parsing wrapped in
  try/except + .get() with explicit `FAIL invalid-json=...` /
  `FAIL missing-field=...` lines, so a partial write or schema
  drift produces an actionable diagnosis instead of an aborting
  stack trace (Copilot R2 #4).

70 tests still green; both shell scripts pass `bash -n`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 26, 2026
- mouse_latency_probe.py: per-attempt connect/recv timeouts now
  bounded by remaining time to the deadline, so a probe near the
  deadline can no longer overrun by up to 10s and consume the
  iperf3 slack budget. Probe runtime now consistently ≤ duration
  + small constant. (Copilot R3 #1)

- mouse_latency_probe.py: payload generated once per coroutine
  instead of per-attempt — uniqueness is irrelevant for the
  byte-stateless echo path; the per-attempt os.urandom was
  avoidable CPU on the source. (Copilot R3 #2)

- test-mouse-latency-matrix.sh: WALL_CAP_HIT flag so the outer
  cell loop also stops once the wall budget is exceeded —
  previously `return 0` only exited run_cell and the outer loop
  would iterate the remaining cells, repeatedly tripping the cap.
  (Copilot R3 #3)

- test-mouse-latency-matrix.sh: rep_is_valid wraps the JSON
  parse + key access in try/except so a malformed probe.json or
  schema drift produces a clean "invalid" verdict instead of a
  Python stack trace in the matrix log. (Copilot R3 #4)

- test-mouse-latency.sh: apply-cos-config now targets the current
  RG0 primary (resolved via current_primary) instead of hard-coded
  fw0. If the cluster is already failed over before the rep
  starts, hardcoding fw0 would attempt to apply the fixture on
  the secondary. (Copilot R3 #5)

- test-mouse-latency.sh: screen-pre snapshot now captured from
  the same node the post-snapshot will follow (current_primary at
  pre-time, same node at post-time). The pre/post diff is now
  consistent regardless of which node was primary at rep start.
  Records the captured node in screen-pre-fw.txt for analyst
  cross-reference. (Copilot R3 #6)

70 tests still green; both shell scripts pass `bash -n`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 26, 2026
…w-up (#906)

* #905 mouse-latency tail: plan v4 (7 Codex rounds, PLAN-NEEDS-MINOR)

Plan for measurement-only PR characterizing mouse-latency tail
under elephant load using the operator-provisioned echo server
on 172.16.80.200:7. 12-cell run matrix (N elephants × M mice),
PASS gate at p99(N=128,M=10) ≤ 2× idle baseline.

Seven Codex hostile review rounds; disposition tables in §11:
- R1: 17 findings, PLAN-NEEDS-MAJOR
- R2: 7 findings, PLAN-NEEDS-MAJOR
- R3: 6 findings, PLAN-NEEDS-MAJOR
- R4: 8 findings, PLAN-NEEDS-MAJOR
- R5: 5 findings, PLAN-NEEDS-MINOR
- R6: 3 findings, PLAN-NEEDS-MINOR
- R7: 2 findings, PLAN-NEEDS-MINOR

Stopping plan iteration at R7 per the #838 lesson: late rounds
surfacing minor spec-clarity issues with no structural defects
indicate it's time to let code review take over. Implementation
follows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency tail: harness implementation

Adds the measurement harness specified in
docs/pr/905-mouse-latency/plan.md. 12 new files in test/incus/:

Python parsers + tests (no third-party deps; stdlib statistics +
asyncio only):
- cluster_status_parse.py — `cli show chassis cluster status` →
  list of (rg_id, node_id, state) triples (incl. secondary-hold)
- iperf3_sum_parse.py — iperf3 text-mode `[SUM]` row → bps
- mouse_latency_probe.py — closed-loop M-coroutine TCP probe
  driver with histogram + statistics.quantiles percentiles
  + per-coroutine RPS distribution + validity gate
- mouse_latency_aggregate.py — per-cell median-by-p99 +
  decision verdict; honours orchestrator INVALID-* markers
- mouse_latency_orchestrate.py — cwnd-settle / collapse /
  RG-flap helper subcommands

Shell orchestrator + matrix wrapper:
- test-mouse-latency.sh — one rep with the full validity
  pipeline from plan §4.5 (CoS preflight, mpstat over the
  probe window only, dual-node journalctl HA-transition diff,
  1Hz RG state polling, post-snapshot follows current primary)
- test-mouse-latency-matrix.sh — 12-cell matrix with
  preflight gating + 10/15-rep accounting per plan §4.7

68 unit tests pass via
`python3 -m unittest discover -s test/incus/ -p '*_test.py'`.

Five rounds of Codex hostile code review; each round's findings
addressed inline. Smoke run on the loss cluster has not yet
been executed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency: address Copilot R1 review (8 inline comments)

- plan.md: filename references hyphenated → underscored to match the
  Python module names actually shipped (mouse_latency_probe.py etc.).
- plan.md: §7.1 merge gate dropped "findings.md exists" (the harness
  PR merges first; findings come in a follow-up after the matrix runs).
- plan.md: §4.5 step 2 now states the implemented mpstat lifecycle
  (`mpstat 1 <duration>` over the probe window only, not duration+30).
- plan.md: §4.5 step 3 clarifies that polling the primary alone is
  sufficient because `Manager.FormatStatus` returns both nodes' rows
  in one query.
- mouse_latency_probe.py: compute_validity uses the previously-unused
  `completed` parameter to surface count-bookkeeping inconsistencies.
  Two new unit tests.
- mouse_latency_probe.py + mouse_latency_aggregate.py: per-coroutine
  RPS distribution renamed to attempts_per_second_per_coroutine_*
  (the previous achieved_rps_per_coroutine_* conflated workload-
  offered attempts with completion-rate, which uses different
  totals).
- test-mouse-latency-matrix.sh: top-of-file comment now describes
  the actual rep-accounting behavior ("up to 15 reps as needed for
  10 valid"), not a >30%-trigger conditional that was simplified
  out.
- test-mouse-latency-matrix.sh: preflight comment explains the 60s
  duration choice (M=1 floor of 500 attempts).

70 tests green via the discover command from §7.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency: address Codex R6 (1 HIGH + 1 MED)

R6 found a stale-artifact contamination bug. Two cells with the
same rep index (e.g. cell_N0_M10/rep_00 and cell_N128_M10/rep_00)
both wrote to /tmp/probe-rep_00.json on the source container; a
failed pull on the second cell silently inherited the first
cell's data. Includes the cell name in REP_TAG so remote temp
paths are unique, plus an explicit `rm -f` of the temp files at
rep start as defense in depth.

R6 MED: a failover/failback hidden inside missed 1Hz RG poll
samples could pass as stable. Added a final RG-state snapshot at
end of rep and an explicit initial-vs-final triple-set comparison
(in addition to the per-sample drift check on the poll log).

Both shell scripts pass `bash -n`; 70 unit tests still green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency: address Codex R7 HIGH (local stale-artifact)

R6 HIGH (remote temp-file collision) and R7 fix moved both REP_TAG
and remote-side rm to be cell-aware. Codex R7 found the same class
of bug on the LOCAL side: if OUT_DIR is reused (e.g. a rerun into
an existing rep dir) and the probe run fails before overwriting
probe.json, the previous run's stale data would silently survive
and be picked up by `rep_is_valid` in the matrix wrapper.

Add an explicit rm of all per-rep local artifacts (probe.json,
iperf3.txt, mpstat.txt, RG state files, INVALID-* markers, etc.)
at rep start.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency: address Copilot R2 review (6 inline comments)

- mouse_latency_probe.py: drop unused `random` and `struct` imports
  (Copilot R2 #6).
- mouse_latency_probe.py: docstring now documents all three
  min-attempts floors (M=1: 500, 2≤M<10: 1000, M≥10: 5000) so the
  intermediate branch is no longer surprising (Copilot R2 #5).
- test-mouse-latency.sh: distinguish iperf3-settle pull failure
  from a real cwnd-not-settled, with `INVALID-iperf3-settle-pull-failed`
  attribution (Copilot R2 #1).
- test-mouse-latency.sh: explicit probe.json availability check —
  `INVALID-probe-pull-failed`/`probe-missing`/`probe-invalid-json`
  instead of letting the matrix wrapper silently lose attribution
  (Copilot R2 #2).
- test-mouse-latency.sh: journalctl stderr now lands in
  `${OUT_DIR}/jc-stderr-${FW}.txt` instead of a per-rep `/tmp/`
  file on the caller, so it's captured alongside the other rep
  artifacts (Copilot R2 #3).
- test-mouse-latency-matrix.sh: preflight JSON parsing wrapped in
  try/except + .get() with explicit `FAIL invalid-json=...` /
  `FAIL missing-field=...` lines, so a partial write or schema
  drift produces an actionable diagnosis instead of an aborting
  stack trace (Copilot R2 #4).

70 tests still green; both shell scripts pass `bash -n`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency: address Codex R9 (2 small fixes)

- test-mouse-latency.sh stale-artifact cleanup list now includes
  jc-stderr-*.txt so a rerun into an existing rep dir doesn't
  inherit prior journalctl stderr (Copilot R2 #3 introduced this
  artifact path; Codex R9 noticed it was missing from the wipe).

- test-mouse-latency-matrix.sh preflight JSON parser now coerces
  `validity` to {} when it isn't a dict, so schema-drifted JSON
  produces a clean FAIL line instead of stack-tracing on
  `validity.get(...)` (Copilot R2 #4 partial residual).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #905 mouse-latency: address Copilot R3 review (6 inline comments)

- mouse_latency_probe.py: per-attempt connect/recv timeouts now
  bounded by remaining time to the deadline, so a probe near the
  deadline can no longer overrun by up to 10s and consume the
  iperf3 slack budget. Probe runtime now consistently ≤ duration
  + small constant. (Copilot R3 #1)

- mouse_latency_probe.py: payload generated once per coroutine
  instead of per-attempt — uniqueness is irrelevant for the
  byte-stateless echo path; the per-attempt os.urandom was
  avoidable CPU on the source. (Copilot R3 #2)

- test-mouse-latency-matrix.sh: WALL_CAP_HIT flag so the outer
  cell loop also stops once the wall budget is exceeded —
  previously `return 0` only exited run_cell and the outer loop
  would iterate the remaining cells, repeatedly tripping the cap.
  (Copilot R3 #3)

- test-mouse-latency-matrix.sh: rep_is_valid wraps the JSON
  parse + key access in try/except so a malformed probe.json or
  schema drift produces a clean "invalid" verdict instead of a
  Python stack trace in the matrix log. (Copilot R3 #4)

- test-mouse-latency.sh: apply-cos-config now targets the current
  RG0 primary (resolved via current_primary) instead of hard-coded
  fw0. If the cluster is already failed over before the rep
  starts, hardcoding fw0 would attempt to apply the fixture on
  the secondary. (Copilot R3 #5)

- test-mouse-latency.sh: screen-pre snapshot now captured from
  the same node the post-snapshot will follow (current_primary at
  pre-time, same node at post-time). The pre/post diff is now
  consistent regardless of which node was primary at rep start.
  Records the captured node in screen-pre-fw.txt for analyst
  cross-reference. (Copilot R3 #6)

70 tests still green; both shell scripts pass `bash -n`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 26, 2026
1. mouse_latency_orchestrate.py: rg-state-flapped sample sort now
   keys on int(ts) rather than the string ts to avoid lexicographic
   mis-ordering if timestamp digit width ever varies (Copilot R3 #1).

2. mouse_latency_aggregate.py: select_valid_reps now requires p99
   to be numerically present, not just validity.ok=True. Median
   selection no longer coerces missing p99 to 0 (which could
   mis-pick a malformed rep as the median and skew the verdict).
   (Copilot R3 #2)

3. test-mouse-latency.sh: rename "SYN-cookie counter snapshot" to
   "Screen flood-counter snapshot" — the underlying CLI command
   `show security screen statistics zone wan` reports SCREEN
   flood-event counters, NOT SYN-cookie-specific counters. The
   manifest's `screen_engaged` field still measures the right
   signal; just the comments were misleading. (Copilot R3 #3, #4)

Edits are mid-flight-safe for the running iperf-c shared matrix
(naming and parsing fixes; no behavior change to active reps).

72 tests still green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request Apr 29, 2026
* docs: #921 plan v1 — ifindex_to_zone + EgressInterface.zone → u16

* docs: #921 plan v2 — address Codex round-1 NEEDS-REVISION

Codex round-1 (task-mojw27u8-zcmsih, 4m50s) found 5 real issues
(plus one wire-compat confirmation):

1. GRE handling was misdescribed: gre.rs:260-265 already produces
   a u16 ingress_zone for UserspaceDpMeta — after this PR it becomes
   a single direct id lookup, not the v1's "translate to name via
   zone_id_to_name" round-trip. Critically, GRE decap fires from
   afxdp.rs:947-955 BEFORE the flow cache, so this is a per-packet
   path on GRE-tunnel workloads (not just session-miss). Real win.

2. Unknown / skipped-zone invariant was underspecified. After this
   PR an interface pointing at a dropped zone (reserved id, > u8 max)
   collapses to zone_id == 0. v2 documents this as the canonical
   "unknown" sentinel and adds two tests:
   - interface_pointing_at_skipped_zone_collapses_to_zone_id_zero
   - egress_with_unknown_zone_name_collapses_to_zone_id_zero

3. Churn inventory was wrong. Real numbers:
   - ifindex_to_zone references: 12 in 5 Rust files.
   - EgressInterface constructors: 31 across 9 files.
   - Total touched: ~14 files; estimate <300 LOC.
   - test_fixtures.rs has NO EgressInterface ctors (v1 was wrong).

5. Debug-log: must update both forwarding_build.rs:440 (FWD_STATE
   ifindex_to_zone) AND :445-451 (per-egress eg.zone). v1 only
   covered the first.

6. EgressInterface read sites: only 5 real reads (forwarding.rs:147,
   :176, forwarding_build.rs:450, tunnel.rs:184-188, frame.rs:3625
   test). No long-lived String borrows that would force translation.

(4) wire compat is correct — confirmed unchanged.

Test count target updated 849 → 851 (+4 new tests including the
two unknown-zone collapse tests).

* docs: #921 plan v2.1 — correct EgressInterface ctor file list (Codex round-2)

* #921: ifindex→zone and EgressInterface.zone string→u16

Eliminates the last two String-valued zone maps in ForwardingState
that survived #919/#922:

1. ForwardingState.ifindex_to_zone: FastMap<i32, String>
   → ifindex_to_zone_id: FastMap<i32, u16>.
2. EgressInterface.zone: String → zone_id: u16.

Both maps are populated once at config commit time (in
forwarding_build.rs) by translating the snapshot's zone NAME via
zone_name_to_id. Hot-path callers now do ONE direct u16 lookup
where they previously did TWO HashMap lookups (ifindex → String,
then String → u16) plus a String hash on the second.

Per Codex round-1 finding #1, GRE decap (afxdp.rs:947-955) fires
BEFORE the flow cache, so gre.rs:260-265's zone-id resolution is
on the per-packet path for matching native GRE traffic — the
direct-id win there is per-packet, not just session-miss.

Per Codex round-1 finding #2, an interface whose snapshot zone
string refers to a zone that was dropped at config build (reserved
id, > u8 max) collapses to zone_id == 0 (the canonical "unknown"
sentinel). Tests pin this invariant.

Files (per Codex round-1 inventory correction):
- types.rs: 2 field type changes.
- forwarding_build.rs: 4 sites — populate ifindex_to_zone_id from
  snapshot zone NAME, populate EgressInterface.zone_id, render
  both debug-log lines via zone_id_to_name, plus 4 new tests.
- forwarding.rs: 3 sites — zone_pair_for_flow_with_override
  (test-only, slow-path String-from-id), zone_pair_ids_for_flow_with_override
  (hot path: direct u16 lookup), cluster_peer_return_fast_path.
- gre.rs:260: per-packet GRE decap direct id lookup.
- tunnel.rs:184: direct EgressInterface.zone_id load.
- afxdp.rs: 3 sites — screen profile lookup (slow path) + DNAT
  fallback (slow path) + zone-encoded fabric redirect (direct id).
- 30 EgressInterface constructors across 9 files (coordinator,
  flow_cache, frame, frame_tx, ha, session_glue, tests, worker,
  forwarding_build): zone: "lan".to_string() → zone_id: TEST_LAN_ZONE_ID.
  Done via targeted Python pass (stateful tracking of EgressInterface
  scope to avoid corrupting the unrelated InterfaceSnapshot.zone and
  TunnelEndpointSnapshot.zone snapshot fields, which this PR DOES NOT
  touch — those remain String wire-protocol fields).
- frame.rs:3625 test assertion: egress.zone → egress.zone_id.

cargo test --release: 851 passed (847 baseline + 4 new), 0 failed.
cargo build --release: clean.

Plan: docs/pr/921-zone-id-maps/plan.md (v2.1, Codex round-3
PLAN-READY at task-mojwmnmr-d63ipm in 17s).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 2, 2026
Underscore prefix is unnecessary — these helpers are used inside
the test module. Renames are cosmetic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 2, 2026
 prereq) (#1141)

* fix(forwarding): correct clamp_tcp_mss incremental checksum update

The incremental TCP checksum update in clamp_tcp_mss had a sign-flipped
delta, producing a checksum that did NOT preserve the sum-over-all = 0xFFFF
invariant a TCP checksum needs to be valid on the wire.

  buggy: sum = ~old_csum + old_val + ~new_val; HC' = ~sum
         => HC' = old_csum + (new_val - old_val)
  right: sum =  old_csum + old_val + ~new_val; HC' =  sum
         => HC' = old_csum + (old_val - new_val)

Per RFC 1624 / RFC 1071: HC' = HC + m + ~m'. The previous formulation
double-negated, flipping the sign of the delta. Independent ones-complement
recomputation of the post-clamp packet sums to 0xFFFF − 2·(old_val − new_val)
instead of 0xFFFF.

Caught by a new regression test (`clamp_tcp_mss_v4_preserves_checksum_invariant`)
that builds a SYN+MSS packet, runs clamp_tcp_mss, and validates the
checksum via independent recomputation. Fails on the buggy formula
(left=65015 right=65535), passes on the fix.

Why the bug never surfaced as wire-visible breakage:

`clamp_tcp_mss` is `#[allow(dead_code)]` but does have callers — the
copy-builder path in `build_forwarded_frame_into_from_frame` at
`frame/mod.rs:315` and `:354`, reachable from TX fallback in
`tx/dispatch.rs:462-469` and `:721-725`. The live GRE encap path masks
the bad incremental result: `force_tunnel_l4_recompute` is set at
`frame/mod.rs:260` whenever `tunnel_tcp_mss > 0`, which causes
`recompute_l4_checksum_*` at `frame/mod.rs:317-318` and `:356-357` to
overwrite the incremental checksum from scratch right after the clamp.
So today's GRE path produces a correct on-wire TCP checksum despite
the broken incremental update — but any future caller that does not
re-emit the L4 checksum from scratch would have hit dropped packets.

The follow-up #989 PR (L4 protocol specialization, `frame/tcp.rs`)
relocates `clamp_tcp_mss` and adds colocated unit tests with the same
independent-recomputation oracle, making this kind of regression
visible immediately at the unit-test layer.

Verified independently that `checksum16_adjust` (frame/checksum.rs:146),
`compute_ip_csum_delta`, and `compute_l4_csum_delta` (afxdp/checksum.rs)
use the conventional RFC-1624 form `~(~HC + ~m + m')` which is correct;
the bug was isolated to `clamp_tcp_mss`.

Closes prerequisite for #989 PR-B.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* PR-A: drop underscore prefix on test helpers (Copilot #1)

Underscore prefix is unnecessary — these helpers are used inside
the test module. Renames are cosmetic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 3, 2026
- worker/mod.rs:92 — update stale comment from "7 TX pipeline
  fields" to "8 TX pipeline fields" (Codex Low #1).
- tx_pipeline.rs — clarify in both module-level and field-level
  docstrings that `outstanding_tx` increments at TX ring descriptor
  insertion, NOT at `sendto` (which is the wake/kick) (Copilot).

Comments-only; no behavioral change. Build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 3, 2026
* #959 Phase 10: relocate outstanding_tx into WorkerTxPipeline

Pure structural follow-on to Phase 7. The `outstanding_tx: u32`
gauge was held back from Phase 7 because of the type-collision
with `BindingStatus.outstanding_tx` (the snapshot mirror at
`coordinator/mod.rs:1216,1353` and `protocol.rs:1394,1515,1611`).
Phase 10 leverages the type-level disambiguation: Rust's compiler
keeps the BindingWorker accesses (now via `tx_pipeline.outstanding_tx`)
distinct from the BindingStatus accesses (still at the flat field).

Field moves from `BindingWorker` directly into the existing
`WorkerTxPipeline` sub-struct, slotted right after `max_pending_tx`
(its semantic neighbor — the TX-pipeline backpressure threshold).
Documented as a transient gauge — incremented at TX insert,
decremented when the completion ring is reaped, mirrored once per
debug tick to `BindingLiveState.debug_outstanding_tx` for the
snapshot reader (#802).

22 BindingWorker accesses rewritten via a per-file perl pass scoped
to the files that touch worker state (tx/{drain,dispatch,transmit,
rings}.rs, cos/queue_service/service.rs, worker/mod.rs `b`/`sb`
iter aliases). The BindingStatus mirror sites are untouched.

Verified: cargo build clean, 952/952 cargo tests pass, 30 Go
packages pass, v4 smoke 957 Mbps + v6 smoke 944 Mbps against
172.16.80.200 / 2001:559:8585:80::200 on the loss userspace cluster.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #959 Phase 10: address Codex + Copilot review

- worker/mod.rs:92 — update stale comment from "7 TX pipeline
  fields" to "8 TX pipeline fields" (Codex Low #1).
- tx_pipeline.rs — clarify in both module-level and field-level
  docstrings that `outstanding_tx` increments at TX ring descriptor
  insertion, NOT at `sendto` (which is the wake/kick) (Copilot).

Comments-only; no behavioral change. Build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 3, 2026
- xsk_rings.rs — remove unused `use super::*;` (Codex Low #1 +
  Copilot). The struct uses fully-qualified `crate::xsk_ffi::*`
  paths; the glob import was an extra-cautious carryover from
  the other phases. Drops 94 warnings → 93.
- tx/rings.rs:218 — clarify the pre-existing TX kick comment
  (Copilot). The reference to a `.wake()` method was inherited
  from before Phase 11; rewritten to describe the actual behavior
  (direct sendto() syscall, errno observable for #825 kick-latency).

Comments-only otherwise; no behavioral change. Build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 3, 2026
…1177)

* #959 Phase 11: extract XSK rings into WorkerXskRings sub-struct

Final phase of the BindingWorker decomposition. The three flat
XSK kernel-ring fields — `device: DeviceQueue`, `rx: RingRx`,
`tx: RingTx` — move into a dedicated `WorkerXskRings` sub-struct
in worker/xsk_rings.rs. Access pattern: `binding.xsk.device`,
`binding.xsk.rx`, `binding.xsk.tx`.

This was held back as the highest-risk #959 phase because of two
name-collision concerns:

- `off.rx` / `off.tx` in `bpf_map.rs` (XDP socket ring offsets,
  read by `read_ring_pair()` for diagnostic ring-state dumps)
- `telemetry.dbg.rx` / `telemetry.dbg.tx` in `poll_descriptor.rs`
  (in-loop diagnostic counters on a *separate* telemetry struct,
  not BindingWorker.telemetry)

Both disambiguate by alias prefix — `off.` and `dbg.` never
overlap with the BindingWorker aliases (`binding`, `b`, `sb`,
`target_binding`, `other_binding`) — so a per-file perl rewrite
scoped to the alias whitelist is safe. Verified by grep that no
`off.xsk.` or `dbg.xsk.` accesses appear in the result.

29 BindingWorker accesses rewritten across:
- afxdp/poll_descriptor.rs (1 + 1 doc-comment)
- afxdp/cos/queue_service/service.rs (5)
- afxdp/tx/rings.rs (8)
- afxdp/tx/transmit.rs (2)
- afxdp/worker/lifecycle.rs (3)
- afxdp/worker/mod.rs (10, including b/sb iter aliases on
  bindings.iter() in the per-second debug-tick and stall-detect
  paths)

Constructor change: `BindingWorker { ..., user, device, rx, tx, ... }`
becomes `BindingWorker { ..., user, xsk: WorkerXskRings { device, rx, tx }, ... }`.

#959 BindingWorker decomposition is now complete (Phases 1-11):
telemetry, scratch, cos, tx_counters, bpf_maps, timers,
tx_pipeline, bind_meta, flow, and xsk are all sub-structs.

Verified: cargo build clean (94 pre-existing warnings, 0 new),
952/952 cargo tests pass (`cargo test --release`), `tx_completion`
named tests 5/5 clean (flake check), 30 Go packages pass, v4
smoke 955 Mbps + v6 smoke 942 Mbps (both 0 retrans) on the loss
userspace cluster against 172.16.80.200 / 2001:559:8585:80::200.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #959 Phase 11: address Codex + Copilot review

- xsk_rings.rs — remove unused `use super::*;` (Codex Low #1 +
  Copilot). The struct uses fully-qualified `crate::xsk_ffi::*`
  paths; the glob import was an extra-cautious carryover from
  the other phases. Drops 94 warnings → 93.
- tx/rings.rs:218 — clarify the pre-existing TX kick comment
  (Copilot). The reference to a `.wake()` method was inherited
  from before Phase 11; rewritten to describe the actual behavior
  (direct sendto() syscall, errno observable for #825 kick-latency).

Comments-only otherwise; no behavioral change. Build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 4, 2026
Codex round 1: PLAN-NEEDS-MAJOR (5 blockers, "core slab idea
can fit"). Gemini round 1: PLAN-NEEDS-MINOR (one finding,
subsumed by Codex blockers).

v2 addresses each Codex blocker:

1. Missing handle→key path: SessionTable callers that need the
   canonical key (find_forward_nat_match, lookup_with_origin,
   etc.) must reach it from the handle. v2 stores
   SessionRecord{key, entry} inside the slab so any handle
   resolves to both. Subsumes Gemini's owner_rg_sessions
   handle→key question.

2. Stale-handle invariant misstated. v1 said "handles are local
   to one method call" — false because secondary indices store
   handles across calls. v2 invariant: handles persist only in
   eagerly-maintained internal indices; every such index must
   be cleaned BEFORE the slab slot is reused. Wheel is the only
   Key-based holdout.

3. Wheel STAYS key-based. v1 had contradictory wording. v2 is
   unambiguous: wheel keys on SessionKey because lazy-delete
   needs a stable identifier (slab handle reuse would point
   stale entries at wrong sessions).

4. entries.get(handle) not entries[handle]. v1's panic-on-stale-
   index pattern is wrong; today's HashMap.get returns None
   gracefully. v2 uses fallible Slab::get throughout.

5. owner_rg_sessions handle→key cost. Resolved by v2 decision
   #1 — SessionRecord stores the key inline.

Plus 2 perf claim corrections:
- "5-10x secondary inserts" overstated; key hash work remains.
  Actual saving is payload size (50→4 bytes) + tens of ns.
- v1 used 1M sessions but DEFAULT_MAX_SESSIONS = 131072. v2
  uses 131072 as baseline and shows ~41% memory reduction
  (~46 MB saved).

Plus one structural change:
- Microbenchmark promoted from optional to MANDATORY (Codex
  round-1 requirement).

5 open questions remain for round 2; the resolved ones from
v1 are dropped from the list.

Posting v2 for round-2 Codex + Gemini review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 4, 2026
* #964 Step 1 plan v1: SessionTable slab + integer handles (DRAFT)

Pre-implementation plan for Step 1 of the #964 SessionTable
multi-index refactor. Step 1 replaces the primary
sessions: FxHashMap<Key, Entry> with Slab<Entry> + key_to_handle:
FxHashMap<Key, u32>, and switches the 4 secondary indices from
Key→Key to Key→u32. Steps 2-4 (intrusive indices, cache packing)
deferred.

Honest framing: SessionTable lives on the SLOW PATH (flow-cache
miss fires <5% of packets). The win is bounded — memory ~33%
reduction at 1M sessions, insert/lookup latency ~2x on cache
miss. No L1-i benefit (data-structure refactor, not control-flow).

Plan calls out:
- Phase 2 dead-end pattern risk (LOW for this refactor — purely
  internal storage layout, no cross-packet reordering).
- HA sync compatibility (deltas stay key-based; handles are
  per-node, not portable).
- 7 open questions for review.

If reviewers conclude the perf gain is too small to justify the
churn, PLAN-KILL is an acceptable verdict — same methodology as
#946 Phase 2.

Posting for adversarial Codex + Gemini review before any code.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964 Step 1 plan v2: address Codex round-1 PLAN-NEEDS-MAJOR

Codex round 1: PLAN-NEEDS-MAJOR (5 blockers, "core slab idea
can fit"). Gemini round 1: PLAN-NEEDS-MINOR (one finding,
subsumed by Codex blockers).

v2 addresses each Codex blocker:

1. Missing handle→key path: SessionTable callers that need the
   canonical key (find_forward_nat_match, lookup_with_origin,
   etc.) must reach it from the handle. v2 stores
   SessionRecord{key, entry} inside the slab so any handle
   resolves to both. Subsumes Gemini's owner_rg_sessions
   handle→key question.

2. Stale-handle invariant misstated. v1 said "handles are local
   to one method call" — false because secondary indices store
   handles across calls. v2 invariant: handles persist only in
   eagerly-maintained internal indices; every such index must
   be cleaned BEFORE the slab slot is reused. Wheel is the only
   Key-based holdout.

3. Wheel STAYS key-based. v1 had contradictory wording. v2 is
   unambiguous: wheel keys on SessionKey because lazy-delete
   needs a stable identifier (slab handle reuse would point
   stale entries at wrong sessions).

4. entries.get(handle) not entries[handle]. v1's panic-on-stale-
   index pattern is wrong; today's HashMap.get returns None
   gracefully. v2 uses fallible Slab::get throughout.

5. owner_rg_sessions handle→key cost. Resolved by v2 decision
   #1 — SessionRecord stores the key inline.

Plus 2 perf claim corrections:
- "5-10x secondary inserts" overstated; key hash work remains.
  Actual saving is payload size (50→4 bytes) + tens of ns.
- v1 used 1M sessions but DEFAULT_MAX_SESSIONS = 131072. v2
  uses 131072 as baseline and shows ~41% memory reduction
  (~46 MB saved).

Plus one structural change:
- Microbenchmark promoted from optional to MANDATORY (Codex
  round-1 requirement).

5 open questions remain for round 2; the resolved ones from
v1 are dropped from the list.

Posting v2 for round-2 Codex + Gemini review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964 Step 1 plan v3: address Codex round-2 PLAN-NEEDS-MAJOR

Codex round 2: PLAN-NEEDS-MAJOR (12 specific findings + memory
math correction; "core direction is salvageable").
Gemini round 2: PLAN-NEEDS-MINOR (2 small recommendations).

v3 addresses each finding:

1-3. Lookup-path safety: lookup_with_origin now spells out
clone-key/drop-borrow sequencing AND adds explicit
key_matches_record validation as defense vs handle-reuse race.
Delete pseudocode order-of-ops bug fixed (read record FIRST,
then use its fields, THEN remove from slab).

4-6. Removal centralization: all session removal goes through
a single remove_entry helper (matches today's shape). Cleanup
is value-guarded (only remove secondary if stored handle ==
freed handle). Mandatory debug assertion scans handle-valued
indices for the freed handle after cleanup.

7-8. Memory math recomputed with hashbrown's actual semantics
(7/8 max load, power-of-two buckets, RawTable<(K,V)> tuple
padding). At 131072 sessions: ~182 MB → ~82 MB = ~55% saving
(better than v2's 41% estimate). FxHashMap = std HashMap
alias which uses hashbrown internally; math is identical.

9. Benchmark scope: covers insert + 3 lookup paths + GC
(expire_stale_entries) + owner_rg_session_keys (HA export
hot path). drain_deltas dropped per Codex (just drains
VecDeque). Bench reimplements SessionTable hot-path shape
because it's pub(crate) in a bin crate (matches established
tx_kick_latency.rs pattern).

10-12. Contradictions resolved: wheel STAYS key-based;
stale-handle invariant is "handles persist in eagerly-cleaned
internal indices" (drop the "local to method call" framing);
owner_rg uses SessionRecord-with-key only (drop reverse
handle→key map mention).

13. slab = "0.4" added to userspace-dp/Cargo.toml dependency
plan.

14. Iterator uses fallible entries.get() with filter_map for
defense-in-depth.

Posting v3 for round-3 Codex + Gemini review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964 Step 1 plan v4: clean rewrite addressing Codex round-3 PLAN-NEEDS-MAJOR

v3 accumulated 764 lines of layered v1/v2/v3 patch sections,
which Codex round 3 correctly noted contained stale
contradictory instructions ("must update wheel" vs "wheel
key-based", "handles local to method call" alongside the
opposite invariant, etc.). v4 is a clean rewrite that drops
the historical sections and presents a single coherent design.

5 Codex round-3 findings addressed:

1. lookup_with_origin alias-path validation: v3's
   key_matches_record was too strict — would reject valid
   reverse_translated_index alias lookups where
   record.key != lookup_key by design. v4 separates the
   validation: direct-primary path checks record.key ==
   lookup_key; alias path trusts the alias index. Plus the
   scoped-mutation block (TCP closing, last_seen_ns,
   expires_after_ns) before drop-borrow + push_to_wheel,
   matching today's session/mod.rs:413-468 pattern.

2. remove_entry cleanup completeness: v3 only removed one
   nat_reverse_index key. Today's session/mod.rs:987 removes
   BOTH reverse_wire AND reverse_canonical. v4's pseudocode
   includes both via guarded_remove helper.

3. Stale contradictory instructions removed: clean rewrite
   drops every "must update wheel" / "handles local to
   method call" / "5-10x" / "2x lookup" stale wording from
   the body. Single coherent design remains.

4. Memory math recomputed honestly: v3's "55% reduction"
   assumed 10 RGs × 100K = 1M owner_rg entries against 131072
   sessions. Codex caught this — owner_rg has at most N entries
   total. v4: realistic ~40% reduction (~129 MB → ~78 MB).

5. demote_owner_rg correctly noted as origin-mutation-in-place,
   not session removal. v4 doesn't route demotion through
   remove_entry.

Plus Gemini round-3 PLAN-NEEDS-MINOR (NAT alias concern) was
the same issue as Codex finding 1 — addressed by v4.

Posting v4 for round-4 review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964 Step 1 plan v5: Codex round-4 PLAN-NEEDS-MINOR addressed (Gemini PLAN-READY)

Round 4 converged: Gemini PLAN-READY, Codex PLAN-NEEDS-MINOR
with 5 tactical findings. v5 is the implementation-ready
version.

5 Codex round-4 findings addressed:

1. remove_entry primary-key guard: after key_to_handle.remove,
   verify record.key == *key BEFORE cleaning. Defends against
   a stale primary mapping pointing at a reused slot. Plus
   no_index_points_at extended to scan key_to_handle.values()
   too.

2. Insert gates match today exactly: forward_wire_index inserts
   only when forward_wire != forward_key (session/mod.rs:951);
   owner_rg_sessions only when owner_rg_id > 0 (session/mod.rs:
   963). v4 was unconditional — v5 mirrors today.

3. Release-mode alias validation: lookup_with_origin's alias
   path now checks `metadata.is_reverse &&
   translated_session_key(&record.key, record.entry.decision.nat)
   == *lookup_key`. Stale alias returns None, never a wrong
   reused-slot session.

4. Memory math caveat: numbers are plausible but the breakdown
   is structural. Implementation MUST replace estimates with
   measured std::mem::size_of output. Direction (~129 MB →
   ~78 MB) is believable; line items may shift 5-10%.

5. Bench scope: add alias-path lookup via reverse_translated_index
   (the path that caused multiple prior review failures). Add
   a NAT churn cycle that exercises both reverse_wire AND
   reverse_canonical cleanup.

Plan is PLAN-READY pending one final round-5 sanity check.
After 5 rounds the architecture has converged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964 Step 1 plan v5.1: drop stale alias-trust wording (Codex round-5 PLAN-NEEDS-MINOR)

Codex round 5 verified all 5 round-4 findings are addressed in
the v5 pseudocode but caught two stale-wording sites
contradicting the new release-mode validation:

- §"Validation rule" (line 222): said "alias path trusts the
  alias index" + "debug assertion catches it." Updated to
  describe the release-mode `metadata.is_reverse +
  translated_session_key` guard explicitly. Debug assertion
  is now correctly framed as additional defense for tests,
  not sole release-mode guard.

- Open question 2 (line 549): repeated v4's wording about
  trusting the alias index. Replaced with a "Resolved in v5"
  pointer to the new validation.

Codex round 5: "Delete/update the stale v4 text and this is
PLAN-READY."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964 Step 1: SessionTable slab + integer handles

Replace SessionTable's primary `sessions: FxHashMap<Key, Entry>`
with `entries: Slab<SessionRecord>` + `key_to_handle:
FxHashMap<Key, u32>`. Switch the 4 secondary indices
(nat_reverse, forward_wire, reverse_translated, owner_rg) from
Key→Key to Key→u32. Wheel STAYS key-based (lazy-delete needs a
stable identifier; slab handle reuse would point stale entries
at the wrong session).

Plan went through 5 rounds of Codex + Gemini adversarial
review (round 1-3 PLAN-NEEDS-MAJOR, round 4 PLAN-READY +
PLAN-NEEDS-MINOR, round 5 PLAN-NEEDS-MINOR doc cleanup).
Plan doc at docs/pr/964-session-multi-index/plan.md; final
v5.1 at commit 88e31a5.

Key design points (all from the plan):
- SessionRecord {key: SessionKey, entry: SessionEntry}
  inside the slab so any handle resolves to both. Required
  because find_forward_nat_match() etc. must return the
  canonical key, and lookup_with_origin's wheel push needs
  it after dropping the entry borrow.
- Centralized remove_entry helper: PRIMARY-KEY GUARD (verify
  record.key == *key before cleaning) + value-guarded
  secondary cleanup (only remove if stored handle == freed
  handle) + mandatory debug assertion that scans every
  handle-valued index for the freed handle.
- Path-specific lookup validation: direct-primary checks
  record.key == lookup_key; alias path verifies
  metadata.is_reverse + translated_session_key roundtrip.
  Stale alias returns None, never a wrong reused-slot session
  (release-mode guard, not just debug).
- Insert gates match today exactly: forward_wire_index only
  when forward_wire != forward_key; owner_rg_sessions only
  when owner_rg_id > 0.
- Wheel stays key-based — lazy-delete via wheel_tick mismatch
  needs a stable identifier.

Mandatory microbench in userspace-dp/benches/session_table.rs
(structural — reimplements SessionTable shape because the
production type is pub(crate) in a bin crate, same pattern as
tx_kick_latency.rs):

  lookup_reverse_nat: current 64ns/op → slab 25ns/op (2.5×)
  lookup_alias:       current 53ns/op → slab 26ns/op (2.0×)
  lookup_forward:     current 25ns → slab 28ns (small reg)
  insert_churn:       current 568ns → slab 754ns (small reg)
  owner_rg_export:    current 11µs  → slab 23µs (2× reg, rare path)

The reverse-nat and alias lookups are the dominant slow-path
operations on cache miss. Forward lookup and insert have
small regressions within the slab free-list overhead. The
owner_rg_export 2× is on an HA-failover-rare path.

Verified:
- cargo build clean (94 pre-existing warnings, 1 new for
  unused-fields-in-bench-types — Cargo dead_code lint).
- 952/952 cargo tests pass (cargo test --release).
- session-suite 5/5 named flake check clean.
- Go test suite: 30 packages pass.
- Deploy clean on loss userspace cluster.
- Per-class CoS smoke ALL CLEAN: 5201/iperf-a (961/945 Mbps)
  through 5206/iperf-f (6.53/6.44 Gbps), 0 retrans across all
  12 v4+v6 measurements against 172.16.80.200 /
  2001:559:8585:80::200.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964: address Codex MERGE-NEEDS-MINOR + Copilot 10 inline findings

Codex hostile (MERGE-NEEDS-MINOR) + Copilot (COMMENTED, 10 inline)
flagged real correctness/quality issues. Fixes:

Production code (Copilot):
- Slab::with_capacity(131072) → Slab::new(): start empty, let it
  grow on demand. The prior FxHashMap grew on demand so this
  matches baseline RSS (Copilot:line 180).
- Iterator paths (iter_with_origin, iter_with_idle_and_origin)
  now walk via key_to_handle and look up the slab via .get(),
  matching the plan's "primary index is authoritative" model
  (Codex Low + Copilot).
- expect message at the wheel re-bucket call now references
  entry_by_key instead of the stale ".get()" wording (Copilot).
- install_with_protocol_with_origin's remove_entry call return
  value is now bound to `_previous`, with a comment explaining
  the guard semantics (Copilot — full guard handling is debug-
  asserted via the existing no_index_points_at scan).

Bench (Codex Med + Copilot):
- Bench scenarios in the header now match what's actually
  implemented (drop nat_churn / gc_drain mentions). Both flagged
  by Codex + Copilot.
- Both CurrentTable and SlabTable gain a full remove() helper
  that cleans the secondary indices + owner_rg_sessions,
  mirroring SessionTable::remove_entry's eager-cleanup
  invariant. Without this, the churn benches grew the maps
  unbounded across criterion iterations, producing misleading
  numbers (Copilot — 4 separate comments).
- Re-bench result: insert_churn slab is now FASTER than current
  (255 vs 361 ns/op), not slower as v1 reported. The prior
  "small regression" was a benchmark artifact of leaked indices.

Plan doc (Codex Low + Copilot):
- Stale line-number refs (session/mod.rs:951, :963) replaced
  with stable symbol references (`index_forward_nat_key`'s
  forward_wire_index / owner_rg_id gates).

952 cargo tests still pass. Re-bench results:
  lookup_reverse_nat: current 55ns → slab 26ns (2.1×)
  lookup_alias:       current 55ns → slab 25ns (2.2×)
  lookup_forward:     current 24ns → slab 28ns (small reg)
  insert_churn:       current 361ns → slab 255ns (faster!)
  owner_rg_export:    current 10µs  → slab 23µs (2× reg, rare HA)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964: Codex round-2 fixes

Codex re-review (MERGE-NEEDS-MINOR) flagged 2 small issues:

1. upsert_synced_with_origin still ignored remove_entry's
   return value (only install was fixed). Now bound to
   `_previous` with the same comment as install.

2. Plan + bench wording mismatch: plan claimed nat_churn and
   gc_drain bench coverage was MANDATORY, while the bench
   header said they were deferred. Plan rewritten to:
   - Drop the claimed mandatory nat_churn/gc_drain coverage
     (insert_churn already exercises guarded-remove; GC pop
     shape adds nothing over session/tests.rs).
   - Acceptance criterion now explicitly: slab must NOT
     regress on reverse_nat/alias (the dominant slow-path
     operations); small regressions on lookup_forward and
     owner_rg_export are accepted in exchange for the
     secondary-index payload reduction.
   Bench header line 32 updated to match.

952 cargo tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964: address Copilot round-2 review (5 real findings)

GitHub re-anchored some of Copilot's round-1 comments to
fe0b16e even though the underlying code was fixed. The 5 truly
new round-2 findings:

1. len() returned entries.len() but key_to_handle is the
   authoritative primary index. Switch to key_to_handle.len()
   so an orphan slab record (theoretical, shouldn't exist
   post-eager-cleanup) doesn't inflate the count. The
   max_sessions gate in install_with_protocol_with_origin now
   uses self.len() too for consistency.

2. install_with_protocol_with_origin's "in debug builds, assert
   the guard did not fire" comment didn't match the actual code
   (no debug assertion at the call site). Comment updated to
   accurately reflect that the only assertion is inside
   remove_entry's no_index_points_at scan.

3. record_by_key / record_by_key_mut didn't validate
   record.key == *key — could return wrong session on stale
   key_to_handle pointing at a reused slab slot. Now checks the
   record's canonical key matches the lookup key before
   returning Some. Defends touch(), entry_with_origin(), etc.

4. remove_entry used .expect() on entries.get(handle) which
   would panic on stale key_to_handle BEFORE the PRIMARY-KEY
   GUARD ran. Made fallible: returns None with debug_assert.
   Now the guard is the only source of remove_entry returning
   None on input that should have succeeded.

5. Bench's forward_wire was unconditionally inserted as
   key→key, not mirroring production's gate
   (forward_wire != forward_key, key=wire_key value=handle).
   Both CurrentTable and SlabTable now compute a distinct
   wire_key (src_port+1) and insert with that gate.

Re-bench numbers after the production-mirroring fix:
  lookup_reverse_nat: current 56ns → slab 26ns (2.2×)
  lookup_alias:       current 57ns → slab 25ns (2.3×)
  lookup_forward:     current 24ns → slab 25ns (within noise)
  insert_churn:       current 324ns → slab 272ns (slab faster)
  owner_rg_export:    current 10µs  → slab 24µs (2.3× reg, rare HA)

952 cargo tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964: address Codex final-pass MERGE-NEEDS-MINOR (2 small)

Codex final pass on b372a9f found 2 small issues:

1. remove_entry's stale-handle path removed key_to_handle then
   returned None without restoring the mapping. Failed remove
   would mutate len() and leave the table inconsistent. Now
   restores the mapping via key_to_handle.insert before
   returning None (matches the existing primary-key-guard
   pattern below).

2. install/upsert comments said "the only assertion is the
   no_index_points_at debug_assert inside remove_entry" — but
   remove_entry now has multiple debug_assert!s (stale-handle
   guard + primary-key guard + no_index_points_at). Comments
   updated to accurately describe both guards as catching
   invariant violations in tests.

952 tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* #964: comment cleanup — enumerate all 3 debug_asserts in install/upsert

Codex sanity-check on 367c271 confirmed the remove_entry
restore fix is correct but flagged that install/upsert comments
mention "stale-handle and primary-key guards" without
enumerating the third no_index_points_at debug_assert.

Both install_with_protocol_with_origin and upsert_synced_with_origin
comments now explicitly list all 3 debug_assert!s in remove_entry:
- stale-handle guard
- PRIMARY-KEY GUARD
- no_index_points_at

952 cargo tests still pass. Doc-only change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 6, 2026
… (Codex round-4 #1) + scrub stale 'leak' / 'the leaked one' framing (Codex round-4 #2) + soften cache-line wording to acknowledge unspecified layout (Codex round-4 #3)
psaab added a commit that referenced this pull request May 6, 2026
…1202)

* #1187 plan v1: extend BatchCounters to cover disposition + screen_drops + tx_errors

* #1187 plan v2: scope narrowed (drop tx_errors, defer 3 small counters), reframe primary value as DDoS resilience per Gemini Pro 3.1, fix forward_candidate_packets leak per Codex finding #5, route through TelemetryContext per Codex finding #4

* #1187 plan v3: screen_drops MANDATORY batching (no fallback) per both reviewers; preserve exception_packets current semantics; correct cache-line wording

* #1187 plan v4: round-3 fixes — TelemetryContext (not BatchCounters), delete stale open-questions, correct cache-line wording, reframe forward_candidate_packets at disposition.rs:161 as cold (not hot-path) leak per Codex finding

* #1187 plan v5: name record_forwarding_disposition_hot/cold explicitly (Codex round-4 #1) + scrub stale 'leak' / 'the leaked one' framing (Codex round-4 #2) + soften cache-line wording to acknowledge unspecified layout (Codex round-4 #3)

* #1187 plan v6: scrub final hot-path leak residue (§4.4 heading + framing); reword section 4.4 from Option A/B to 'out of scope'; update §10 question 3 to reflect softened cache-line wording

* #1187 plan v7: scrub stale Option A at lines 95+194; correct §4.4 framing — coordinator/inject.rs is RPC-driven cold path, not 1Hz status poll

* #1187 Phase 1: extend BatchCounters with disposition + screen_drops counters

Per docs/pr/1187-telemetry-double-buffer/plan.md v7 (PLAN-READY).
Codex rounds 1-7 + Gemini Pro 3.1 rounds 1-2.

Adds 8 new u64 fields to BatchCounters (afxdp/mod.rs:308-336):
screen_drops, policy_denied_packets, route_miss_packets,
neighbor_miss_packets, discard_route_packets, next_table_packets,
local_delivery_packets, exception_packets. Extends flush() with
8 new conditional flush blocks.

Introduces DispositionCounters<'a> enum in afxdp/disposition.rs
with Hot(&mut BatchCounters) / Cold(&BindingLiveState) variants and
per-counter bump methods. Refactors record_disposition and
record_forwarding_disposition to take DispositionCounters instead
of `live: &BindingLiveState`. Hot callers in
poll_descriptor.rs:2071 and 2096 pass
DispositionCounters::Hot(telemetry.counters); cold callers in
coordinator/inject.rs:43 and 59 pass DispositionCounters::Cold(live).

stage_screen_check at poll_stages.rs:227 now takes
counters: &mut BatchCounters in place of binding_live (only used
for screen_drops). Drop verdict bumps counters.screen_drops += 1
through the batch — DDoS resilience: SYN flood is the primary
trigger, unbatched atomics here would cause MESI ping-pong with
the coordinator's status reads under volumetric attack.

Three counters explicitly DEFERRED to a follow-up PR (plan §2):
config_gen_mismatches, fib_gen_mismatches, unsupported_packets —
they fire only during reconcile windows AND are gated by
record_exception()'s mutex/timestamp/string work, so the per-atomic
saving is dominated by other costs. They keep direct fetch_add to
live.

forward_candidate_packets at disposition.rs:161 (now in cold-path
ForwardCandidate arm) is reachable only from coordinator/inject.rs
RPC path, not the worker per-packet hot path. Hot path already
routes through telemetry.counters.forward_candidate_packets at
poll_descriptor.rs:213,1706 (existing). PR keeps the cold-path
direct write — bumping a counter on the coordinator-inject path
that fires at RPC rate has no MESI-thrash concern.

tx_errors batching DROPPED from this PR (Codex round-1 finding):
real fan-out spans 6+ sites including umem/mod.rs, tx/drain.rs,
tx/transmit.rs, cos/queue_service/mod.rs, worker/cos.rs. Also,
BatchCounters is created AFTER the first drain_pending_tx() call
in worker/lifecycle.rs:59, so a TX-only error during early drain
would be silently lost. Needs separate design.

cargo build --release: clean
cargo test --release: 974/974 pass

* #1187 review: add #[inline] to DispositionCounters::bump_*, privatize 8 new BatchCounters fields, fix plan version comment

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/312e8185-0492-4934-ae75-9da522b56a35

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

* #1187 review: fix stale BatchCounters comment, update plan §4.2+§4.5 to match impl, add 3 hot/cold regression tests

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/232c0153-2abb-447d-86ad-668df14a1d44

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: psaab <196946+psaab@users.noreply.github.com>
psaab added a commit that referenced this pull request May 6, 2026
Both reviewers converged on the same architectural corrections:

1. queue_vtime moved from CoSQueueHotState to FlowFairState
   (types/cos.rs:524 documents 'Meaningful only on flow_fair
   queues'; non-flow-fair codepaths early-return without touching
   it).

2. worker_id moved from CoSQueueConfigState to VMinQueueState
   (tied directly to vtime_floor.slots indexing per
   admission.rs:468).

3. flow_hash_seed moved from CoSQueueConfigState to FlowFairState
   (flow-fair-only by use; non-flow-fair queues keep seed=0 today
   and don't read it).

4. CoSQueueTelemetry: explicit drop_counters and owner_profile
   fields (real runtime fields at types/cos.rs:621,636) — no
   '...' ellipsis.

5. V_min scratch counters preserved as u32 per current code
   (types/cos.rs:656); v1 sketch widened to u64 with no
   justification.

6. CRITICAL: no double-boxing. v1 had flow_bucket_items:
   Box<[VecDeque<...>; N]> INSIDE an Option<Box<FlowFairState>>.
   Gemini caught this as a major perf flaw — pointless second
   indirection on every hot-path access with zero memory benefit.
   v2 inlines the array within FlowFairState.

7. Box-deref hoisting: hot-path helpers bind &mut FlowFairState
   exactly once at flow-fair branch entry; no repeated
   .as_mut().unwrap() per field access.

8. Helper-method strategy clarified: pass-through #[inline] only
   for immutable config bits (queue_id, flow_fair, shared_exact,
   transmit_rate_bytes); mutable sub-structs exposed directly to
   avoid partial-borrow checker errors.

9. pop_snapshot_stack moves to FlowFairState; non-flow-fair
   .clear() callers wrap in `if let Some(ff) = ...`.

10. Single PR, not staged. Compiler enforces correctness across
    workspace; staging adds review friction with no incremental
    safety.

11. Order: #1206 lands BEFORE #1207/#1209. Both reviewers agree.

Reviewers (round-1):
  Codex task-mou6u7ou-4h4a6u PLAN-NEEDS-MINOR (5 field-placement +
    nested-box + helper strategy + scratch types)
  Gemini Pro 3 task-mou6vue9-thj49g PLAN-NEEDS-MINOR/MAJOR
    (double-boxing #1.A as 'major design flaw'; queue_vtime
    placement; #1207/#1209 ordering)
psaab added a commit that referenced this pull request May 6, 2026
Both reviewers converged on the same core finding: v1 adapter trait
underspecified for the 4 current variants' divergences. Concrete
fixes:

1. Trait owns descriptor insertion + stamping (Gemini Trait Boundary
   Flaw). Skeleton can't iterate A::Scratch to map XdpDesc or extract
   offsets for stamp_submits — the adapter must.

   New methods:
     fn insert_descriptors(scratch, writer) -> u32
     fn stamp_accepted(scratch, inserted, ts_submit, tx_submit_ns)

2. Rollback unified into single cancel_submission method (Gemini #2)
   — local flow-fair restores TX frames AND queue items;
   prepared flow-fair pushes items back without touching TX frames;
   FIFO drops/releases. cancel_submission takes Option<&mut Queue>
   + free_tx_frames + scratch.

3. drain_to_scratch signature widened with required resources for
   prepared variants (pending_fill_frames, in_flight_prepared_recycles)
   and free_tx_frames for local variants — Codex finding #1.

4. settle_submission stays for queue accounting + partial-rollback
   of inserted prefix; stamping moves OUT of settle into stamp_accepted
   (Codex finding #3).

5. Plan slop fixed (Codex): VecDeque<u64> not Vec<TxFrame>;
   pseudocode return is sent_packets > 0 not !sent_packets == 0.

6. Generic monomorphization confirmed acceptable by both — no dyn
   dispatch risk. Hot-path inlining discipline: zero-sized adapters,
   #[inline] tiny methods, verify with size/cargo bloat if needed.

7. Order serialized: implement #1207 ONLY after #1206 lands on
   master, then rebase. Concurrent work creates avoidable conflicts
   in the same hot-path files.

Reviewers (round-1):
  Codex task-mou6u87v-3y6ih0 PLAN-NEEDS-MAJOR
  Gemini Pro 3 task-mou6vz0k-2bk35b PLAN-NEEDS-MAJOR
psaab added a commit that referenced this pull request May 6, 2026
Both reviewers converged on the same architectural corrections:

1. queue_vtime moved from CoSQueueHotState to FlowFairState
   (types/cos.rs:524 documents 'Meaningful only on flow_fair
   queues'; non-flow-fair codepaths early-return without touching
   it).

2. worker_id moved from CoSQueueConfigState to VMinQueueState
   (tied directly to vtime_floor.slots indexing per
   admission.rs:468).

3. flow_hash_seed moved from CoSQueueConfigState to FlowFairState
   (flow-fair-only by use; non-flow-fair queues keep seed=0 today
   and don't read it).

4. CoSQueueTelemetry: explicit drop_counters and owner_profile
   fields (real runtime fields at types/cos.rs:621,636) — no
   '...' ellipsis.

5. V_min scratch counters preserved as u32 per current code
   (types/cos.rs:656); v1 sketch widened to u64 with no
   justification.

6. CRITICAL: no double-boxing. v1 had flow_bucket_items:
   Box<[VecDeque<...>; N]> INSIDE an Option<Box<FlowFairState>>.
   Gemini caught this as a major perf flaw — pointless second
   indirection on every hot-path access with zero memory benefit.
   v2 inlines the array within FlowFairState.

7. Box-deref hoisting: hot-path helpers bind &mut FlowFairState
   exactly once at flow-fair branch entry; no repeated
   .as_mut().unwrap() per field access.

8. Helper-method strategy clarified: pass-through #[inline] only
   for immutable config bits (queue_id, flow_fair, shared_exact,
   transmit_rate_bytes); mutable sub-structs exposed directly to
   avoid partial-borrow checker errors.

9. pop_snapshot_stack moves to FlowFairState; non-flow-fair
   .clear() callers wrap in `if let Some(ff) = ...`.

10. Single PR, not staged. Compiler enforces correctness across
    workspace; staging adds review friction with no incremental
    safety.

11. Order: #1206 lands BEFORE #1207/#1209. Both reviewers agree.

Reviewers (round-1):
  Codex task-mou6u7ou-4h4a6u PLAN-NEEDS-MINOR (5 field-placement +
    nested-box + helper strategy + scratch types)
  Gemini Pro 3 task-mou6vue9-thj49g PLAN-NEEDS-MINOR/MAJOR
    (double-boxing #1.A as 'major design flaw'; queue_vtime
    placement; #1207/#1209 ordering)
psaab added a commit that referenced this pull request May 7, 2026
…-KILL

Codex (task-moupsqds) PLAN-NEEDS-MAJOR — not KILL on race-safety; v1
issues fixable. v2 applies all 5 fixes:

#1. AFD accounting moves from pop-time to settle-time.
    Pop-time fetch_add over-states served work because TX-failure
    restoration pushes items back. Same pitfall that killed #1215 v1.
    v2 hooks into apply_cos_send_result family in tx_completion.rs
    and increments per-bucket served_bytes only for the inserted
    prefix. No fetch_sub anywhere; rebuild bucket hash from
    settle-time item (cheap, batch-amortized).

#2. Published summary carries window DELTA, not cumulative.
    v1 compared cumulative aggregate_served to one-window fair_share
    -> would mark/drop active buckets forever. v2 adds
    window_served_delta + fair_share_window_bytes (matching units);
    snapshot owner stores prior_aggregate in its own state and
    publishes the delta.

#3. Batch-hoist ArcSwap.load AND ECN-write costs.
    v1 budget claim 3-5ns for ArcSwap.load was wrong; real cost ~30ns
    per load (arc-swap v1.8.2 docs/source). v2 hoists ArcSwap.load
    to drain-batch entry (~30ns / TX_BATCH_SIZE=64 = ~0.5ns
    amortized per pop). Per-pop cost: 2 array reads + 1 conditional
    ECN write = ~10ns marked, ~4ns unmarked.

#4. ECN write fast-path helper.
    v1 said ~5ns; ecn.rs:98 actually parses IPv4 + updates checksum.
    v2 adds cos_item_mark_ce_fast that uses cached parsed metadata
    from xdp_main / forward.rs instead of re-parsing. ~10ns marked
    write vs ~50ns full parse.

#5. Smoothed CSFQ drop probability (vs Gemini's drop-cliff critique).
    v1 binary 'if delta > 2*fair { drop }' creates bursty-drop
    pathology. v2 uses CSFQ formula:
      p_drop = max(0, (delta - fair) / delta)
    scaled to 0-255 for cheap random-byte comparison. Matches the
    classic CSFQ paper.

Gemini (task-mouptde4) PLAN-KILL: cache-line bouncing concern still
on the table for ArcSwap.load (Gemini disputes the hazard-pointer
fast path is contention-free under 12M loads/sec). v2 mitigates by
batch-hoisting (#3) but Gemini may still PLAN-KILL on its position
that 'PR #1217 should be the steady-state product.' That's
acceptable: v2 keeps the research-only framing; #1211 does NOT
block #1217 shipping.

ECN deployment reality (Gemini B): also acceptable — even if <50%
of TCP receivers honor CE marks, the CSFQ probabilistic-drop
fallback (#5) still applies pressure to ECN-stripped flows.

Submit-failure pitfall (Gemini E + Codex #1) FIXED by #1 above —
not a residual concern in v2.
psaab added a commit that referenced this pull request May 7, 2026
…ILL #1)

Round-2 verdicts:
- Codex (task-mousvzy4): PLAN-NEEDS-MAJOR — 4 majors (canonical text
  contradicts v2 fixes; bpf_random_u32 won't compile; ECN metadata
  premise wrong; settle hook location wrong)
- Gemini (task-mousw4d4): PLAN-KILL — RFC 3168 violation (binary
  100% ECN mark vs probabilistic drop) is a fairness cliff that
  starves ECN flows; document inconsistency between v2 fixes section
  and §3.2 hot path; complexity-vs-value (with #1217 as alternative)

v3 fixes:

#6 RFC 3168-compliant ECN marking (Gemini #1 FATAL):
   v2 had `if delta > fair && ECT { mark }` — unconditional 100% mark.
   RFC 3168 mandates same probability for mark vs drop. v3 unifies
   under the smoothed CSFQ probability: same p applies to ECT (marks)
   and non-ECT (drops).

#7 Per-worker non-atomic PRNG (Codex #2):
   bpf_random_u32() doesn't exist in Rust userspace. v3 uses splitmix64
   with per-worker state seeded from existing cos_flow_hash_seed_from_os
   path; no rand crate dep needed.

#8 ECN-mark moves to pre-submit (Codex #3 + design reorg):
   TxRequest doesn't carry L3 offsets, so the 'fast-path' claim was
   wrong. v3 reorganizes: per-pop reads delta/fair (batch-hoisted),
   pre-submit in tx/dispatch.rs decides mark/drop/pass with cached
   L3 offsets, settle increments served_bytes for inserted prefix
   (Fix #1 unchanged).

#9 Correct settle hook reference (Codex #4):
   settle_exact_*_scratch_submission_flow_fair (queue_service/mod.rs
   ~740), not apply_cos_send_result.

§3.2 reconciliation (both reviewers' document inconsistency finding):
   Added explicit NOTE pointing readers to the v2 fixes section as
   the canonical design, and listing each §3.2 element that's
   superseded. The original §3.2 text retained for traceability of
   design evolution but marked clearly as not-the-implementation.

Gemini's PLAN-KILL on complexity-vs-value remains a legitimate
position. v3 doesn't dispute that view; it ensures the technical
flaws Gemini and Codex flagged are fixed so the value/complexity
debate is the only remaining issue. Per user directive: '#1211
stays research-only; PR #1217 ships even if AFD dies'. PLAN-KILL
on complexity-vs-value grounds is acceptable.
psaab added a commit that referenced this pull request May 7, 2026
User directive (2026-05-07): 'Complexity is ok'. Gemini's round-2
and round-3 PLAN-KILL on complexity-vs-value grounds is overridden
by user direction. Technical fixes (Codex's findings) still
required.

Fixes:

1. Settle hook location now consistent throughout the doc.
   v3 had Fix #1 saying 'apply_cos_send_result family in
   tx_completion.rs' (line 116) AND Fix #9 saying
   'settle_exact_*_scratch_submission_flow_fair (queue_service/
   mod.rs ~740)' (line 357), AND the §3.2 NOTE pointing to
   tx_completion.rs (line 408). All three now reference the
   correct symbol path: queue_service/mod.rs:740+795.

2. bpf_random_u32() removed from §3.2 inline drop snippet (line
   385). Replaced with afd_per_worker_random_byte(ff) referring
   to Fix #7's splitmix64 implementation.

3. Fix #10 (NEW): TxRequest metadata prerequisite is explicit.
   Phase 0 extends TxRequest + PreparedTxRequest in
   types/tx.rs:12+54 to carry cached L3+L4 offsets + IP version.
   Phase 1 adds AFD module. Phase 2 wires pre-submit AFD decision
   in tx/cos_classify.rs or tx/dispatch.rs. Phase 3 wires
   settle-time accounting. Fallback to reparse path in
   enqueue_cos_item if Phase 0 too invasive (with documented
   ~50ns budget impact).

Codex round-3 PLAN-NEEDS-MAJOR findings 1-3 all addressed. Plan
is now internally consistent. Ready for implementation-phase
plan-review.
psaab added a commit that referenced this pull request May 7, 2026
Codex round-4 (task-mouwk42c) identified 2 real issues:

1. Phase 0 ECN metadata premise was wrong: ingress L3 offsets
   are unsafe for TX ECN marking because rewrites/VLAN/tunnel
   change the offset. Existing cos/ecn.rs:179 re-parses outgoing
   wire bytes for exactly this reason; frame/mod.rs:245 computes
   output eth_len from tx_vlan_id at rewrite time.

   v5 reformulates Phase 0 with two viable sub-options:
   - 0a (lower invasive): re-parse post-rewrite TX-frame at AFD
     decision site (~50ns). Reuses existing cos/ecn.rs path.
   - 0b (lower runtime cost): extend TxRequest/PreparedTxRequest
     to carry POST-REWRITE offsets computed at frame/mod.rs:245
     rewrite time (~10ns).
   Recommendation: ship 0a first; evaluate 0b empirically.

2. Hook contradictory: Fix #10 said tx/cos_classify.rs or
   tx/dispatch.rs; §4 still said cos/queue_ops/pop.rs.

   v5 §4 now explicitly:
   - REJECTS cos/queue_ops/pop.rs as the hook (pop runs before
     TX rewrite; frame offsets aren't stable yet).
   - Hook lives wherever post-rewrite frame is available
     immediately before TX submit (0a or 0b).
   - settle-time accounting (per Fix #1) stays in
     settle_exact_*_scratch_submission_flow_fair.

Plan is now internally consistent and the rewrite-time offset
issue is correctly framed.
psaab added a commit that referenced this pull request May 7, 2026
…xes)

Round-2 verdicts:
- Gemini (task-mov1moka): PLAN-READY
- Codex (task-mov1mk84): PLAN-NEEDS-MAJOR with 2 NEW blockers

Codex new blocker #1: FlowCacheEntry has no last_used_ns field.
v2's '100ms scan flow_cache for fresh entries' was unimplementable.

v3 Fix #1: add last_used_epoch: u16 to FlowCacheEntry. Owner-only
single u16 store on every lookup() hit (~1-2 ns/lookup). Per-binding
current_epoch atomic incremented at the worker's existing 100ms
tick. Active-flow count = entries with epoch within last 10 ticks
(1s window). Tick-side O(N) scan over 8192 entries every 100ms =
80K loads/sec/worker = ~10us/sec. Wraparound-safe via wrapping_sub
on u16 (256 epochs = 25.6s headroom vs 1s window).

Codex new blocker #2: per-queue + ≥1% throughput qualification not
joinable. iperf3 -P N uses single destination port; RSS decides
binding; no way to map 'stream X had ≥1% throughput' to 'stream X
landed on binding Y'.

v3 Fix #2: distinct source ports per iperf3 stream
(--cport CPORT_BASE+i). Each stream has unique 5-tuple. Harness
reads kernel RSS config (ethtool -x for indirection table; ethtool
-u for n-tuple rules; RSS key from ethtool) and computes Toeplitz
hash for each stream's 5-tuple to deterministically map streams to
RX queues. RX queue → binding via existing BindingStatus.bound_rx_queue
field. ≥1% qualification applied at harness using mapped streams.
Self-test on harness startup: single-stream prediction must match
observed binding TX bytes; fail-fast on mismatch.

Hot-path cost: single u16 store per lookup. Order of magnitude
smaller than v1's HashMap-insert proposal (60 ms/sec/core).

Production observability still deferred to #1220.
psaab added a commit that referenced this pull request May 7, 2026
Codex round-3 (task-mov1wpqo) PLAN-NEEDS-MAJOR with 4 findings:
1. v3 harness sketch ran iperf3 -P N without --cport, contradicting
   the per-stream --cport claim earlier
2. RSS tuple wrong for -R workload (data direction reversed; on-wire
   RX tuple at measured-direction interface, not control tuple;
   plus possible NAT translation)
3. bound_rx_queue field doesn't exist in BindingStatus (real fields:
   QueueID/WorkerID/Interface/Ifindex). No public proto for binding
   status — must specify control-socket vs Manager.Status() vs new gRPC
4. Stale impl details on last_used_epoch

v4 architectural simplification: drop RSS-join entirely. After
verifying the actual codebase via pkg/dataplane/userspace/protocol.go:615:

- Data plane has flow_cache + epoch counter (Fix #v3-1 unchanged)
- 100ms-tick scan publishes per-binding active_flow_count to a
  Prometheus metric: xpf_userspace_binding_active_flow_count{binding_slot=N}
- Harness scrapes /metrics every 1s during the iperf3 run
- {a_i} = per-binding active_flow_count read directly. No RSS hash
  computation, no indirection-table read, no direction-reversal
  complications for -R, no kernel hash key extraction.

This addresses ALL 4 Codex findings:
- #1: dropped per-stream --cport. Single iperf3 -P N -J. Per-stream
  join via start.connected[].socket <-> intervals[].streams[].socket
  (the canonical iperf3 JSON join Codex suggested)
- #2: irrelevant; we don't compute RSS hashes
- #3: harness reads via existing /metrics endpoint. ONE new metric
  added (snapshot, not rolling-window — much narrower than v1's
  production exports deferred to #1220)
- #4: last_used_epoch impl details cleaned up

Trade-offs explicitly acknowledged + documented:
- Per-binding count includes all flows, not just per-CoS-queue.
  Fine for iperf3-only test workloads. Production observability with
  per-queue qualification deferred to #1220.
- No ≥1% throughput qualification at data plane. Conservative effect
  (slightly higher Cstruct than strict contract reading; gate slightly
  more forgiving). Starved-flow gate (Gate 1) catches starved flows
  exactly via iperf3 per-stream throughputs at harness layer.

Net architecture: data plane tracks epoch (1-2 ns/lookup), publishes
binding count on tick (10 us/sec/worker), exposes via /metrics.
Harness uses 1 new Rust binary, 1 new Prometheus metric, 1 new
flow_cache field. Total ~300 LOC vs v1's ~600 LOC.
psaab added a commit that referenced this pull request May 7, 2026
….md cleanup

Codex round-2 (task-mov55tld) MERGE-NEEDS-MAJOR. Round-1 finding #1
(start.connected[] seeding) regressed when I reset to Copilot agent's
87973ff during the rebase conflict — the agent's version of
fairness-eval.rs didn't include the seeding, and my reapplied seeding
got dropped. Re-applied here:

1. Seed per_stream_buckets from start.connected[].socket so streams
   that contributed zero bytes for the entire steady-state window
   appear in the map with empty bucket vec → starved_flow_count
   correctly increments.

2. n_iperf_streams derives from start.connected[].len() when
   non-empty, falling back to test_start.num_streams. This makes the
   harness fail-fast guard actually catch missing-from-intervals
   streams.

3. plan.md cleanup (Codex round-2 finding #2):
   - 8192 entries → 4096 entries (matches FLOW_CACHE_SIZE constant)
   - 100ms tick → ~65ms gate (call-rate dependent)
   - 256 epochs → fixed already in code, plan was already correct

Test verification:
- 2-stream JSON with socket 5 active + socket 7 silent →
  starved_flow_count: 1 (was 0 before fix), verdict: FAIL with
  Gate 1 diagnostic. Codex's round-2 reproducer scenario.
- All 22 fairness pure-fn tests + 22 flow_cache tests + 8 drift CI pass.
- cargo build --release clean (1 warning: dead-code on local_port,
  marked #[allow] earlier).
psaab added a commit that referenced this pull request May 7, 2026
…ount + Prometheus + fairness-eval (#1220)

* #1219 plan v1: fairness harness — Cstruct + distinct-flow-count + Prometheus

Implementation plan for the harness work mandated by the
fairness-regimes contract (PR #1217 e1ec6b9). Three deliverables:

1. Rust pure-fn module (userspace-dp/src/fairness/mod.rs):
   - compute_cstruct(distribution: &[u32]) -> f64
   - compute_observed_cov(per_flow: &[u64]) -> f64
   - starved_flow_count(per_flow_buckets: &[Vec<u64>]) -> u32
   - is_saturated(buckets: &[u64], cap: u64) -> bool
   Unit-tested against the contract's 5-row worked-example table
   (0,0.47,0.20,0.58,0,0).

2. Per-binding distinct-flow-count signal (DistinctFlowTracker
   with bounded LRU set, single-writer record() on flow-cache hit
   path, periodic age_out() on worker tick, atomic snapshot read).
   MAX_TRACKED_FLOWS=1024, FLOW_AGE_OUT_NS=1s.

3. Prometheus exports (4 metrics: cstruct, observed_cov,
   starved_flows, saturated). Re-implemented in Go from same spec
   with shared test-vector CI gate to prevent Rust/Go drift.

Test harness fairness-harness.sh wraps iperf3 -> per-flow buckets
-> Go helper -> contract gates -> PASS/FAIL.

Smoke fixture (deterministic RSS-skew per Codex Path 0) is a
separate follow-up issue; v1 uses iperf3's natural RSS.

7 open questions for adversarial review including the distinct-
flow cap, hash collisions, Rust/Go drift strategy, HA failover
timing of the tracker, steady-state window detection, Prometheus
cadence, harness self-test scope.

PLAN-KILL is acceptable if the harness logic complexity outweighs
the operational value (the immediate goal: answer 'is 47% iperf3
CoV at structural ceiling or scheduler bug').

* v2: massively reduced scope addressing 4 Codex blockers + 3 Gemini fatals

Round-1 verdicts both PLAN-NEEDS-MAJOR with convergent fatals:
- Codex: DistinctFlowTracker locality wrong (Arc + &mut self
  conflict); 6% CPU on hot path; semantic drift (per-binding vs
  per-queue); Go can't compute observed_cov from current status
- Gemini: HashMap on hot path = FATAL; 1024 cap = FATAL high-fan-in
  false-pass; Rust+Go formula drift = FAIL

v2 fundamental rescope:

DROPPED FROM v1:
- Production Prometheus exports (xpf_fairness_*) -> deferred to #1220
- DistinctFlowTracker HashMap entirely
- Per-packet record() on flow-cache lookup
- Rust+Go re-implementation
- Continuous starved-flow counter

KEPT FROM v1:
- Rust pure-fn module (Codex independently verified Cstruct math correct)
- gRPC per-binding active_flow_count field

NEW IN v2:
- Active flow count comes from EXISTING flow_cache via O(N) scan at
  worker's existing 100ms tick (Gemini's remediation). Cost ~10us/sec/
  worker on a tick path that has spare time. NO per-packet write.
- Per-queue active flows + ≥1% throughput qualification: computed
  AT THE HARNESS from iperf3 JSON output, not in the data plane
  (addresses Codex blocker #3).
- Single source of truth = Rust binary 'fairness-eval' that reads
  iperf3-out.json + binding-flows.jsonl and emits {Cstruct,
  observed_cov, regime, verdict}. No Go side at all in v2.
  (Addresses Gemini D / Codex #4)

Net effect: v2 ships ~150 LOC pure-fns + ~30 LOC flow_cache scan +
~20 LOC gRPC plumbing + ~120 LOC eval binary + ~80 LOC bash harness =
total ~400 LOC vs v1's ~600 LOC, with simpler hot path (no new
HashMap), no Prometheus changes, no Rust+Go drift.

Production observability (Prometheus + rolling windows) becomes
issue #1220 to file after v2 lands.

Operational goal preserved: answer 'is 47% iperf3 CoV at structural
ceiling or scheduler bug?' This is the deliverable test in §7.

* v3: add flow_cache epoch + RSS-join harness mapping (Codex round-2 fixes)

Round-2 verdicts:
- Gemini (task-mov1moka): PLAN-READY
- Codex (task-mov1mk84): PLAN-NEEDS-MAJOR with 2 NEW blockers

Codex new blocker #1: FlowCacheEntry has no last_used_ns field.
v2's '100ms scan flow_cache for fresh entries' was unimplementable.

v3 Fix #1: add last_used_epoch: u16 to FlowCacheEntry. Owner-only
single u16 store on every lookup() hit (~1-2 ns/lookup). Per-binding
current_epoch atomic incremented at the worker's existing 100ms
tick. Active-flow count = entries with epoch within last 10 ticks
(1s window). Tick-side O(N) scan over 8192 entries every 100ms =
80K loads/sec/worker = ~10us/sec. Wraparound-safe via wrapping_sub
on u16 (256 epochs = 25.6s headroom vs 1s window).

Codex new blocker #2: per-queue + ≥1% throughput qualification not
joinable. iperf3 -P N uses single destination port; RSS decides
binding; no way to map 'stream X had ≥1% throughput' to 'stream X
landed on binding Y'.

v3 Fix #2: distinct source ports per iperf3 stream
(--cport CPORT_BASE+i). Each stream has unique 5-tuple. Harness
reads kernel RSS config (ethtool -x for indirection table; ethtool
-u for n-tuple rules; RSS key from ethtool) and computes Toeplitz
hash for each stream's 5-tuple to deterministically map streams to
RX queues. RX queue → binding via existing BindingStatus.bound_rx_queue
field. ≥1% qualification applied at harness using mapped streams.
Self-test on harness startup: single-stream prediction must match
observed binding TX bytes; fail-fast on mismatch.

Hot-path cost: single u16 store per lookup. Order of magnitude
smaller than v1's HashMap-insert proposal (60 ms/sec/core).

Production observability still deferred to #1220.

* v4: drop RSS-join entirely; harness reads {a_i} via Prometheus scrape

Codex round-3 (task-mov1wpqo) PLAN-NEEDS-MAJOR with 4 findings:
1. v3 harness sketch ran iperf3 -P N without --cport, contradicting
   the per-stream --cport claim earlier
2. RSS tuple wrong for -R workload (data direction reversed; on-wire
   RX tuple at measured-direction interface, not control tuple;
   plus possible NAT translation)
3. bound_rx_queue field doesn't exist in BindingStatus (real fields:
   QueueID/WorkerID/Interface/Ifindex). No public proto for binding
   status — must specify control-socket vs Manager.Status() vs new gRPC
4. Stale impl details on last_used_epoch

v4 architectural simplification: drop RSS-join entirely. After
verifying the actual codebase via pkg/dataplane/userspace/protocol.go:615:

- Data plane has flow_cache + epoch counter (Fix #v3-1 unchanged)
- 100ms-tick scan publishes per-binding active_flow_count to a
  Prometheus metric: xpf_userspace_binding_active_flow_count{binding_slot=N}
- Harness scrapes /metrics every 1s during the iperf3 run
- {a_i} = per-binding active_flow_count read directly. No RSS hash
  computation, no indirection-table read, no direction-reversal
  complications for -R, no kernel hash key extraction.

This addresses ALL 4 Codex findings:
- #1: dropped per-stream --cport. Single iperf3 -P N -J. Per-stream
  join via start.connected[].socket <-> intervals[].streams[].socket
  (the canonical iperf3 JSON join Codex suggested)
- #2: irrelevant; we don't compute RSS hashes
- #3: harness reads via existing /metrics endpoint. ONE new metric
  added (snapshot, not rolling-window — much narrower than v1's
  production exports deferred to #1220)
- #4: last_used_epoch impl details cleaned up

Trade-offs explicitly acknowledged + documented:
- Per-binding count includes all flows, not just per-CoS-queue.
  Fine for iperf3-only test workloads. Production observability with
  per-queue qualification deferred to #1220.
- No ≥1% throughput qualification at data plane. Conservative effect
  (slightly higher Cstruct than strict contract reading; gate slightly
  more forgiving). Starved-flow gate (Gate 1) catches starved flows
  exactly via iperf3 per-stream throughputs at harness layer.

Net architecture: data plane tracks epoch (1-2 ns/lookup), publishes
binding count on tick (10 us/sec/worker), exposes via /metrics.
Harness uses 1 new Rust binary, 1 new Prometheus metric, 1 new
flow_cache field. Total ~300 LOC vs v1's ~600 LOC.

* v5: address Codex round-4 (4 findings; task-mov2afuw)

1. Stale v3 RSS section deleted entirely (lines 340-403 in v4
   still had the 6-step ethtool/Toeplitz/RSS-join design that
   v4 was supposed to replace). v5 leaves only a one-line
   '§3.4.1 (DELETED) v3 RSS-join steps removed' marker.

2. Metric pipeline path explicit. v4 was internally inconsistent
   between 'harness polls gRPC' (line 175) and 'Prometheus
   unchanged' (line 487) and 'harness scrapes new Prometheus'
   (round-3 resolution). v5 spells out the full plumbing:
   Rust BindingLiveState.active_flow_count (AtomicU32, owner
   100ms tick) -> Rust snapshot BindingStatus.active_flow_count ->
   helper-process control-socket JSON -> Go BindingStatus
   ActiveFlowCount uint32 (pkg/dataplane/userspace/protocol.go:615)
   -> Prometheus emitter in pkg/api/metrics.go:424 with new
   metrics_test.go case.

3. Harness fail-fast guard for {a_i} correctness. Without ≥1%
   throughput qualification at the data plane, over-counting
   could move Cstruct either direction. v5 mandates: during
   steady-state, sum(per_binding_active_flow_count) must match
   non-starved iperf3 stream count within max(2, 0.10 * N)
   tolerance, else harness exits with diagnostic.

4. fairness-eval binary location corrected: was userspace-dp/bin
   (doesn't exist); v5 uses canonical Cargo src/bin/ subdirectory:
   userspace-dp/src/bin/fairness-eval.rs (auto-discovered, no
   [[bin]] entry needed in Cargo.toml).

Section 4 (Public API preservation) updated: Prometheus is no
longer 'unchanged' — adds 1 new metric
xpf_userspace_binding_active_flow_count{binding_slot=N} (snapshot,
not rolling window; rolling-window production exports remain
deferred to #1220).

* v6: purge gRPC/proto references; pin to helper-process JSON only

Codex round-5 (task-mov2ix1o): PLAN-NEEDS-MINOR with single
residual issue — §3.3 still framed active_flow_count as 'published
to gRPC' with a proto/xpf/v1/dataplane.proto example block, even
though v5's intended path is JSON over the helper-process control
socket. §4 'gRPC: 1 new field on per-binding status JSON' had the
same ambiguity.

v6 fixes:

§3.3 'Atomic gauge published to gRPC' renamed to 'Atomic gauge
published via existing JSON status path'. Replaced the proto
example with a Rust BindingStatus snapshot struct that's
serialized into the helper-process control-socket JSON and
decoded by Go BindingStatus (pkg/dataplane/userspace/protocol.go
:615). Explicit note: 'No public gRPC / proto change'.

§4 Public API: 'gRPC: 1 new field on per-binding status JSON'
replaced with 'gRPC / public proto: no change' + separate bullet
clarifying the new field is on the internal helper-process
status JSON, backward-compatible via json:omitempty.

All other Codex round-4 findings remain addressed. v6 is a
single-edit cleanup; expecting PLAN-READY.

* v7: purge 6 remaining stale gRPC refs (Codex round-6 minor)

Codex round-6 (task-mov2qck7) PLAN-NEEDS-MINOR. v6 fixed §3.3 + §4
correctly but 6 stale 'gRPC' references remained scattered:

- line 175-178: 'BindingStatus.active_flow_count copied to gRPC status'
  -> 'copied to the helper-process status JSON'
- line 240: snapshot doc-comment 'reads via the gRPC status surface'
  -> 'reads via the helper-process status JSON surface'
- line 524: test plan 'verify gRPC field is populated'
  -> 'verify the new active_flow_count field is populated in the
     helper-process status JSON (and surfaced through Manager.Status())'
- line 563: open question 'gRPC poll cadence'
  -> 'Prometheus scrape cadence'
- line 578: HA-failover detection 'gRPC reports role_change_at'
  -> 'status JSON reports role_change_at'
- line 583: verdict request 'gRPC field' -> 'helper-process status
  JSON field'

Remaining 'gRPC' mentions are all legitimate: header + the §3.3 +
§4 'No public gRPC / proto change' negations + line 23/48 'not via
a new gRPC RPC'.

Plan should now be internally consistent on the helper-process JSON
path. Expecting PLAN-READY.

* #1219 part 1/N: Rust fairness pure-fns module

Pure-fn module for the fairness contract gate computations per
docs/fairness-regimes.md (merged via PR #1217 e1ec6b9).

Functions:
- compute_cstruct(distribution: &[u32]) -> f64
  Population CoV across the per-flow share multiset
  {1/a_i : repeated a_i times for active workers}. Idle workers
  excluded.
- compute_observed_cov(per_flow_throughputs: &[u64]) -> f64
- starved_flow_count(per_flow_buckets: &[Vec<u64>]) -> u32
  Counts flows below 1% of mean per-cell throughput for the
  ENTIRE window (transient dips don't count).
- is_saturated(aggregate_buckets: &[u64], cap: u64) -> bool
  >= 95% of cap for >= 80% of buckets.

Tests pin all 5 worked-example Cstruct values from the contract:
- {2,2,2,2,2,2}: 0.00
- {1,1,2,2,3,3}: 0.47
- {0,2,2,2,3,3}: 0.20
- {1,3,0,0,0,0}: 0.58
- {6,0,0,0,0,6}: 0.00

22/22 tests pass. cargo build clean. Zero hot-path impact (the
module is invoked only by the harness binary, not by the data
plane).

Next parts: flow_cache last_used_epoch counter (part 2);
BindingLiveState + 100ms tick scan (part 3); status JSON
extension + Go decoder (part 4); Prometheus metric (part 5);
fairness-eval binary (part 6); harness script (part 7).

Plan PLAN-READY through 7 review rounds; Codex final round-7
(task-mov2x81x) no findings.

* #1219 part 2/N: flow_cache last_used_epoch + tick_advance_epoch + count_active_flows

Per plan §3.2 (Fix #v3-1): add a per-binding epoch counter to
FlowCache and a u16 last_used_epoch on FlowCacheEntry written
on every successful lookup() hit.

Per-hit cost: single u16 store on a struct already in cache
from the key check. ~1-2 ns/lookup vs the v1 HashMap-insert
proposal at ~30 ns.

Periodic count_active_flows scan iterates entries in O(N) =
8192 cells, comparing wrapping_sub(current_epoch, last_used_epoch)
< 10 (1s window at 100ms tick cadence). Wraparound-safe via u16
wrapping arithmetic; window << 25.6s wraparound period.

Worker tick will call tick_advance_epoch() at 100ms cadence
(part 3 will wire this into the per-binding live state +
status snapshot path).

Tests added (5 new):
- count_active_flows_starts_at_zero
- count_active_flows_excludes_never_touched_entries
- count_active_flows_marks_recently_hit
- count_active_flows_ages_out_after_window
- count_active_flows_handles_epoch_wraparound

cargo test 999/999 + 32 flow_cache (was 27). cargo build clean.
No existing tests break.

* #1219 part 3/N: wire active_flow_count into BindingLiveState + ~65ms tick

Hooks the flow_cache epoch counter (part 2) into the worker's
existing periodic tick at update_binding_debug_state (umem/mod.rs:986).

BindingLiveState gains:
- active_flow_count: AtomicU32 (initialized to 0)

update_binding_debug_state now:
- calls binding.flow.flow_cache.tick_advance_epoch() per tick
- stores binding.flow.flow_cache.count_active_flows() into
  binding.live.active_flow_count via Relaxed atomic store

Tick cadence note: the actual cadence is ~65ms (driven by the
0xFFFF debug-state counter at ~1M calls/sec / 65536 = ~15 Hz).
The plan's 100ms target is approximate; with ACTIVE_WINDOW_EPOCHS=10
the active-flow window is ~650ms, comfortably under the 1s target
and well clear of u16 wraparound (~16.6s).

cargo test 1004/1004 (was 999). cargo build clean. Hot-path
unaffected — no per-packet write change beyond part 2's u16 store.

* #1219 part 4/N: BindingStatus snapshot + JSON pipeline

Wires the BindingLiveState.active_flow_count atomic (part 3) through
the snapshot path so the helper-process control-socket JSON carries
the field for the Go decoder + Prometheus emitter (part 5).

Changes:
- BindingDebugSnapshot (worker/mod.rs:1778): add active_flow_count: u32
- BindingStatus (protocol.rs:1149): add #[serde] active_flow_count: u32
- BindingCountersSnapshot (protocol.rs:1502): add active_flow_count: u32
  + propagate via b.active_flow_count in From impl (protocol.rs:1656)
- umem/mod.rs:672 snapshot construction: load active_flow_count.load(Relaxed)
- main_tests.rs: 3 BindingCountersSnapshot test sites add field

cargo test 1004/1004 + 8/8 drift CI clean.

* #1219 part 5/N: Go BindingStatus.ActiveFlowCount + Prometheus emitter

Plumbs the helper's active_flow_count snapshot through the Go
manager and emits it as a new Prometheus gauge.

Changes:
- pkg/dataplane/userspace/protocol.go: add ActiveFlowCount uint32
  on BindingStatus + on the lean per-binding snapshot struct
  (`active_flow_count` JSON, omitempty for forward-compat)
- pkg/api/metrics.go:
  - new xpfCollector.bindingActiveFlowCount Desc
  - new prometheus.NewDesc("xpf_userspace_binding_active_flow_count",
    labels: binding_slot, queue_id, worker_id, iface)
  - emitBindingActiveFlowCount() walks status.Bindings, emits
    one GaugeValue per binding
  - hooked into collectUserspaceStatus()
  - registered in Describe()

go build ./... clean. go test ./pkg/api/ ./pkg/dataplane/ clean.

Net Rust → Go → Prometheus pipeline now complete:
flow_cache.last_used_epoch (per-hit u16 store) → BindingLiveState
.active_flow_count (~65ms tick scan) → BindingDebugSnapshot
→ helper-process status JSON → Go BindingStatus.ActiveFlowCount
→ Prometheus xpf_userspace_binding_active_flow_count gauge.

Per the plan: NO public gRPC change. The new field is on the
internal helper-process control-socket JSON only.

* #1219 part 6/N: fairness-eval Rust binary

Test harness binary at userspace-dp/src/bin/fairness-eval.rs (Cargo's
auto-discovered src/bin/ subdirectory; no [[bin]] entry needed).

Reads:
- iperf3 -P N -J --forceflush JSON output
- binding-flows.tsv (timestamp, binding_slot, count) — produced by
  the harness scraping /metrics for xpf_userspace_binding_active_flow_count

Computes per docs/fairness-regimes.md:
- {a_i}: median per-binding count over the steady-state window
- Cstruct: structural CoV ceiling from {a_i}
- observed_CoV: sample CoV across per-stream window-mean throughputs
- starved_flow_count: streams below 1% of mean for entire window
- saturated: aggregate vs (Na/Nv) × shaper_rate
- gap = observed - Cstruct; PASS iff gap <= epsilon (0.05) AND
  starved_flow_count == 0 AND harness fail-fast guard holds

Harness fail-fast guard (Codex round-4 finding #3):
sum(a_i) ≈ non-starved iperf stream count within max(2, 10% × N).
Disagreement flags background-flow pollution / RSS unexpected
behavior; harness should not report a verdict from inconsistent
inputs.

Pure-fns shared with the main binary via #[path = '../fairness.rs'] mod
so the math is single-source-of-truth (Rust only; no Go re-impl).

Output: JSON verdict {distribution_a_i, n_active, cstruct,
observed_cov, gap, epsilon, saturated, aggregate_mbps,
starved_flow_count, a_i_sum_check_ok, verdict, failure_reasons}.
Exit code 0 on PASS, 1 on FAIL, 2 on parse/IO error.

Smoke-tested with synthetic data: 2-stream {1Gbps, 500Mbps} test
correctly reports CoV=0.33, gap=0.33, FAIL with Gate 2 diagnostic.

Build: cargo build --release --bin fairness-eval clean

* #1219 part 7/N: fairness-harness.sh test driver script

Bash wrapper at test/incus/fairness-harness.sh that:

1. Runs iperf3 -P N -J --forceflush against $TARGET on $PORT for $T s
2. Concurrently scrapes /metrics every 1s, extracts
   xpf_userspace_binding_active_flow_count{binding_slot=N} into a TSV
3. Calls fairness-eval Rust binary on both inputs
4. Returns the binary's exit code (0 PASS, 1 FAIL, 2 parse error)

Defaults match the iperf-c P=12 -R workload that produced today's 47%
per-flow CoV measurement — the immediate operational target.

Configurable via positional args + env (METRICS_URL, N_WORKERS,
SHAPER_RATE_BPS, WARMUP, FINAL_BURST, FAIRNESS_EVAL path).

awk extracts the metric using the standard Prometheus exposition
format (single value at end of line; binding_slot label parsed via
regex). Per-second cadence over the steady-state window gives 60+
samples per binding for stable {a_i} median.

bash -n syntax-clean.

Parts 1-7 SHIPPED. Implementation pipeline complete:
- Rust pure-fns (part 1) ✓
- flow_cache last_used_epoch (part 2) ✓
- BindingLiveState + ~65ms tick (part 3) ✓
- Snapshot → JSON pipeline (part 4) ✓
- Go BindingStatus + Prometheus emitter (part 5) ✓
- fairness-eval binary (part 6) ✓
- harness script (part 7) ✓

Next: smoke matrix on loss userspace cluster + open PR + dispatch
triple-review of code.

* #1219 part 4 fix: bridge active_flow_count through coordinator::refresh_bindings

Smoke-test discovery: even though parts 1-7 shipped, the metric
xpf_userspace_binding_active_flow_count stayed at 0 during iperf3
load. Tracing the snapshot pipeline revealed that
coordinator::refresh_bindings (afxdp/coordinator/mod.rs:1130) is
where BindingLiveSnapshot fields are copied into BindingStatus
(the wire-visible JSON struct). I missed adding active_flow_count
to that copy block.

Without this bridge, BindingLiveState.active_flow_count atomic was
correctly being updated by the worker tick (visible via the snapshot
fn at umem/mod.rs:669), but the value was never landing on the
BindingStatus that gets serialized to the helper-process status JSON
(and thus never reached the Go side / Prometheus emitter).

One-line fix: copy snap.active_flow_count -> binding.active_flow_count
alongside the existing flow_cache_collision_evictions copy.

cargo test 1004/1004 + 22 fairness + 8 drift CI clean.

Pipeline now end-to-end:
flow_cache.last_used_epoch (per-hit u16 store)
  -> count_active_flows on 100ms tick
  -> BindingLiveState.active_flow_count atomic
  -> BindingLiveSnapshot.active_flow_count
  -> coordinator::refresh_bindings copies it into BindingStatus  ← THIS WAS MISSING
  -> Helper-process status JSON
  -> Go BindingStatus.ActiveFlowCount
  -> Prometheus xpf_userspace_binding_active_flow_count gauge.

Re-deploying to verify metric now reports non-zero during load.

* #1219 part 7 fix: bidirectional flow_cache entries + portable awk

End-to-end smoke discoveries:

1. mawk on the cluster doesn't support gawk's 3-arg match($0, re, arr).
   The harness scrape_metrics function fell back to no extraction,
   producing an empty binding-flows.tsv. Replaced with portable
   sed -nE pattern that works on both gawk and mawk.

2. Each TCP flow creates flow_cache entries on BOTH ingress AND
   egress bindings (forward flow + reverse flow). With 12 streams,
   sum(a_i) is ~24 not ~12. Updated the harness fail-fast guard:
     expected_sum = 2 × n_non_starved
     tolerance = max(2, 0.10 × expected_sum)

OPERATIONAL ANSWER (the deliverable that motivated the harness):

End-to-end run on the loss userspace cluster, 12 stream iperf3
P=12 t=30 -R against 172.16.80.200:5201:

  distribution_a_i = [2,7,0,2,2,0,0,2,0,3,0,0,1,2,0,4,1,0]
  n_active        = 10 / 18
  cstruct         = 59.9%   (structural CoV ceiling for this RSS distribution)
  observed_cov    = 51.4%
  gap             = -8.45pp (observed BELOW ceiling)
  saturated       = true (22.3 Gbps)
  starved_flow_count = 0
  verdict         = PASS

This is the answer to the user's mandate question: 'is 47% iperf3 CoV
at structural ceiling or scheduler bug?'. The 47-51% per-flow CoV is
8.45pp BELOW the structural ceiling for the observed RSS distribution.
The per-worker scheduler is doing better than the RSS placement
allows. There is no scheduler bug. The remaining variance is pure
RSS placement skew.

The fairness contract gates that PR #1217 codified are now empirically
verifiable on the test bench. The user's 'flow evenness' work is
either:
- Done (gate passes; current implementation is at-or-below the
  structural ceiling)
- Or requires moving to the killed-architectural-paths
  (#937 ingress XDP_REDIRECT — kernel-blocked; #1211 AFD ECN
  overlay — research-only)

Per the fairness-regimes contract, this measurement result LANDS the
contract: gate satisfied for this run.

* Remove fairness-eval binary committed by accident; built artifact, not source

* review fixes: epoch-0 sentinel, hot-path double-borrow, slow-path inline, parse_args panic

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/0bcf2f3a-3a88-4a11-a42c-0ec3d48666b0

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

* review nits: clarify BUG message + assertion text

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/0bcf2f3a-3a88-4a11-a42c-0ec3d48666b0

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

* test: add TestEmitBindingActiveFlowCount_LabelsAndValue (Codex round-1 finding #2)

Pins the Prometheus emitter's wire shape per Codex round-1 review:
3-binding fixture → 3 metrics, slot=0 ActiveFlowCount=5 → gauge value 5
with correct labels {binding_slot=0, queue_id=0, worker_id=0,
iface=ge-0-0-1}. Mirrors the existing emitWorkerRuntime test pattern.

Atop Copilot SWE Agent's autonomous review-fix commits (87973ff +
1526ff5) which addressed: epoch-0 sentinel, hot-path double-borrow,
slow-path inline, parse_args panic, BUG message clarity. My local
161a3435 'round-1 review fixes' commit got 2-way conflict on
flow_cache.rs + fairness-eval.rs vs the agent's edits — agent's
versions were equivalent on the substantive fixes (epoch-0, panic
exits, etc) plus added the double-borrow + slow-path-inline fixes
I missed. Reset local to the agent's tip and re-applied only the
Go metric test (which the agent didn't add).

* #1219 round-2 review fixes: re-apply start.connected[] seeding + plan.md cleanup

Codex round-2 (task-mov55tld) MERGE-NEEDS-MAJOR. Round-1 finding #1
(start.connected[] seeding) regressed when I reset to Copilot agent's
87973ff during the rebase conflict — the agent's version of
fairness-eval.rs didn't include the seeding, and my reapplied seeding
got dropped. Re-applied here:

1. Seed per_stream_buckets from start.connected[].socket so streams
   that contributed zero bytes for the entire steady-state window
   appear in the map with empty bucket vec → starved_flow_count
   correctly increments.

2. n_iperf_streams derives from start.connected[].len() when
   non-empty, falling back to test_start.num_streams. This makes the
   harness fail-fast guard actually catch missing-from-intervals
   streams.

3. plan.md cleanup (Codex round-2 finding #2):
   - 8192 entries → 4096 entries (matches FLOW_CACHE_SIZE constant)
   - 100ms tick → ~65ms gate (call-rate dependent)
   - 256 epochs → fixed already in code, plan was already correct

Test verification:
- 2-stream JSON with socket 5 active + socket 7 silent →
  starved_flow_count: 1 (was 0 before fix), verdict: FAIL with
  Gate 1 diagnostic. Codex's round-2 reproducer scenario.
- All 22 fairness pure-fn tests + 22 flow_cache tests + 8 drift CI pass.
- cargo build --release clean (1 warning: dead-code on local_port,
  marked #[allow] earlier).

* #1219 round-3 review fixes: per-WORKER {a_i} aggregation + Copilot test additions

Codex round-3 (task-mov5ervt) MERGE-NEEDS-MAJOR + Gemini round-1
(task-mov4lrtd) MERGE-NEEDS-MAJOR convergent fatal: harness was
building {a_i} by binding_slot across ALL interfaces (3 ifaces × 6
queues = 18 bindings), producing fictitious 18-element distribution
instead of the contract's per-worker count for the test's
data-direction interface.

v3 fix:

1. fairness-harness.sh scrape_metrics now writes 6 columns
   (timestamp, binding_slot, queue_id, worker_id, iface, count)
   so fairness-eval can filter and aggregate properly.

2. fairness-eval --iface arg added; backward-compat parser handles
   both new 6-column TSV and legacy 3-column.

3. fairness-eval {a_i} computation rewritten:
   - filter rows to --iface (test's data-direction)
   - aggregate by (timestamp, worker_id) summing counts
   - take median per-worker over the steady-state window
   - distribution_a_i is now per-worker, not per-binding-slot

4. Default IFACE=ge-0-0-2 in harness script (loss cluster's data
   direction; configurable via env).

Plus Copilot 5 comments addressed:

5. Test pinning active_flow_count projection through
   BindingCountersSnapshot::From: assert snap.active_flow_count == 71
   added to the existing projection test.

6. Wire-key assertion now includes 'active_flow_count' in the
   binding_counters_snapshot_serializes_with_expected_wire_keys
   test; fixture value changed from 0 → 31 so omitempty doesn't
   skip the key.

7. local_port: u32 marked #[allow(dead_code)] with rationale.

Test verification:
- 6-col TSV with balanced 6-worker distribution → {a_i} =
  [2,2,2,2,2,2], Cstruct=0.00 (correct).
- cargo test 1006/1006 + 22 fairness + 8 drift CI clean.
- cargo build --release clean (1 expected dead-code warning on
  dropped local_port path).

* review fixes (round 3): fix 100ms doc comments, u16 wraparound math, read_to_string panic, mod fairness cfg(test)

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/a70b49a8-2bcc-408d-9839-19d6b7ba6ff2

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

* Codex round-4 (task-mov5x7j7) MERGE-NEEDS-MAJOR — caught a real bug
in the round-3 fix:

1. **Major: fail-fast guard was still 2× n_streams.** With the
   --iface filter introduced in round-3, sum(a_i) is one direction
   only. P=12 should expect ~12, not ~24 — round-3's untouched
   guard would reject correct single-iface data. Fixed: select
   multiplier based on whether iface filtering is in effect.

2. **Minor: silent zeros on legacy-input + --iface combo.** A
   legacy 3-col TSV (iface="") combined with a fresh harness's
   --iface flag silently dropped every row → empty {a_i} → bogus
   PASS. Fixed: detect "filter set but no iface label in any
   row" and warn loudly + treat --iface as unset for that input.

3. **Minor: stale 100ms / 8192-entry prose.** flow_cache.rs:359
   doc-comment and plan.md:291 still claimed the old 100ms tick
   and ~8192-entry cap — now ~65ms (umem 0xFFFF gate) and 4096
   throughout.

4. **Real test gap (Codex check #6):** added two unit tests in
   src/bin/fairness-eval.rs:
   - six_col_multi_iface_per_worker_aggregation: 2 ts × 3 ifaces
     × 6 workers, verify --iface filter collapses to per-worker
     sums on the filtered iface only and noise on other ifaces
     is dropped.
   - three_col_legacy_parses_with_empty_iface: pin legacy parser
     produces iface="" and worker_id=binding_slot.

Test verification:
- cargo test --release --bin fairness-eval: 24/24 pass
  (was 22 fairness pure-fns; +2 new tsv_tests).
- Full cargo test --release: 1006/1006 + 24 + 8 drift CI clean.

* #1219 round-5 review fixes: extract aggregate_per_worker helper + clean up stale prose

Codex round-5 (task-mov69swk) MERGE-NEEDS-MINOR. Two findings:

1. **Tests didn't cover the production fix.** Round-4's added tests
   only exercised the parser/filter shape — not iface_filter_active,
   per_ts_worker grouping, or direction_multiplier. A reverted
   `direction_multiplier = 2` would not have failed any test.

   Round-5 fix: extract two helpers that ARE the production fix:
   - `aggregate_per_worker(rows, iface_arg, n_workers, warmup, final)`
     → returns AggregateResult { distribution_a_i, iface_filter_active }.
   - `direction_multiplier(iface_filter_active: bool) -> u32`.

   main() now calls both. New aggregation_tests module exercises:
   - filter_iface_and_groups_by_worker (noise on other iface MUST NOT
     contaminate filtered distribution)
   - sums_multiple_queues_per_worker (BTreeMap<(ts, worker_id), u32>
     accumulator must SUM across queues, not replace)
   - legacy_3col_disables_filter (legacy parser produces iface=""
     even with --iface set; filter must collapse to inactive)
   - missing_workers_default_to_zero
   - median_smooths_jitter (single outlier on either side is filtered)
   - direction_multiplier_iface_filter_active_is_one (this would have
     caught the round-3 bug Codex round-4 found)
   - direction_multiplier_no_iface_filter_is_two (legacy bidirectional
     fall-through preserved)

   These tests directly fail under any of:
     - per-binding grouping in place of per-worker;
     - reverted direction_multiplier;
     - filter applied to legacy iface="" rows;
     - sum-then-median replaced by raw count.

2. **Stale 100ms / 8192 prose** at plan.md:34, 169, 235, 270, 279,
   364, umem/mod.rs:235, flow_cache.rs:497. All mechanically replaced
   with ~65 ms (umem 0xFFFF gate) and 4096 entries / ~650 ms window
   throughout. Final grep confirms zero residual matches.

Test verification:
- cargo test --release --bin fairness-eval: 31/31 pass (was 24; +7 new)
- Full suite: 1006/1006 + 31 + 8 drift CI clean.
- Final grep -rn '100 ms|100ms|~8192|8192 entries|256 epochs|25\.6 s|256 ticks'
  on docs/pr/1219-fairness-harness/, userspace-dp/src/afxdp/, and
  userspace-dp/src/bin/fairness-eval.rs returns zero matches.

* #1219 round-6 review fixes: metrics.go HELP text + plan.md "1s" → "~650 ms"

Codex round-6 (task-mov6lp4k) MERGE-NEEDS-MINOR. Two findings, both
addressed:

1. **pkg/api/metrics.go:362 stale Prometheus HELP text.** The
   xpf_userspace_binding_active_flow_count gauge was documented as
   "active in the last ~1s" but the ACTIVE_WINDOW_EPOCHS=10 ×
   ~65ms tick = ~650 ms. User-visible metric documentation; fixed
   to match the implementation.

2. **plan.md:95 / :281 / :366 still said "1 second window"** /
   "1s" / "1 second" — three more sites missed in the round-5
   sweep. Now consistent with the rest of the doc at ~650 ms.

   Note: plan.md:36 ("Harness scrapes /metrics every 1 second")
   and :466 ("last 1s" final-burst) and umem/mod.rs:384/:409
   ("~1s debug-report tick") are about UNRELATED 1-second
   intervals (Prometheus scrape cadence, iperf3 final-burst
   window, and the existing debug-report tick) — they are
   correct as-is and were intentionally left untouched.

Gemini round-3 (task-mov6mk28) verdict: **MERGE-READY**. Helper
extraction parity, test coverage of subtle bugs (worker_id !=
binding_slot, multi-queue summation), direction_multiplier code
paths, Cstruct math regression, plan doc consistency, worker_id
stability, hot-path impact, architectural completeness, JSON wire
schema, and arg validation all PASS. "You are good to merge."

Test verification:
- go test ./pkg/api/...: pass.
- cargo test --release: 1006 + 31 + 8 unchanged.

* review fixes (round 5): min steady-state guard, worker_id validation, metrics 1s→650ms, omitempty comment, plan.md stale text, harness label-order comment

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/93fd3daa-a3d2-472e-843c-8413351b77bd

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

* review fixes (round 5+): min steady-state guard, worker_id Result, omitempty comment, harness label-order comment, plan.md stale text

Agent-Logs-Url: https://github.com/psaab/xpf/sessions/93fd3daa-a3d2-472e-843c-8413351b77bd

Co-authored-by: psaab <196946+psaab@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: psaab <196946+psaab@users.noreply.github.com>
psaab added a commit that referenced this pull request May 7, 2026
…_bytes-based rate

Both round-2 reviewers converged on the same fix; v3 adopts both:

Codex round-2 (task-mow38bom, PLAN-NEEDS-MAJOR) — 5 findings:
1. active_flow_count is binding-wide, not per (egress_ifindex, queue_id).
2. Naive cap-check on MQFQ front HOL-blocks; need eligible-bucket scan.
3. observed_bps NOT existing flow_cache state.
4. SharedCoSQueueLease doesn't auto-free tokens for nonempty queues.
5. class_rate concrete source needs definition (exact vs surplus phase).

Gemini round-2 (task-mow3904l, PLAN-KILL) — narrow but valid:
- C. observed_bps via TX completion path crosses worker boundaries
  → would re-introduce v1's contention problem. FATAL for v2.
- B. SharedCoSQueueLease behavior 'elegantly correct' for goal.
- F. PerClassFairnessState belongs in CoSQueueConfigState (Arc'd
  cross-worker), NOT FlowFairState (per-worker Box).
- Concrete v3 alternative: track bytes natively via existing
  flow_bucket_bytes [u64; COS_FLOW_FAIR_BUCKETS] in FlowFairState
  (verified at types/cos.rs:563).

User calibration ('gemini can be wrong a lot') applied: this PLAN-KILL
is the substantive case — narrow point with code-cited grounds and
a concrete fix. Adopted, not capitulated.

v3 design:

1. PerClassFairnessState in CoSQueueConfigState (per egress_ifindex,
   queue_id) — Codex #1, Gemini F.
2. Per-bucket rate via flow_bucket_bytes diff + 10ms local timestamp.
   No flow_cache touch. No cross-worker write — Gemini C fix.
3. Cap-aware MQFQ selector scans active buckets, skips over-cap,
   falls back to lowest-finish if all capped — Codex #2.
4. Per-queue active_flow_count via extension of count_active_flows scan
   (Option A: single extra pass on owner-only path) — Codex #1.
5. class_rate concrete: exact-phase = transmit_rate / N_active_workers;
   surplus-phase = root_shaping_rate × local_share — Codex #5.
6. Conditional shared_lease.release_local_tokens() when locally-capped
   for N consecutive batches — Codex #4.

Acceptance unchanged: per-flow CoV ≤ Cstruct + 0.10 on the user's exact
command, no aggregate regression > 5%.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 7, 2026
Both round-3 reviewers caught the same fatal flaw, code-cited:
flow_bucket_bytes is decremented on dequeue at
cos/queue_ops/accounting.rs:109 (backlog gauge, not monotonic).
v3's diff math would underflow → deterministic crash.

Codex round-3 (task-mow3ndit, PLAN-NEEDS-MAJOR) — 6 findings:
1. flow_bucket_bytes underflow (also Gemini PLAN-KILL).
2. Cap-aware selector wired into both cos_queue_front AND
   cos_queue_pop_front (queue_ops/mod.rs:109 + pop.rs:58).
3. COS_FLOW_FAIR_BUCKETS = 4096 (verified types/cos.rs:105),
   not 32-64. v3's all-bucket scan unacceptable.
4. CoSQueueConfigState is per-worker local (built per-worker
   at builders.rs:96), not Arc-shared. Need to plumb through
   coordinator + WorkerCoSQueueFastPath like queue leases.
5. No release_local_tokens() helper exists.
6. No local_share() / n_active_workers() lease methods.

Gemini round-3 (task-mow3nz4o, PLAN-KILL):
- A,B,D,E: agreed v3 is on right architectural path.
- C: 10ms periodic sample creates burst-stall sawtooth (real).
- F: PLAN-KILL on flow_bucket_bytes underflow (convergent with
  Codex finding #1).

Both reviewers' grounds verified against codebase. v4 fixes:

1. NEW monotonic flow_bucket_tx_bytes [u64; COS_FLOW_FAIR_BUCKETS]
   on FlowFairState. Updated only at dequeue (TX commit). Never
   decremented. Fixes the underflow.

2. Event-driven EWMA (α=1/8) updates flow_bucket_observed_bps on
   every dequeue. Eliminates 10ms-tick sawtooth. ~12 KB burst
   bound post-backoff.

3. Cap-aware selector reuses existing active-ring scan pattern
   from cos_queue_min_finish_bucket (queue_ops/mod.rs:82).
   O(active_ring) ≈ 2-16 entries, not O(4096).

4. cos_queue_min_eligible_bucket returns (eligible, fallback) so
   both selection sites stay aligned. Fall back to lowest-finish-
   time if all buckets over-cap.

5. Arc<PerClassFairnessState> plumbed through coordinator/
   cos_state.rs and worker/cos.rs alongside queue_leases (same
   pattern as #915 V_min floors).

6. count_active_flows_per_queue extension uses existing
   flow_cache descriptor.egress_ifindex + tx_selection.queue_id.

7. target_rate_bps concrete: exact-phase = transmit_rate /
   total_flows; surplus-phase = root × surplus_share / total_flows.

8. release_local_tokens_when_capped extends release_unused on
   cap-stuck-for-N-batches threshold.

User mandate ('gemini can be wrong a lot') applied: v3's PLAN-KILL
was substantive AND code-cited, with concrete v4 path. Adopted,
not capitulated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
psaab added a commit that referenced this pull request May 8, 2026
…t parity

Codex MERGE-NEEDS-MAJOR (task-moxb516r-4c2nxg) on PR #1230 v8
implementation. Spine sound; one HIGH bug + two MEDIUM doc fixes.

HIGH (Codex finding #1) — rehydration clobber:
rehydrate_worker_active_count used store(count) but multiple
runtimes can share a worker thread + lease (multi-binding
same-worker case). The second install would overwrite the
first runtime's count, eventually allowing decrements to drive
the slot to 0 while the other runtime still has active flows.

Fix: change store -> fetch_add. Per-runtime install + Arc::ptr_eq
detection (token_bucket.rs::ensure_v8_lease_attached) guarantees
each runtime fires rehydration exactly once per (runtime, lease-Arc)
pair, so the additive sum across runtimes is correct.

Also: skip rehydrate when count == 0 (saves a no-op atomic).

Tests: rename v8_rehydrate_worker_active_count_idempotent to
v8_rehydrate_worker_active_count_is_additive; add new
v8_rehydrate_multiple_workers_isolated +
v8_rehydrate_multi_binding_same_worker_summation tests proving
the Codex finding's failure mode is fixed.

MEDIUM (Codex finding #2 + Copilot) — misleading helper invariant:
active_buckets.rs documented helpers as MUST-call but they're
dead code; production sites inline. Update doc to reflect
inlined reality + clarify helpers are reference-only for
future call sites without an existing as_mut borrow.

Plus: add debug_assert in inlined unbump path (accounting.rs:84
dequeue site) to match dead-code helper's assertion. Codex
note #2 verbatim. Catches saturating_sub-past-zero in dev
builds.

Doc fixes:
- flow_bucket_tx_bytes wrap-time math: was '~5e9 seconds at 100
  Gbps' (wrong); is '~47 years at 100 Gbps sustained' (correct,
  u64 wraps at 2^64 bytes / 12.5 GB/s).
- flow_bucket_observed_bps unit: was 'bytes/sec' (wrong); is
  'BITS/sec' matching account_flow_bucket_tx's bytes × 8 × 1e9
  / dt_ns formula. Aligns with name suffix (_bps = bits-per-sec).
- flow_bucket_tx_bytes update sites: now lists all 4 paths,
  not just the post-settle 2.

Tests: 1032 + 47 = 1079 cargo test --release pass (was 1077).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant