Planner proposal: Investigate Hardware Sentinel fail

[secup]

Planner generated this proposal from local file:

agents/planner/proposals/002-20260430_175558-investigate-hardware-sentinel.md

Approval commands:

• /approve codex: queue for Codex
• /approve claude: queue for Claude
• /hold: keep open but stop approval polling
• /reject: reject this proposal

Only usernames listed in AGENT_APPROVERS on the agentic laptop are honored.
The approval watcher only writes a local ignored task file; it cannot merge PRs or push to main.

Proposal

Investigate Hardware Sentinel fail

Goal

Classify the latest hardware sentinel fail and decide whether it is a real
modem regression, hardware/audio setup problem, or transient run noise.

Project goal: hardware reality and channel robustness from
docs/PROJECT_GOALS.md.

Context

• Latest sentinel summary: agents/reports/hardware_sentinel_20260430_173626/summary.txt
• Planner report: agents/planner/reports/20260430_175558/report.md

Allowed Files

• No code changes for initial classification.
• If a code defect is proven, write a separate bounded repair task.

Out Of Scope

• Do not change calibration constants from a single failed run.
• Do not weaken ARQ/LDPC/sync invariants to make the symptom disappear.

Required Local Gate

Not required for initial classification.

Required Hardware Or Benchmark Gate

Rerun once to confirm:

SSH_KEY="$HOME/.ssh/id_pi5" ./agents/run_hardware_sentinel.sh

Reject Conditions

• No comparison between the latest run and at least one prior run.
• No classification of PHY loss, ACK loss, audio clipping/silence, decode backlog, or ARQ timing.

Expected Output

• Confirmed/rejected failure.
• Suspected layer.
• Exact log paths and metrics.
• Next bounded task if code work is justified.

[secup]

/approve codex

[secup]

Queued for codex as agents/queue/codex/017-github-issue-investigate-hardware-sentinel-fail.md after approval by secup. The worker will create a draft PR if it makes tracked changes; report-only tasks will comment their result back here.

[secup]

Agent task completed for codex.

• Task: agents/queue/codex/017-github-issue-investigate-hardware-sentinel-fail.md
• Report: agents/reports/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail
• Branch: agent/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail
• Commits ahead of main: 0
• Draft PR: not created because there are no commits beyond main.
• Hardware gate: ran with rc=0.
• Hardware gate report: agents/reports/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail/hardware_gate.

Agent Log Tail

/tmp/ultra_hw_20260430_165403/A.log:338:[ 35.357][INFO ][MODEM] [ALPHA] SR-ARQ: ACK seq=8 bitmap=0x00000000 (base=8, in_flight=1)
/tmp/ultra_hw_20260430_165403/A.log:348:[ 35.534][INFO ][MODEM] ChannelInjector: process #8 samples=10368 gain=0.700 rms_in=0.2002 peak_in=0.7370 rms_ch=0.2008 peak_ch=0.7438 clip_ch=0 rms_out=0.1406 peak_out=0.5207 clip_out=0
/tmp/ultra_hw_20260430_165403/A.log:371:[ 37.406][INFO ][MODEM] [ALPHA] [ALPHA] Protocol RX: 20 bytes from modem (buffer now 20)
/tmp/ultra_hw_20260430_165403/A.log:442:║                     TEST PASSED                              ║
/tmp/ultra_hw_20260430_165403/A.log:446:  ARQ:  frames_sent=9  frames_rcvd=0  retransmissions=4  timeouts=3  failed=0
/tmp/ultra_hw_20260430_165403/A.log:448:  ACK:  acks_sent=0  acks_rcvd=9  sacks_sent=0  sacks_rcvd=5
/tmp/ultra_hw_20260430_165403/A.log:450:  RX:   frames_decoded=7  frames_failed=0  pings=1  overflows=0  drop_samples=0  resets=0  unsearched=13094  backlog_ms=272.8  peak_backlog_ms=3580.0
/tmp/ultra_hw_20260430_165403/A.log:452:  Rate: frame_success=100.0%
agents/reports/hardware_sentinel_20260430_173626/awgn_1k_r12_snr15.log:13:socket: Operation not permitted

codex
**Results**

Latest sentinel failure is confirmed, but rejected as a modem regression. Both latest cases in [metrics.tsv](/home/ultra-agent/Projects/ProjectUltra-codex/agents/reports/hardware_sentinel_20260430_173626/metrics.tsv:1) failed with `rc=255` before station B started. The wrapper logs show `socket: Operation not permitted` and `ssh: connect to host 192.168.160.112 port 22: failure`, so this is an agent sandbox/network execution failure, not PHY, ACK, ARQ, LDPC, sync, or calibration evidence.

Comparison:
- Prior rerun `20260430_171908` also failed pre-modem, but from DNS: `ssh: Could not resolve hostname pi5tester`.
- Prior valid-ish run `20260430_165403` had AWGN pass with warnings: `retransmissions=4`, `timeouts=3`, `frame_success=100.0`, `peak_backlog_ms=3580.0`.
- Its Good case failed during MC-DPSK connect: ALPHA received PONG and sent CONNECT, but BRAVO never decoded CONNECT; metrics show `frames_sent=0`, `acks_rcvd=0`, `peak_backlog_ms=3470.0`.

Layer classification:
- PHY/sync: latest not observable. Prior Good suggests CONNECT acquisition/sync loss under injected Good fading.
- ACK loss: not primary; latest no traffic, prior AWGN ACK/SACK path worked.
- Audio clipping/silence: latest not observable; no A/B modem logs. Prior runs include one audio device open failure and Good had long low-RMS stretches, but no `clip_out`.
- Decode backlog: latest not observable. Prior backlog was elevated but no overflows/drops/resets.
- ARQ timing: latest not reached. Prior Good never entered data ARQ; prior AWGN completed with bounded retries/timeouts.

Required external hardware rerun, for the runner outside this sandbox:

```bash
SSH_KEY="$HOME/.ssh/id_pi5" ./agents/run_hardware_sentinel.sh

Changed Files

None. git diff --stat is empty.

Commands Run

Read required docs/task with sed; inspected reports/logs with rg, find, wc, tail, and sed; checked repo state with git status --short --branch, git log --oneline -10, and git diff --stat.

Residual Risks

A real Good-fading handshake issue remains possible from the prior valid run, but the latest failure proves only that the hardware sentinel was run where SSH is blocked. If the external rerun restores the hardware lane and Good still fails, the next bounded task should target MC-DPSK CONNECT acquisition/retry under injected Good fading.
2026-04-30T22:02:12.447532Z ERROR codex_core::session: failed to record rollout items: thread 019de067-b05e-7172-9590-1ca0bd04bf00 not found
tokens used
96,713
Results

Latest sentinel failure is confirmed, but rejected as a modem regression. Both latest cases in metrics.tsv failed with rc=255 before station B started. The wrapper logs show socket: Operation not permitted and ssh: connect to host 192.168.160.112 port 22: failure, so this is an agent sandbox/network execution failure, not PHY, ACK, ARQ, LDPC, sync, or calibration evidence.

Comparison:

• Prior rerun 20260430_171908 also failed pre-modem, but from DNS: ssh: Could not resolve hostname pi5tester.
• Prior valid-ish run 20260430_165403 had AWGN pass with warnings: retransmissions=4, timeouts=3, frame_success=100.0, peak_backlog_ms=3580.0.
• Its Good case failed during MC-DPSK connect: ALPHA received PONG and sent CONNECT, but BRAVO never decoded CONNECT; metrics show frames_sent=0, acks_rcvd=0, peak_backlog_ms=3470.0.

Layer classification:

• PHY/sync: latest not observable. Prior Good suggests CONNECT acquisition/sync loss under injected Good fading.
• ACK loss: not primary; latest no traffic, prior AWGN ACK/SACK path worked.
• Audio clipping/silence: latest not observable; no A/B modem logs. Prior runs include one audio device open failure and Good had long low-RMS stretches, but no clip_out.
• Decode backlog: latest not observable. Prior backlog was elevated but no overflows/drops/resets.
• ARQ timing: latest not reached. Prior Good never entered data ARQ; prior AWGN completed with bounded retries/timeouts.

Required external hardware rerun, for the runner outside this sandbox:

SSH_KEY="$HOME/.ssh/id_pi5" ./agents/run_hardware_sentinel.sh

Changed Files

None. git diff --stat is empty.

Commands Run

Read required docs/task with sed; inspected reports/logs with rg, find, wc, tail, and sed; checked repo state with git status --short --branch, git log --oneline -10, and git diff --stat.

Residual Risks

A real Good-fading handshake issue remains possible from the prior valid run, but the latest failure proves only that the hardware sentinel was run where SSH is blocked. If the external rerun restores the hardware lane and Good still fails, the next bounded task should target MC-DPSK CONNECT acquisition/retry under injected Good fading.

## Hardware Gate Summary

```text
hardware_sentinel=warn
mode=quick
timestamp=20260430_180329
head=4ba3b10
inject_gain=0.70
audio_check=0
dry_run=0
cases=2
failures=0
warnings=1
metrics=agents/reports/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail/hardware_gate/metrics.tsv

Hardware Gate Log Tail

Hardware sentinel mode=quick report=agents/reports/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail/hardware_gate
Skipping audio_path check (AGENT_HW_SENTINEL_AUDIO_CHECK=0)
==> awgn_1k_r12_snr15
PASS awgn_1k_r12_snr15
==> good_1k_r12_snr15
PASS good_1k_r12_snr15
Hardware sentinel overall=warn. Reports: agents/reports/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail/hardware_gate

[secup]

Completed. The runner-owned hardware gate executed outside the Codex sandbox and posted results above. Closing this planner proposal so future hardware follow-up proposals are not suppressed.

[secup]

Clarification of the valid runner-owned hardware result from agents/reports/20260430_175943-017-github-issue-investigate-hardware-sentinel-fail/hardware_gate/metrics.tsv:

case                  status rc channel snr rate bytes retx timeouts frame_success peak_backlog_ms
awgn_1k_r12_snr15     warn   0  awgn    15  r1_2 1024  9    9        100.0         3550.0
good_1k_r12_snr15     pass   0  good    15  r1_2 1024  0    0        100.0         3440.0

The earlier socket: Operation not permitted text in the long agent log section is historical context from the pre-fix sandboxed run, not the current hardware-gate result.