Smoke test Screen Recording and Accessibility permission flow

[haasonsaas]

Summary

Run a hardware-backed smoke test for the packaged macOS app permission flow. This covers the remaining #1079 acceptance that CI cannot exercise because TCC prompts require a signed app on a real desktop session.

Acceptance

• Install or launch the packaged EvalOps agentd.app artifact from the release packaging path.
• Confirm macOS prompts for Screen Recording and Accessibility use the app bundle identity/display name.
• Grant both permissions in System Settings -> Privacy & Security and relaunch.
• Confirm the menu-bar item starts, pause/resume works, and a local-only batch can be flushed to ~/.evalops/agentd/batches/.
• Confirm denied app/path/secret smoke fixtures still drop frames before persistence.
• Attach the tested artifact checksum, macOS version, and concise pass/fail notes before closing.

Notes

Automated tests cover the scrub/drop, batching, direct Chronicle submit, Secret Broker submit, local fallback, and packaging paths. This issue is specifically for the TCC/user-permission behavior that must be validated on hardware.

[haasonsaas]

Progress update: PR #27 merged in 9806c9b.

What changed:

• Added scripts/permission_smoke.sh to make the hardware permission pass reproducible.
• The helper packages the app when needed, records macOS version, app/zip checksums, and codesign authority evidence into dist/permission-smoke-report.md, and opens the app unless --no-launch is supplied.
• README now documents the helper next to the package commands.

Validation before merge:

• scripts/permission_smoke.sh --help
• scripts/permission_smoke.sh --no-launch
• shellcheck scripts/package_app.sh scripts/permission_smoke.sh
• swift test
• swift build -Xswiftc -warnings-as-errors
• xcrun swift-format lint --strict --recursive Sources Tests Package.swift
• git diff --check
• GitHub required ci and Socket checks passed on Add permission smoke helper #27.

I intentionally did not launch the screen-capture app from this non-interactive run. To close this issue, run scripts/permission_smoke.sh in an interactive macOS desktop session, grant Screen Recording and Accessibility when prompted, complete the generated report, and attach the report notes/checksum here.

[haasonsaas]

Progress update: PR #48 merged in c47f483 / squash on 2026-04-29.

The permission smoke helper now captures codesign authority evidence correctly with codesign -dvv, joins authorities with a real comma-space separator, and still generated dist/permission-smoke-report.md successfully in --no-launch mode.

Validated locally: scripts/permission_smoke.sh --no-launch plus the full Swift/package/shellcheck gates on #48.

Still required to close this issue: run scripts/permission_smoke.sh in an interactive macOS desktop session, grant Screen Recording and Accessibility, relaunch, complete the generated report, and attach the pass/fail notes/checksums here.

[haasonsaas]

Progress update: PR #49 merged in e95c602 on 2026-04-29.

Permission-flow recovery and local tooling improved:

• Added script/build_and_run.sh and .codex/environments/environment.toml, giving Codex/local developers a single Run action for package-launch-verify, logs, telemetry, debug, and permission-smoke paths.
• Added capture-start retry after TCC denial; after Screen Recording/Accessibility are granted, agentd will retry capture startup instead of requiring a full relaunch.
• Used Computer Use to inspect System Settings Accessibility. I did not unlock or change settings. EvalOps agentd.app was visible and enabled there, while ScreenCaptureKit still denied screen capture for the current packaged app copy.
• Desktop logs confirmed the new retry path: initial TCC denial followed by another capture-start attempt 15 seconds later.

Still required to close: user-approved interactive Screen Recording/System Settings validation, because granting/changing Privacy & Security permissions is a local system-setting change and needs action-time confirmation.

[haasonsaas]

Progress update: PR #50 merged in 1ea7c71 on 2026-04-29.

I re-ran the desktop permission path after Security & Privacy approval and found two separate blockers:

• The app was ad-hoc signed, so macOS can bind TCC approval to the exact CDHash/designated requirement. Rebuilding can make the same bundle path look like a different TCC client.
• The real foreground app in our current dogfood workflow is often Codex (com.openai.codex), which was not in the default local capture allowlist. That made ScreenCaptureKit start successfully but local batches drop frames as deniedApp.

Landed fixes:

• script/build_and_run.sh --tcc-verify relaunches the exact packaged app without rebuilding and inspects info-level agentd logs for capture start/failure.
• script/build_and_run.sh --local-batch-verify verifies end-to-end ScreenCaptureKit -> scrub/policy -> local batch persistence and prints only counts/metadata lengths, not OCR text.
• scripts/permission_smoke.sh --no-launch now records codesign signature, CDHash, and designated requirement in the smoke report.
• Default developer allowlist now includes Codex, ChatGPT, Claude Desktop, Ghostty, and Cursor helper bundle IDs.
• Pipeline logs now include allowlist miss bundle=... when a captured foreground app is blocked by policy.

Interactive evidence from this machine:

• ./script/build_and_run.sh --tcc-verify on the exact approved packaged app: capture started.
• ./script/build_and_run.sh --local-batch-verify on the exact approved packaged app with a temporary config expansion: persisted 2 sanitized Codex frames, deniedApp=0, deniedPath=0, no OCR text printed.

Remaining reason to keep this issue open: Accessibility trust still needs a stable signed/notarized app identity to be durable across rebuilds. That is effectively #24; once Developer ID signing/notarization is in place, this smoke can be rerun without the ad-hoc CDHash footgun.

[haasonsaas]

Follow-up: the durable signing blocker from #24 is cleared.

Evidence:

• package-release succeeded on main: https://github.com/evalops/agentd/actions/runs/25095755314
• Notary request id: ae3c9c4f-9f9f-4a01-baec-7a6ccc1da69d
• Notary status: Accepted
• Stapler succeeded.
• Gatekeeper accepted the downloaded artifact locally as source=Notarized Developer ID with origin Developer ID Application: Jonathan Haas (7BPHA88333).

This means the next permission smoke should use the notarized artifact from that workflow instead of an ad-hoc local rebuild, so Screen Recording/Accessibility approvals bind to the stable Developer ID identity rather than a changing CDHash.