wip ci(smoke): enable agent isolation on smoke runs

[akshaylive]

Summary

• Set `agent.isolation: true` in `tests/experiments/default.yaml` so smoke tasks exercise coder_eval's filesystem-perimeter sandbox
• Pin the e2e smoke job to `ubuntu-24.04` and install `bubblewrap` + `socat` (with the AppArmor sysctl + `aa-disable` workarounds) so the SDK's bwrap sandbox actually engages instead of silently falling back to no-op

Mirrors UiPath/coder_eval#207.

Test plan

• Smoke Skill Tests workflow passes on this PR
• Spot-check one task's `sdk_options.json` (or task.json) to confirm `sandbox.enabled=true` was sent to the SDK

🤖 Generated with Claude Code

[github-actions]

Claude finished @akshaylive's task in 1m 39s —— View job

---

Review: PR #525 — ci(smoke): enable agent isolation on smoke runs

• Read review criteria and project conventions
• Read full diff and changed files
• Check PR conversation for existing feedback
• Perform review against criteria
• Post review

---

Summary

Enables filesystem-perimeter sandboxing (agent.isolation: true) for all smoke test tasks by updating the default experiment config, and pins the CI runner to ubuntu-24.04 with bubblewrap + socat installed (plus AppArmor workarounds) so the sandbox actually engages.

Change-by-Change Review

1. .github/workflows/smoke-skills.yml

Severity: OK

Clean, well-scoped changes:

• ubuntu-latest → ubuntu-24.04 (line 157): Correct — ubuntu-latest currently resolves to 22.04, which ships bubblewrap 0.6.1 lacking the mount overlay features the SDK requires. The explanatory comment is thorough.
• New step: Install OS sandbox dependencies (lines 183–197): Installs bubblewrap and socat with --no-install-recommends (minimal footprint). The AppArmor workarounds are properly guarded:
  • echo 0 | sudo tee /proc/sys/kernel/apparmor_restrict_unprivileged_userns || true — gracefully no-ops if the sysctl doesn't exist.
  • The for loop over AppArmor profiles checks [ -f "$prof" ] before disabling, with || true catching both missing-file and aa-disable failures. Correct bash short-circuit.
• The detect job stays on ubuntu-latest — appropriate since it only does git operations and doesn't need bwrap.

One observation: when GitHub eventually moves ubuntu-latest to 24.04 (expected mid-2025, may already be in progress), the pin becomes redundant but harmless. No action needed now, just noting for future cleanup.

2. tests/experiments/default.yaml

Severity: OK

isolation: true added at line 21, correctly placed under agent: at the same indentation level as max_turns, allowed_tools, and plugins. The YAML structure is valid.

What's Missing

Nothing identified. This is a CI infrastructure change — no skill files are modified, so no CODEOWNERS, frontmatter, or skill-body checks apply.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill files changed
E2E Tests	N/A	Test config updated, not test tasks
Skill Body	N/A	No skill files changed
References & Assets	N/A	No skill files changed
Repo Hygiene	OK	Scoped to CI + test config, no secrets, no cross-skill refs

Issues for Manual Review

• Confirm the upstream UiPath/coder_eval#207 change is merged/available so the SDK actually honors isolation: true when it receives it.
• Per the PR's test plan: spot-check one task's sdk_options.json (or task.json) in a CI run to confirm sandbox.enabled=true was sent to the SDK.

Conclusion

Clean, well-documented CI change. The runner pin and sandbox dependency installation are correctly implemented with appropriate fallbacks. No issues found — approve.

[akshaylive]

After debating back and forth, while this change works, it's very brittle. Closing this with no-op.