feat: add workspace skills for pi-cli eval execution#776
Merged
Conversation
- Add agentic-engineering plugin + agentv-eval-review to workspace template - Add .allagents/workspace.yaml for pi-cli skill discovery - Fix skill-trigger field (value → skill) - Remove skill-trigger assertions (pi-cli plugin discovery not working yet) - Add workers: 1 to prevent concurrent workspace corruption - Baseline: 5/9 tests pass without skill loaded Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Deploying agentv with
|
| Latest commit: |
77d049b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://7bc55310.agentv.pages.dev |
| Branch Preview URL: | https://fix-eval-workspace-skills.agentv.pages.dev |
…igger Pi discovers skills from .agents/skills/ in the workspace, not from plugin directories. Move skills there and remove .allagents/workspace.yaml. Remove skill-trigger assertions — workspace materialization for pi-cli needs separate investigation (pi runs in its own workspace, not the eval's materialized workspace). Baseline: 5/9 tests pass without skill triggering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pi-cli was always creating its own temp workspace and ignoring the eval's materialized workspace (request.cwd). This meant pi couldn't see files from the workspace template. Now consistent with copilot-cli: when request.cwd or config.cwd is provided, use it directly. Only create a temp workspace when no external cwd is available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove copied skills from workspace template. Add scripts/setup.sh that copies skills from source at eval runtime, preventing staleness. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Node.js is more universally available than bun. Script only uses stdlib modules (fs, path, child_process, url). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pi discovers workspace skills from .codex/skills/, not .agents/skills/. Copy to both directories so any provider can find them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pi discovers workspace skills from .agents/skills/ and .pi/skills/ per pi-mono docs. Replace .codex/skills/ with .pi/skills/. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…very Pi-cli does not discover skills from workspace-local .agents/skills/ or .pi/skills/ directories. Content assertions still validate review quality. Skill-trigger to be re-added once pi workspace discovery works. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The workspace hook parser requires command as an array, not a string. String values are silently ignored, causing the hook to not run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The before_all hook runs with cwd=evalDir, not the workspace. The orchestrator passes workspace_path via stdin JSON. Updated setup.mjs to read it from stdin and copy skills there. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Hook commands run from evalDir, not the workspace. Use the interpolation variable to resolve the script from the workspace. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The hook runs with cwd=evalDir which is inside the git repo. Use process.cwd() for git rev-parse instead of __dirname (which is inside the materialized workspace copy, not the repo). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This was referenced Mar 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.agents/skills/)workers: 1to prevent concurrent workspace corruptionPi-cli cwd fix
Pi-cli was always creating its own temp workspace, ignoring the eval's materialized workspace. Now when
request.cwdorconfig.cwdis provided, pi-cli uses it directly — same behavior as copilot-cli. Only creates a temp workspace when no external cwd is available.Test results
Workspace fix confirmed working — pi now sees mock plugin files (previously reported "deploy-auto plugin does not exist"). Results are nondeterministic across runs (3-5/9 pass), which is expected without the skill loaded to guide the agent.
Next step: add skill-trigger assertions once pi-cli skill discovery from
.agents/skills/in the materialized workspace is verified.🤖 Generated with Claude Code