Add restriction against accessing installed package versions#687
Add restriction against accessing installed package versions#687juanmichelini merged 1 commit intomainfrom
Conversation
Prevent agents from accessing, copying, or importing from the installed version of the target package (e.g., site-packages, .venv). This closes the cheating loophole where agents could study the reference implementation instead of implementing from scratch.
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
🟢 Good taste — Simple, pragmatic fix to a real problem.
[RISK ASSESSMENT]
[Overall PR]
This is a prompt text change that closes a documented cheating loophole. Limited blast radius (single template file), no code logic changes, and the behavioral impact (preventing access to installed packages) is the intended purpose.
VERDICT:
✅ Worth merging
KEY INSIGHT:
Pragmatic solution that directly addresses observed cheating behavior with clear, comprehensive restrictions.
|
once finished we will use https://github.com/All-Hands-AI/research/pull/35 to check for cheating |
|
@VascoSch92 checked and it seems clean
|
3 participants
Summary
This PR adds a restriction to prevent agents from accessing the installed version of the target package, closing the cheating loophole discovered in the jinja benchmark run.
Problem
In the jinja benchmark, the agent tried to copy the installed jinja2 from site-packages to study the reference implementation:
The current prompt only warned against pip installing, but did not prohibit accessing the already-installed version.
Change
Added a new paragraph after the existing anti-cheating restrictions:
This is general enough to cover any Python package, not just jinja.
Testing
The detect_cheating.py script in https://github.com/All-Hands-AI/research/pull/35 can be used to verify that future benchmark runs do not exhibit this pattern.