feat: add inspect_ai task wrapper for ELEPHANT sycophancy benchmark#2
Merged
Conversation
Adds an inspect_ai task that wraps the 40 ELEPHANT scenarios in elephant/scenarios.json as an Inspect-runnable task, scored by model_graded_qa against a three-dimensional sycophancy rubric (validation, indirectness, framing) that matches the dimensions the underlying ELEPHANT benchmark targets. This is the upstream artifact required by the Inspect Evals Register (register/<eval>/eval.yaml needs an upstream repo with pyproject.toml, inspect_ai dep, and tasks defined via @task decorator). Files: - pyproject.toml: minimal hatchling-based package, declares inspect_ai >= 0.3.0 as runtime dep, ships elephant_inspect as the wheel. - elephant_inspect/__init__.py: re-exports elephant_sycophancy. - elephant_inspect/task.py: @task definition; loads scenarios from the sibling elephant/ directory at runtime. Headline benchmark numbers (5.8% composite sycophancy under augmentation on GPT-4o) live in elephant/README.md. This task replicates the scoring shape so external inspect_ai users can run the eval themselves. Signed-off-by: Ejentum <info@ejentum.com>
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an
inspect_aitask that wraps the 40 ELEPHANT scenarios inelephant/scenarios.jsonas an Inspect-runnable task. Scoring usesmodel_graded_qaagainst a three-dimensional sycophancy rubric (validation, indirectness, framing) that mirrors the dimensions the original ELEPHANT benchmark targets.Why now
This is the upstream artifact required by the Inspect Evals Register (see UKGovernmentBEIS/inspect_evals/register/README.md). A register entry there points at this repo + a pinned commit + a task path, so external
inspect_aiusers can run the ELEPHANT eval from their own checkout.Files
pyproject.toml: minimal hatchling-based package, declaresinspect_ai >= 0.3.0as a runtime dep, shipselephant_inspect/as the wheel.elephant_inspect/__init__.py: re-exportselephant_sycophancy.elephant_inspect/task.py:@taskdefinition; loads scenarios from the siblingelephant/directory at runtime; scorer usesmodel_graded_qawith a rubric mapping to the three ELEPHANT dimensions.132 lines added, 0 removed. No changes to existing files. Existing benchmark folders (
elephant/,arc-agi-3/, etc.) are untouched.Test plan
elephant/scenarios.json; no data duplication.inspect eval src/elephant_inspect/task.py@elephant_sycophancy --limit 5(cannot run in this environment without live keys; deferred to a follow-up).Follow-up
Once merged, the register PR to
UKGovernmentBEIS/inspect_evalswill reference the merge commit SHA and theelephant_inspect/task.pypath.