Skip to content

Harden CRUXEval output comparison - #2580

Draft
Danielxu0208 wants to merge 1 commit into
open-compass:mainfrom
Danielxu0208:agent/fix-cruxeval-equality
Draft

Harden CRUXEval output comparison#2580
Danielxu0208 wants to merge 1 commit into
open-compass:mainfrom
Danielxu0208:agent/fix-cruxeval-equality

Conversation

@Danielxu0208

Copy link
Copy Markdown

Summary

  • build the CRUXEval output assertion from parsed expressions instead of interpolated source
  • reuse OpenCompass's type-aware equality helper to reject forged __eq__ results
  • execute dataset code before injecting the protected comparison helper
  • add adversarial and compatibility regression coverage

Why

CRUXEval previously evaluated assert {expected} == {prediction}. A generated object with an always-true __eq__, or an expression that changed the outer boolean assertion, could be scored as correct without matching the expected output. This bypasses benchmark correctness rather than solving the task.

Validation

  • rejected top-level and nested forged-equality objects
  • rejected outer-assertion injection payloads
  • retained valid nested builtin values, multiple predictions, and echo anti-cheat behavior
  • python -m py_compile opencompass/datasets/cruxeval.py tests/datasets/test_cruxeval.py
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants