fix: math_equal rejects leading-zero integers incorrectly equated by sympy (#319)#331
Merged
James-CUDA merged 1 commit intoJul 15, 2026
Conversation
…equates (James-CUDA#319) AIME answers are zero-padded 3-digit integers (000-999). When a model answers '005' and the reference is '009', _as_number strips the leading zeros (5.0 vs 9.0, not close), then the sympy fallback evaluates both as integers and returns False (5 != 9). However, a model that answers '5' against reference '005' would hit the numeric path since _as_number('5') == _as_number('005') == 5.0, grading the wrong answer as correct. Fix: reject both numeric and sympy fallback when either normalized answer is a zero-padded all-digit string (starts with '0', has length > 1, and consists entirely of digits). This preserves legitimate comparisons like '0.5' vs '1/2'. Fixes James-CUDA#319
This was referenced Jul 15, 2026
carlh7777
added a commit
to carlh7777/Gittensor-TinyRouter
that referenced
this pull request
Jul 15, 2026
…ero-padding checks Updated the `_is_zero_padded` function to first check the length of the string before accessing its first character, preventing an IndexError when handling empty strings. Added a regression test to ensure that empty strings are correctly handled without raising exceptions. This change addresses issues surfaced during CI tests related to scoring empty probes in the math_equal function. Fixes James-CUDA#319, James-CUDA#331
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix: AIME leading-zero answers corrupted by sympy fallback
AIME answers are zero-padded 3-digit integers (000-999). When a model
answers
5against reference005, the numeric comparison bridges thembecause
_as_numberstrips leading zeros (both evaluate to 5.0).Fix
Reject numeric and sympy fallback when either normalized answer is a
zero-padded all-digit string (starts with
0, length > 1, all digits).This forces AIME-style answers to match only via exact string equality.
The guard is deliberately narrow (
.isdigit()) so legitimate comparisonslike
0.5vs1/2and0.333vs1/3are unaffected.Fixes #319
Support my work: If this fix helped, consider buying me a coffee at https://buymeacoffee.com/muhamedfazalps ☕