feat: graphwalks token filter

[nmayorga7]

Summary

Adds optional token-count–based filtering to the GraphWalks dataset pipeline.
Samples that exceed user-specified max_context_size are now dropped during dataset preparation, constraining evals to contexts within a token budget.

What are you adding?

• Bug fix (non-breaking change which fixes an issue)
• New benchmark/evaluation
• New model provider
• CLI enhancement
• Performance improvement
• Documentation update
• API/SDK feature
• Integration (CI/CD, tools)
• Export/import functionality
• Code refactoring
• Breaking change
• Other

Changes Made

• Extended record_to_sample in graphwalks.py to:
  • Compute input token counts via get_token_count.
  • Drop records whose token length exceeds max_context_size.
  • Attach raw_input_tok_cnt to sample metadata for downstream metrics
• Updated get_dataset to accept max_context_size and pass it through to the mapper.

Testing

• Ran evaluation with and without -T max_context_size
• I have run the existing test suite (pytest)
• I have added tests for my changes
• I have tested with multiple model providers (if applicable)
• I have run pre-commit hooks (pre-commit run --all-files)

Checklist

• My code follows the project's style guidelines
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (if applicable)
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Additional Context

This feature mirrors the MRCR evaluation’s token gating strategy, but is adapted for GraphWalks.
It provides a consistent mechanism for controlling dataset size relative to model context limits, and sets up future work on token-binned scoring.

[AarushSah]

@claude please review

[claude]

Claude encountered an error —— View job

Failed with exit code 128

---

I'll analyze this and get back to you.

[AarushSah]

Thanks for the PR! Left some comments.

[nmayorga7]

Thanks for comments :)
See changes in latest commit:
fix: simplify tokenization, binning, and scoring logic