Skip to content

v0.1.5 - Benchmark harness + cross-file call edge fix + docs

Choose a tag to compare

@knight22-21 knight22-21 released this 17 Sep 18:06
· 89 commits to main since this release
1e5535f

What's new in v0.1.5

Indexer fix

  • Cross-file call edges now resolve correctly. BaseParser.resolve_intrafile_refs was consuming CALLS refs that matched import stubs, preventing them from reaching cross-file resolution. Fixed by skipping CALLS-to-import-stub wiring intra-file. Impact: run_payment -> compute_checksum and all similar cross-file calls now appear correctly in get_callers().

Level 1 benchmark harness

  • Token reduction measurement via tiktoken (no API key required)
  • LLM-as-judge accuracy eval via Ollama cloud (gpt-oss:120b)
  • Fixture corpus (10 tasks) + psf/requests v2.32.3 corpus (10 tasks)
  • CI workflow: runs on every push/PR, token-only, results as build artifact

Measured results

  • 88.5% token reduction on psf/requests v2.32.3 (real-world, 6k-12k token files)
  • Accuracy: baseline 0.81 / CodePrism 0.66 on requests corpus

Documentation

  • docs/benchmark-results.md - full methodology and all run results
  • docs/architecture.md - 5-layer system design and data flow
  • docs/changelog.md - version history
  • README: Benchmarks section + Documentation table

Run the benchmark yourself

pip install -e ".[bench]" tree-sitter-python
python -m benchmarks.setup_repos
python -m benchmarks.run_token_benchmark --tasks benchmarks/tasks/requests_tasks.json