Skip to content

chore: remove deprecated eval infra and update docs with new eval results#422

Merged
BYK merged 1 commit into
mainfrom
chore-cleanup-eval-infra
May 20, 2026
Merged

chore: remove deprecated eval infra and update docs with new eval results#422
BYK merged 1 commit into
mainfrom
chore-cleanup-eval-infra

Conversation

@BYK
Copy link
Copy Markdown
Owner

@BYK BYK commented May 20, 2026

Summary

Removes ~10MB of dead eval infrastructure and updates website/README with context retention results.

Cleanup

Item Size Reason
auto-mem0.ts 295 lines Dead code — never imported, Python sidecar bridge unreachable
cost-verifier.ts 268 lines Dead code — never imported, only referenced in string literals
fixtures/recorded-responses/ 2.0 MB (23 files) Old v1 fixtures, superseded by v3
fixtures/recorded-responses-full/ 7.8 MB (16 files) Old v2 fixtures, superseded by v3
fixtures/projects/ 17 files Unreferenced fake project trees
fixtures/sessions/ 4 empty dirs Vestigial from earlier design
auto-mem0 references ~10 locations Removed from types.ts, scenario files, CLI help

Docs Updates

Website (docs/index.html)

  • Replace "19x Compression" chip with "400K+ Token Sessions"
  • Update hero stats: +50% vs tail-window, 4.8/5.0 detail retention

README

  • Add context retention results table (Lore 3.9 vs tail-window 2.6 at 400K)
  • Update eval suite description (16 scenarios, 5 dimensions)
  • Update v5 changelog

Remove ~10MB of dead eval infrastructure:
- auto-mem0.ts: external Python baseline, never imported (295 lines)
- cost-verifier.ts: cost verification module, never imported (268 lines)
- fixtures/recorded-responses/: v1 fixtures, superseded by v3 (2MB)
- fixtures/recorded-responses-full/: v2 fixtures, superseded by v3 (8MB)
- fixtures/projects/: unreferenced fake project trees (17 files)
- fixtures/sessions/: empty vestigial directories
- auto-mem0 references cleaned from types.ts, scenario files, CLI help

Also updates website hero stats and README eval section with context
retention results from #414.
@BYK BYK self-assigned this May 20, 2026
@BYK BYK enabled auto-merge (squash) May 20, 2026 10:08
@BYK BYK merged commit c7a8202 into main May 20, 2026
13 of 15 checks passed
@BYK BYK deleted the chore-cleanup-eval-infra branch May 20, 2026 10:14
This was referenced May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant