Skip to content

fix: derive eval subjects from document content (prof-cgu)#43

Merged
ckrough merged 1 commit into
mainfrom
issue/prof-cgu
May 29, 2026
Merged

fix: derive eval subjects from document content (prof-cgu)#43
ckrough merged 1 commit into
mainfrom
issue/prof-cgu

Conversation

@ckrough
Copy link
Copy Markdown
Owner

@ckrough ckrough commented May 29, 2026

Why: Encode the synthetic-eval subject convention in code so ground-truth rows describe document content rather than the document-type form the file's header prohibits.

Changes:

  • Derive each row's subject via a model call against generated text, with a structural forbidden= guard that rejects the f"{category} {doctype}" form
  • Add scripts/backfill_eval_subjects.py to apply the same derivation in place across existing PDFs, preserving comments, key order, and entity fields
  • Validate --ai-model against MODEL_PRICING at startup; collect-then-apply on cost-cap abort; share an atomic-write helper with the generator
  • Add tests for sanitizer normalization, forbidden-form retry, empty-text raise, and fallback behavior
  • Rewrite the 80 subjects in eval/ground_truth/synthetic.jsonl to content-based form

Issues: prof-cgu

Why: Encode the synthetic-eval subject convention in code so ground-truth rows describe document content rather than the document-type form the file's header prohibits.

Changes:
- Derive each row's subject via a model call against generated text, with a structural forbidden= guard that rejects the f"{category} {doctype}" form
- Add scripts/backfill_eval_subjects.py to apply the same derivation in place across existing PDFs, preserving comments, key order, and entity fields
- Validate --ai-model against MODEL_PRICING at startup; collect-then-apply on cost-cap abort; share an atomic-write helper with the generator
- Add tests for sanitizer normalization, forbidden-form retry, empty-text raise, and fallback behavior
- Rewrite the 80 subjects in eval/ground_truth/synthetic.jsonl to content-based form

Issues: prof-cgu
@ckrough ckrough merged commit d1cb867 into main May 29, 2026
5 checks passed
@ckrough ckrough deleted the issue/prof-cgu branch May 29, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant