feat: add evaluate and repair workflows

[lipikaramaswamy]

Summary

Implements Step 5 of the rewrite pipeline: evaluate rewritten text for quality and privacy, then repair if privacy leaks are detected.

EvaluateWorkflow (engine/rewrite/evaluate.py) — 6-column NDD pipeline: LLM quality re-answer, LLM privacy re-answer, QA pairing, LLM quality comparison, metric computation (utility_score, leakage_mass, any_high_leaked), and repair-needs determination.

RepairWorkflow (engine/rewrite/repair.py) — 2-column NDD pipeline: leaked-items formatting and LLM repair via PydanticResponseRecipe inside a custom column generator with dynamically-resolved model alias.

Schema changes: PrivacyAnswer enum enforces yes/no only — privacy questions are binary ("can entity X be deduced?"), so "unknown" is a hedge that doesn't serve the pipeline; format_for_rewrite_context() now uses protected_entities instead of medium_and_high_sensitivity_entities — the rewriter and evaluator must operate on the same entity set, otherwise LOW-sensitivity entities flagged for protection (due to combinatorial re-identification risk) would be evaluated but never shown to the rewriter.

Design decision - separate workflows instead of one combined pipeline, so the orchestrator can run evaluate on all rows, filter to failing rows, run repair on just those, and re-evaluate only repaired rows (no wasted LLM calls on passing rows)

Prompts ported from the research repo with XML section tags, <<PLACEHOLDER>> + .replace() convention, and schema field names validated at import time via _field().

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring

Testing

• Tests pass locally
• Added/updated tests for changes

Related Issues

Closes #34

[asteier2026]

Only one small comment in evaluate, in the re-answer prompt line 84

[andreatgretel]

The workflow structure is solid and the metric helpers are clean and well-tested. Two concerns on the evaluation logic:

1. compute_leakage_mass and compute_any_high_leaked don't validate that the returned answer IDs match the QA IDs. If the evaluator drops a HIGH-sensitivity answer, the leak isn't counted and the row can skip repair. For a privacy gate, this should fail closed.

2. compute_utility_score averages whatever compare scores it gets without checking coverage. A partial compare response inflates the score.

Also flagged: _COL_NEEDS_REPAIR should probably move to constants.py before the orchestrator PR, and the repair module could use more test coverage.

[andreatgretel]

repair.py:69-74 and evaluate.py:163-168

def _parse_privacy_answers(raw: Any) -> list[PrivacyAnswerItemSchema]:
    if isinstance(raw, PrivacyAnswersSchema):
        return raw.answers
    if isinstance(raw, dict):
        return PrivacyAnswersSchema.model_validate(raw).answers
    raise TypeError(...)

_parse_privacy_answers and _parse_privacy_qa are identical between evaluate and repair - maybe extract to a shared module or add classmethods on the schemas?

[lipikaramaswamy]

repair.py:69-74 and evaluate.py:163-168

def _parse_privacy_answers(raw: Any) -> list[PrivacyAnswerItemSchema]:
    if isinstance(raw, PrivacyAnswersSchema):
        return raw.answers
    if isinstance(raw, dict):
        return PrivacyAnswersSchema.model_validate(raw).answers
    raise TypeError(...)

_parse_privacy_answers and _parse_privacy_qa are identical between evaluate and repair - maybe extract to a shared module or add classmethods on the schemas?

Extracted to rewrite/parsers.py. The shared parse helpers (parse_privacy_answers, parse_privacy_qa, parse_quality_qa, parse_quality_answers, parse_quality_compare, parse_sensitivity_disposition) and the field() validator live there. Both evaluate.py and repair.py import from it.

[andreatgretel]

nice progress on this PR. one thing i’d consider as follow-up, borrowing a bit from DD’s patterns, is tightening the workflow contract around the custom columns:

• make required_columns exhaustive for anything the generator actually reads, since DD uses that metadata for DAG ordering and missing-input checks
• keep pushing validation to the parser/schema boundary, not the metric helpers. missing IDs are fixed now, but duplicate / extra IDs would be good to reject there too
• add one tiny workflow-level preview/smoke test that runs the evaluate/repair columns together on a row, so metadata issues show up before orchestrator wiring

none of that needs to block this PR beyond the concrete bugs already called out, but it feels like the next step that would make these workflows more DD-native and harder to misuse later.

[lipikaramaswamy]

nice progress on this PR. one thing i’d consider as follow-up, borrowing a bit from DD’s patterns, is tightening the workflow contract around the custom columns:

• make required_columns exhaustive for anything the generator actually reads, since DD uses that metadata for DAG ordering and missing-input checks
• keep pushing validation to the parser/schema boundary, not the metric helpers. missing IDs are fixed now, but duplicate / extra IDs would be good to reject there too
• add one tiny workflow-level preview/smoke test that runs the evaluate/repair columns together on a row, so metadata issues show up before orchestrator wiring

none of that needs to block this PR beyond the concrete bugs already called out, but it feels like the next step that would make these workflows more DD-native and harder to misuse later.

First two done: required_columns is now exhaustive across all 7 generators (audited), and duplicate/extra IDs are rejected at the schema boundary via _validate_id_coverage(). The workflow-level smoke test we'll pick up in the orchestrator PR.

[andreatgretel]

Approving. The blocking issues I called out are addressed on the current head.

One follow-up to keep in mind: the placeholder-collision fix looks complete in the evaluate prompts, but _render_repair_prompt() still appears to have the same sequential .replace() risk if user text contains one of the later placeholder tokens. I do not think that needs to hold this PR, but it would be good to clean up next.

[lipikaramaswamy]

Approving. The blocking issues I called out are addressed on the current head.

One follow-up to keep in mind: the placeholder-collision fix looks complete in the evaluate prompts, but _render_repair_prompt() still appears to have the same sequential .replace() risk if user text contains one of the later placeholder tokens. I do not think that needs to hold this PR, but it would be good to clean up next.

Decided to fix here. Added render_template() to parsers.py — single-pass regex substitution that replaces all placeholders simultaneously. Both evaluate and repair prompts now use it, so no .replace() chain ordering risk anywhere. Also added test_parsers.py with a dedicated cross-contamination test that verifies <<SCORE>> inside user text doesn't get substituted. (and tracking #66)

Other changes I've made since your review:

• .model_dump() on all three evaluate custom column outputs (DD's sanitize_record() needs JSON-serializable values)
• Repair column writes to COL_REWRITTEN_TEXT_NEXT instead of overwriting the seed column (DD rejects generated columns with the same name as seed columns)