Skip to content

Concepts

Jump to bottom
Praveen Kumar edited this page Jul 11, 2026 · 2 revisions

Concepts

The recipe

A recipe is a versionable YAML document that describes exactly how to clean a table: column renames, ordered ops, frame-level ops (dedup), and validation rules.

version: 1
columns:
  "Customer Name":
    rename_to: customer_name
    ops: [strip_whitespace, title_case]
validate:
  - {column: email, check: valid_email, on_fail: quarantine}
  • Generated by CleanFrame (rules or LLM), edited by you, owned by git.
  • Replayed by the executor with zero AI calls.
  • Round-trips losslessly: Recipe.from_yaml(r.to_yaml()) == r.

See Recipe specification.

Pipeline stages

DataFrame / file
   → profile          (semantic types, stats)
   → detect           (Issues + proposed ops)
   → plan             (RulesPlanner or LLMPlanner → Recipe)
   → execute          (pure pandas, 4 phases)
   → validate         (quarantine / error / warn / drop / null)
   → diff             (cell-level lineage)

cf.report() stops after detect. cf.apply_recipe() skips profile/detect/plan and only executes (+ optional drift check).

Modes

Modes gate which detector proposals enter the recipe (by confidence) and how strict execution/validation is:

Mode Min confidence Drift / missing columns
review 0.50 Warn / skip missing columns
auto 0.65 Same as review for missing cols
strict 0.85 Raise on drift, missing columns, validation failures

Quarantine (not silent delete)

Failed validation rows go to result.quarantine with a _cf_quarantine_reason column by default (on_fail: quarantine). CleanFrame never silently drops bad rows unless you explicitly set on_fail: drop.

Cell-level diff & lineage

Every changed cell is attributed: row_id, column, before, after. The executor keeps a stable positional row id and column lineage through renames and derived columns (extract_currency).

On very large dirty frames, detail storage is capped (default 100k cells) so memory stays bounded; the count of changed cells remains exact. See Production guide.

Schema drift

When you replay a recipe, CleanFrame fingerprints the incoming frame and compares it to source_fingerprint in the recipe. Renamed columns, new formats, and type shifts raise DriftError by default (on_drift="error").

cf.apply_recipe(df, recipe, on_drift="error")   # default
cf.apply_recipe(df, recipe, on_drift="warn")
cf.apply_recipe(df, recipe, on_drift="ignore")  # dangerous; CLI --force

cf.suggest_update() applies safe mechanical patches (repoint renamed sources, extend parse_date formats).

Target schema

Optional. When provided, drives:

  1. Column mapping (fuzzy name + type)
  2. Validation synthesis (not_null, >= min, valid_email, …)

Infer a draft with cf.infer_schema(df) or cleanframe infer-schema FILE.

Invariants (non-negotiable)

  1. Determinism — same input → same recipe / frame / diff
  2. LLM never touches rows — metadata (or approved sample) only
  3. Recipes round-trip — YAML ↔ object lossless
  4. Nothing silently imputed or dropped
  5. Every changed cell is tracked

Details in CONTRIBUTING.md.

Clone this wiki locally