Skip to content

How Diffing Works

vibecoder880 edited this page Nov 19, 2026 · 1 revision

How Diffing Works

This page walks the full pipeline: extract → (ocr) → align → diff → enrich → report, as implemented in core/src/pipeline.rs. Everything is 100% local — extraction, embeddings and heuristics never touch the network — and everything is deterministic: no wall-clock reads, no randomness, no HashMap-iteration dependence in decision paths.

Stage 1 — Extraction (per format)

Each format has a dedicated module under core/src/extract/ that converts the file into a normalized DocumentModel: an ordered list of blocks (headings, paragraphs, table cells, slide runs…), a format tag, page count, and any extraction warnings.

Format Engine Details
PDF PDFium (pinned chromium/8165 build, loaded dynamically via pdfium-render) Text layer per page, per-character boxes rebuilt into lines and paragraphs (Y-cluster/X-order grouping). Pages with an empty text layer are counted; when more than 30% of pages are textless the model gets a "scanned PDF suspected" warning and becomes eligible for the OCR fallback.
DOCX raw OOXML — zip + roxmltree over word/document.xml Own parser maps w:p/w:r/w:t to paragraphs/runs, tables (w:tbl), and tracked changes: w:ins is rendered as accepted-inserted text and w:del (w:delText) as accepted-removed.
XLSX calamine Cell values (Excel-cached results) plus formula strings, so "formula changed but cached value same" is visible. Rows are capped with a truncation warning.
PPTX zip + roxmltree on ppt/slides/slideN.xml <a:t> runs in slide order.
TXT / Markdown direct read (64 MB cap) Text split into block structure by headings/paragraphs.

Zip safety: all OOXML parsing runs under ZipBudget — declared sizes are reserved up-front, per-entry reads are capped, and "lying headers" are detected by reading one byte past the declared size. roxmltree keeps its default entity-expansion limits, so billion-laughs XML is defused.

Normalization (phase 02): text is Unicode NFC-normalized and whitespace-collapsed before anything downstream sees it, so visually identical text compares identically regardless of encoding quirks.

The two documents must have the same format — a .txt vs .docx pair is rejected with a format-mismatch error, not silently coerced.

Stage 1.5 — OCR fallback (experimental, PDF only)

Only when the document is a suspected-scanned PDF (> 30% textless pages) and a user-installed Tesseract with the Vietnamese traineddata is detectable, textless pages are rasterized at 300 DPI and OCR'd page by page under a hard 90-second per-page timeout (max 50 pages per job). OCR blocks are merged into the DocumentModel as ordinary blocks with anchor: "ocr" provenance — everything downstream works unchanged and the report honestly marks which blocks came from OCR. Extraction never fails because of OCR. Details: OCR-experimental.

Stage 2 — Three-pass alignment (core/src/align/)

This is the heart of DiffLens: pairing old blocks to new blocks by meaning so moved clauses match instead of showing as delete+insert noise.

Pass 1 — deterministic anchors

Pure regex + hash maps, no embeddings:

  • Clause headings matched by their clause key when unique in both documents: Điều N, Article N, Mục N, Phụ lục, and hierarchical numbers like 1.2.3.
  • Unique headings / long identical text: exact normalized-text matches occurring exactly once per document (headings of any length; other blocks only when ≥ 40 chars).
  • Long money strings: 12.500.000, 1,234,567, 12345678 — ≥ 7 digits, unique on both sides.

Anchors are hard constraints: pass 2 never realigns them. This is why contract-style documents align crisply even before any model runs.

Pass 2 — semantic Needleman–Wunsch

A cosine-similarity matrix is computed over block embeddings (multilingual-e5-small ONNX, mean pooling, "query:"/"passage:" prefixes), then a monotonic Needleman–Wunsch alignment with gap penalty produces pairs. Key properties:

  • Anchors win — they are masked out of the NW matrix.
  • Matches below the similarity threshold (default 0.55) are rejected; unmatched blocks fall through.
  • Gap penalty is −0.2 per skipped block, weighted ×1.5 for headings (skipping a heading is more suspicious than skipping a paragraph).
  • The NW traceback recomputes cell values exactly and uses a fixed preference order (diag > up > left), and large documents are chunked at 2000 blocks per matrix side with chunk windows locked in lockstep — both required for run-to-run determinism.

Pass 3 — move repair

Blocks still unmatched after pass 2 get a window search (default half-width 3) around the position the monotonic pass predicted. A counterpart is accepted only above a higher threshold (default 0.85) — that is what turns "clause relocated" into a moved pair instead of noise. Tie-breaks use (similarity, distance, index) with epsilon guards.

Every pair records how it was established (anchor / semantic / move_repair) and its cosine similarity — this is the alignment half of the audit trail. Quality is gated in CI by a gold corpus requiring precision ≥ 0.85 and recall ≥ 0.80 with the real model.

Stage 3 — Word-level diff (core/src/diff/)

Within each aligned pair, similar (Myers + Patience) produces inline spans — equal / insert / delete — so the UI can highlight the exact words that changed inside a paragraph. Added and removed changes carry their full block text. Stats (added/removed/modified/equal counts) are summarized from the change list.

Stage 4 — Heuristic intelligence (core/src/heuristic/)

This is the "deterministic AI-free intelligence" layer. It runs entirely with regexes, fixed lexicons and arithmetic.

Entity extraction (Vietnamese + English)

  • Money: currency tokens (₫ VNĐ đồng VND đ USD $ EUR €) with Vietnamese scale words — triệu ×1e6, tỷ ×1e9, nghìn/ngàn/k ×1e3. 500 triệu → 5e8. Scale word with no currency marker defaults to VND; the ambiguous bare 1.500.000 (dot-thousands vs decimal) is rejected as money without a currency marker.
  • Dates: ISO and Vietnamese forms (ngày 15 tháng 3 năm 2026), converted to absolute day numbers.
  • Durations ("30 ngày", "2 months") converted to day-equivalents — with a disambiguation pass so ngày 15 tháng… is a date, never a duration.
  • Percent, references (normalized ids), clause names.

Delta computation (rule → evidence)

Paired old→new entities produce flags with explicit math, e.g.:

  • Money: 5,000,000 → 2,500,000 VND → relative delta (new − old)/|old| = −50%, plus absolute delta and currency/scale-change notes (currency comparison guarded by |old| > 1e-9).
  • Dates: 2026-01-30 → 2026-03-01 → "+30 days" (day-number difference).
  • Durations: 30 → 15 days = −50% relative delta on day-equivalents.
  • Percent: relative delta and percentage-point change.

Rules: MONEY_CHANGED, DATE_CHANGED, DURATION_CHANGED, PERCENT_CHANGED, REFERENCE_CHANGED, CLAUSE_RENAMED; unpaired leftovers become ENTITY_REMOVED / ENTITY_ADDED. Every rule maps to an i18n label key (flag.moneyChanged, flag.dateChanged, flag.durationChanged, flag.percentChanged, flag.referenceChanged, flag.clauseRenamed, flag.entityAdded, flag.entityRemoved, flag.keywordRisk) — the frontend's EN/VI parity test fails if a rule lacks a translation.

Severity banding

score = min(100, max(entity_rule_points) + keyword_uplift)
  • Rule points (max, not sum): money 35, date 30, duration 30, percent 30, reference 20, clause-renamed 15, added/removed entity 15 (rich) / 10 (plain).
  • Keyword uplift: a fixed risk lexicon of VN+EN surface forms in categories like termination, liability, indemnity, with category weights; ≥ 3 distinct categories add a small breadth bonus (+5).
  • Bands: 0 = info, 1–29 = low, 30–59 = medium, 60–79 = high, ≥ 80 = critical. Pure formatting changes (whitespace/punctuation/case only) score 0 → info.

Determinism claim

Same input documents → byte-identical report JSON. Concretely: lexicon matches are collected then sort_unstable()ed; anchor candidates are sorted before use; pairing keys are quantized to i64 for total ordering; move-repair tie-breaks are fixed; the store injects timestamps from the shell so the core stays wall-clock-free. There is a dedicated test asserting serde_json equality across repeated runs. This is why the audit trail can honestly say "why this severity": every input to the score is inspectable and reproducible.

Stage 5 — Report

The change list, stats, embeddings/OCR flags and the heuristic audit trail become a DiffResult (JSON) consumed by the UI, persisted to SQLite, and rendered by the HTML and Markdown exporters (hostile-content escaping with golden tests — see Privacy-and-Security).

What the AI layer does — and does NOT do

The optional local LLM (see AI-Explanations-Setup) sits strictly outside this pipeline:

It DOES It does NOT
Explain one selected change in plain EN/VI language, streamed token by token Re-compute, reorder or re-severity any change
Receive only the change's context (old/new text, flags, audit evidence) under a token budget See your other documents or the full file contents
Run on your machine on a loopback-only port Touch the network (the sidecar binds 127.0.0.1 only)

It is not legal advice. Severity bands come from rules with published point values; the LLM's summary is generated text from an Apache-2.0 Qwen model running locally — DiffLens never depends on it for any decision, and the app remains fully functional (all diffs, severities, exports) without any model installed.

Clone this wiki locally