Repository navigation
How Diffing Works
This page walks the full pipeline: extract → (ocr) → align → diff → enrich → report, as implemented in core/src/pipeline.rs. Everything is 100% local — extraction, embeddings and heuristics never touch the network — and everything is deterministic: no wall-clock reads, no randomness, no HashMap-iteration dependence in decision paths.
Each format has a dedicated module under core/src/extract/ that converts the file into a normalized DocumentModel: an ordered list of blocks (headings, paragraphs, table cells, slide runs…), a format tag, page count, and any extraction warnings.
| Format | Engine | Details |
|---|---|---|
PDFium (pinned chromium/8165 build, loaded dynamically via pdfium-render) |
Text layer per page, per-character boxes rebuilt into lines and paragraphs (Y-cluster/X-order grouping). Pages with an empty text layer are counted; when more than 30% of pages are textless the model gets a "scanned PDF suspected" warning and becomes eligible for the OCR fallback. | |
| DOCX | raw OOXML — zip + roxmltree over word/document.xml
|
Own parser maps w:p/w:r/w:t to paragraphs/runs, tables (w:tbl), and tracked changes: w:ins is rendered as accepted-inserted text and w:del (w:delText) as accepted-removed. |
| XLSX | calamine | Cell values (Excel-cached results) plus formula strings, so "formula changed but cached value same" is visible. Rows are capped with a truncation warning. |
| PPTX |
zip + roxmltree on ppt/slides/slideN.xml
|
<a:t> runs in slide order. |
| TXT / Markdown | direct read (64 MB cap) | Text split into block structure by headings/paragraphs. |
Zip safety: all OOXML parsing runs under ZipBudget — declared sizes are reserved up-front, per-entry reads are capped, and "lying headers" are detected by reading one byte past the declared size. roxmltree keeps its default entity-expansion limits, so billion-laughs XML is defused.
Normalization (phase 02): text is Unicode NFC-normalized and whitespace-collapsed before anything downstream sees it, so visually identical text compares identically regardless of encoding quirks.
The two documents must have the same format — a .txt vs .docx pair is rejected with a format-mismatch error, not silently coerced.
Only when the document is a suspected-scanned PDF (> 30% textless pages) and a user-installed Tesseract with the Vietnamese traineddata is detectable, textless pages are rasterized at 300 DPI and OCR'd page by page under a hard 90-second per-page timeout (max 50 pages per job). OCR blocks are merged into the DocumentModel as ordinary blocks with anchor: "ocr" provenance — everything downstream works unchanged and the report honestly marks which blocks came from OCR. Extraction never fails because of OCR. Details: OCR-experimental.
This is the heart of DiffLens: pairing old blocks to new blocks by meaning so moved clauses match instead of showing as delete+insert noise.
Pure regex + hash maps, no embeddings:
-
Clause headings matched by their clause key when unique in both documents:
Điều N,Article N,Mục N,Phụ lục, and hierarchical numbers like1.2.3. - Unique headings / long identical text: exact normalized-text matches occurring exactly once per document (headings of any length; other blocks only when ≥ 40 chars).
-
Long money strings:
12.500.000,1,234,567,12345678— ≥ 7 digits, unique on both sides.
Anchors are hard constraints: pass 2 never realigns them. This is why contract-style documents align crisply even before any model runs.
A cosine-similarity matrix is computed over block embeddings (multilingual-e5-small ONNX, mean pooling, "query:"/"passage:" prefixes), then a monotonic Needleman–Wunsch alignment with gap penalty produces pairs. Key properties:
- Anchors win — they are masked out of the NW matrix.
- Matches below the similarity threshold (default 0.55) are rejected; unmatched blocks fall through.
- Gap penalty is −0.2 per skipped block, weighted ×1.5 for headings (skipping a heading is more suspicious than skipping a paragraph).
- The NW traceback recomputes cell values exactly and uses a fixed preference order (diag > up > left), and large documents are chunked at 2000 blocks per matrix side with chunk windows locked in lockstep — both required for run-to-run determinism.
Blocks still unmatched after pass 2 get a window search (default half-width 3) around the position the monotonic pass predicted. A counterpart is accepted only above a higher threshold (default 0.85) — that is what turns "clause relocated" into a moved pair instead of noise. Tie-breaks use (similarity, distance, index) with epsilon guards.
Every pair records how it was established (anchor / semantic / move_repair) and its cosine similarity — this is the alignment half of the audit trail. Quality is gated in CI by a gold corpus requiring precision ≥ 0.85 and recall ≥ 0.80 with the real model.
Within each aligned pair, similar (Myers + Patience) produces inline spans — equal / insert / delete — so the UI can highlight the exact words that changed inside a paragraph. Added and removed changes carry their full block text. Stats (added/removed/modified/equal counts) are summarized from the change list.
This is the "deterministic AI-free intelligence" layer. It runs entirely with regexes, fixed lexicons and arithmetic.
-
Money: currency tokens (
₫ VNĐ đồng VND đ USD $ EUR €) with Vietnamese scale words —triệu×1e6,tỷ×1e9,nghìn/ngàn/k×1e3.500 triệu→ 5e8. Scale word with no currency marker defaults to VND; the ambiguous bare1.500.000(dot-thousands vs decimal) is rejected as money without a currency marker. -
Dates: ISO and Vietnamese forms (
ngày 15 tháng 3 năm 2026), converted to absolute day numbers. -
Durations ("30 ngày", "2 months") converted to day-equivalents — with a disambiguation pass so
ngày 15 tháng…is a date, never a duration. - Percent, references (normalized ids), clause names.
Paired old→new entities produce flags with explicit math, e.g.:
- Money:
5,000,000 → 2,500,000 VND→ relative delta(new − old)/|old|= −50%, plus absolute delta and currency/scale-change notes (currency comparison guarded by|old| > 1e-9). - Dates:
2026-01-30 → 2026-03-01→ "+30 days" (day-number difference). - Durations: 30 → 15 days = −50% relative delta on day-equivalents.
- Percent: relative delta and percentage-point change.
Rules: MONEY_CHANGED, DATE_CHANGED, DURATION_CHANGED, PERCENT_CHANGED, REFERENCE_CHANGED, CLAUSE_RENAMED; unpaired leftovers become ENTITY_REMOVED / ENTITY_ADDED. Every rule maps to an i18n label key (flag.moneyChanged, flag.dateChanged, flag.durationChanged, flag.percentChanged, flag.referenceChanged, flag.clauseRenamed, flag.entityAdded, flag.entityRemoved, flag.keywordRisk) — the frontend's EN/VI parity test fails if a rule lacks a translation.
score = min(100, max(entity_rule_points) + keyword_uplift)
- Rule points (max, not sum): money 35, date 30, duration 30, percent 30, reference 20, clause-renamed 15, added/removed entity 15 (rich) / 10 (plain).
- Keyword uplift: a fixed risk lexicon of VN+EN surface forms in categories like termination, liability, indemnity, with category weights; ≥ 3 distinct categories add a small breadth bonus (+5).
- Bands: 0 = info, 1–29 = low, 30–59 = medium, 60–79 = high, ≥ 80 = critical. Pure formatting changes (whitespace/punctuation/case only) score 0 → info.
Same input documents → byte-identical report JSON. Concretely: lexicon matches are collected then sort_unstable()ed; anchor candidates are sorted before use; pairing keys are quantized to i64 for total ordering; move-repair tie-breaks are fixed; the store injects timestamps from the shell so the core stays wall-clock-free. There is a dedicated test asserting serde_json equality across repeated runs. This is why the audit trail can honestly say "why this severity": every input to the score is inspectable and reproducible.
The change list, stats, embeddings/OCR flags and the heuristic audit trail become a DiffResult (JSON) consumed by the UI, persisted to SQLite, and rendered by the HTML and Markdown exporters (hostile-content escaping with golden tests — see Privacy-and-Security).
The optional local LLM (see AI-Explanations-Setup) sits strictly outside this pipeline:
| It DOES | It does NOT |
|---|---|
| Explain one selected change in plain EN/VI language, streamed token by token | Re-compute, reorder or re-severity any change |
| Receive only the change's context (old/new text, flags, audit evidence) under a token budget | See your other documents or the full file contents |
| Run on your machine on a loopback-only port | Touch the network (the sidecar binds 127.0.0.1 only) |
It is not legal advice. Severity bands come from rules with published point values; the LLM's summary is generated text from an Apache-2.0 Qwen model running locally — DiffLens never depends on it for any decision, and the app remains fully functional (all diffs, severities, exports) without any model installed.