DocxDiff: Word-parity arrangement of unrelated regions, surplus table cells, imported paired styles, attribute-wise default spacing by JSv4 · Pull Request #693 · JSv4/Docxodus

JSv4 · 2026-09-03T08:20:41Z

Why

Comparing two documents that share little text — a template against a filled-in copy, two fixtures from the same suite, a memo against a contract — is where our redlines diverged most from Word's compare output. The divergence was almost entirely about arrangement, not content: which paragraph marks the two documents share, and therefore where the inserted and deleted blocks land on the page. Six changes here each take one decoded rule from Word's output and put it at the architectural owner of that decision.

What changed

Empty paragraphs no longer anchor unrelated regions (IrBlockAligner). A blank has no words, so the only thing that can make its paragraph mark "the same paragraph" on both sides is the matched content around it. The aligner nevertheless pinned a blank that was unique on each side as an exact-match anchor, which split what Word treats as one replace region into halves with the deletions scattered between them. A blank pairing now survives only when the pair before it is in place, when it is the document-final pair, or when a run of blanks reaches straight back to the preceding pair on both sides (identical blank runs). Blanks between edited paragraphs pair even when the next document restyled them (retained mark plus w:pPrChange, as Word writes it); trailing blanks are left to the chain below.

The trailing paragraph-mark chain follows Word's base-empty rule (IrMarkupRenderer.EmitGapArranged). Inside the document, the chain opens on a trailing blank on both sides and walks backwards while the base member is empty (a wordful next member fuses its runs into the first deleted paragraph); a wordful base member facing an empty next member cancels the whole chain, a wordful-against-wordful stop keeps it only when the paragraphs left before it balance, and a fused next member needs a deleted paragraph to host its runs. At the story end, a wordful member in the structural final pair blocks any further pairing. A base story that ends with a table against a next story that ends with an empty paragraph now places that final mark after the deleted table, as the wordful case already did.

A paired table row keeps a surplus base cell in place (RenderModifyRow). Two unrelated tables paired positionally, or a row that lost a column, used to lower the whole table to a deleted table followed by an inserted one. The surplus base cell now stays where it was with w:cellDel and struck content, mirroring the right-only w:cellIns case; accept removes it, reject restores it.

A paired paragraph keeps a right-side style that inserted content imports. The renderer tested only the left document's style definitions, so an edited title paragraph in a document whose inserted body uses the same title style rendered as Normal while the definition sat, imported, in the output styles part.

Default paragraph spacing is neutralized attribute by attribute. A right document whose defaults declared only line against a left declaring after=160 line=278 produced a Normal style saying just line=276, and every accepted paragraph inherited the left's space below. Word writes after=0 line=276 there; so do we now.

Default paragraph spacing attributes the right document declares are materialized, never reset. A first cut of the neutralizer decided "left-only" from the right style's effective properties, which exclude the right's docDefaults, and reset every heading's line to the built-in; the right's own declared value is carried over instead.

An imported right-only paragraph style is expressed under the left document's defaults. A style that exists only in the right document is copied into the output, where it then lives under the left docDefaults, so it was rendering every paragraph in it with the left's spacing: a long inserted document whose own defaults said line=259 after=160 came out a page longer than Word's under the left's line=276 after=200. The imported style now carries the same docDefaults delta an updated shared style gets, stated once along the basedOn chain (an import under a Normal that was itself updated inherits it and stays raw, as Word writes it), with measures written in twips.

Split/merge detection prefilters its windows. Releasing the blank anchors exposed a whole-document rewrite region (a template against a 120-page document) in which the 1:N containment scan scored hundreds of thousands of windows — 49 seconds where the comparison had taken 1.3. An exact necessary condition (two members must each share at least two words with the singular paragraph) now skips windows that cannot qualify; outputs are byte-for-byte unchanged and the pair aligns in 3 seconds.

How it was decided

Every rule is decoded from Word's own compare output rather than tuned by hand. The chain rule was checked against every interior region in the reference outputs that ends with a blank on both sides: it predicts Word's choice in 331 of 334 (the three misses are pure blank runs of unequal length between paired tables). The paired-style rule holds in 17 of 17 imported cases and 5 of 6 non-imported ones. Same-slot paragraph pairing (one shared content word) and displaced pairing (three or more) were also measured and found to already agree with the aligner's decisions, so nothing there was touched.

Validation

dotnet test Docxodus.Tests: 3964 passed, 0 failed, 3 skipped, run on the final build.
New pins in DocxDiffGapArrangementTests (unrelated-content blank, restyled blank beside edited neighbours, interior regions that chain and that cancel, one-sided blank region, wordful final pair blocking the chain, base-ends-with-table tail, surplus base cell) and DocxDiffSeamDisciplineTests / DocxDiffStyleProvenanceTests (imported paired style kept, non-imported dropped, attribute-wise spacing both ways). Three older arrangement pins that encoded the previous blank anchoring are re-pinned to the single-region shapes; every pin keeps accept ≡ right and reject ≡ left.
Warning baselines unchanged: library 113, test project 707 (the latter measured on main and corrected in CLAUDE.md, where 694 was stale).
dotnet test Docxodus.Tests: 3966 passed, 0 failed, 3 skipped, on the final build.
Render-fidelity measurement across an 803-pair reference corpus, rendering both sides through the same LibreOffice build and scoring the rasterised pages:

metric	before	after
mean	84.15	87.18
median	96.39	97.38
pairs scoring 90+	485	545
pairs scoring under 50	73	54
exact matches	254	286

123 pairs improved and 24 regressed. The largest movers are whole-rewrite comparisons whose deletions had been scattered around a spurious blank anchor, several going from the 40s to 100.

What is left

Substituting Word's own body into our package and re-rendering lifts the remaining sub-70 pairs from 57 to 88, with 25 of 35 reaching 90+. So the parts we build — styles, numbering, fonts, section geometry — are essentially right, and what remains is the arrangement of the body itself: on documents that share almost no text, Word runs one word-level diff across the whole story with paragraph marks as ordinary tokens, which interleaves insertions and deletions where we emit them in blocks. Reproducing that is a second alignment mode rather than another rule, so it is left for its own change.

Two things were measured and deliberately not done, because each turned out to be worth nothing: normalising the font table to Word's (+0.1 over 32 affected pairs) and leaving the body's final paragraph mark unmarked as Word does (0.0 over 57 pairs — a paragraph mark's revision flag does not affect layout).

Known exceptions

Three reference outputs keep both sides' marks on pure blank runs of unequal length between paired tables, where the chain rule would share one; they are single pairs against hundreds and the rule follows the majority. One pair loses ground for a reason the styles part does not explain (its updated Title style now matches Word's payload byte for byte, yet LibreOffice lays the title out differently); it is recorded for a follow-up rather than patched around.

🤖 Generated with Claude Code

…chain pairs empty marks only An empty paragraph has no words, so nothing but the matched content around it can make its paragraph mark "the same paragraph" on both sides. The block aligner nevertheless pinned a unique blank on the exact-match spine (and paired further blanks first-fit in the gap pass), which split what Word treats as one replace region into two halves: a base that ends with an empty paragraph against a next that begins with one came out as all deletions, a live empty line, then all insertions, where Word emits insertions, deletions, then the shared final paragraph mark. A blank pairing now survives only when a neighbouring block pair on both sides is paired in place — a run of blanks borrows its support from the content at either end — and the document-final pair is supported structurally, because Word always pairs the two final marks. The rule runs twice: after the spine, so the gap fill sees the whole region, and after the gap fill, so a blank beside a paragraph the gap fill paired as edited keeps its retained mark. Two trailing-region arrangement rules follow from the same evidence. The backward paragraph-mark chain continues from the final pair only through paragraphs that are empty on both sides, the final pair included; it used to open on any empty/empty candidate and continue while either side was empty, which turned the next document's last wordful paragraph into a shared mark. And a base story ending with a table against a next story ending with an empty paragraph now places that final mark after the deleted table, as the wordful case already did, instead of leaving the document ending with the deleted table. Three synthetic arrangement pins encoded the old blank anchoring and are re-pinned to the single-region shapes; new pins cover an unrelated-content unique blank, a blank beside edited neighbours, and the table-then-blank tail. Accept ≡ right and reject ≡ left hold for every shape.

…deleted cell When two rows pair but the base row has more cells than the next row — a column the next document no longer has, or two unrelated tables paired positionally — the modified-table renderer bailed on the left-only cell and the whole table fell back to a deleted table followed by an inserted one, doubling it on the page. The right-only case (w:cellIns on the accepted grid) was already handled. The surplus base cell is now emitted in its place, whole-marked deleted: w:tcPr/w:cellDel plus struck content, exactly the mark the composite renderer already uses. Accept removes the cell (= next), reject restores it (= base). Word's compare output keeps such a cell in the merged row with its content deleted, so the page shape now follows it. The bail-out that remains is the one that protects reversibility: with table format tracking off, a one-sided cell still lowers to the whole-table pair, because the cloned grid could not be restored on reject.

…content imports Word expresses a paired paragraph's format change within the style universe the output resolves. A right-only paragraph style is brought into the result only by wholly inserted paragraphs — but once one of them has imported it, a retained or edited paragraph naming the same style keeps its w:pStyle too; there is nothing to lower to direct properties. Across the reference outputs, every paired paragraph whose right style an inserted paragraph also uses keeps the reference (17 of 17), and paired paragraphs whose right style nothing inserted uses drop it (5 of 6). The renderer tested only the left document's style definitions, so a whole- document rewrite whose final paragraph pair adopted the right's heading or list style, or an edited title paragraph in a document whose inserted body uses the same title style, rendered with the default paragraph style while the definition it needed sat, imported, in the output styles part. The inserted-style set is collected once from the edit script — inserted blocks, inserted rows of paired tables, and inserted blocks inside paired cells — and both the stamped-pPr drop and the pPrChange comparison consult it alongside the left registry. The existing pin kept its shape (nothing inserted names the style, so it is dropped) and a sibling pin covers the imported case, where both shared-pilcrow paragraphs keep it.

…es from a shared blank Refines the blank-support rule from the previous commit with three findings from Word's compare output. A blank that follows an edited or retained pair now pairs with the blank across from it even when the next document restyled it — a retained mark carrying a w:pPrChange — where the gap passes (which must not let blanks claim slots ahead of content) had left a deleted blank followed by an inserted one. Trailing blanks of a region take support from the pair after them only when the region's paragraph counts balance: across the reference corpus, interior regions ending in a blank on both sides share that blank in 300 of 301 equal-count cases and in 21 of 24 cases where one side is nothing but the blank, and keep the marks on their own sides in 7 of 10 other unequal cases. The renderer's backward paragraph-mark chain applies the same gate to interior regions, and a region the aligner closed with such a shared blank continues the chain from it (the next side's last wordful paragraph fuses at the head, as Word writes it). A wordful member in the story-final structural pair still blocks the chain; the original empty-opening rule otherwise stands, since interior regions with equal counts do chain through a wordful member. Both blank passes now run after crossing normalization: a same-identity blank pair that crossed a table pair was lending support it could not keep, which retained a blank Word marks. Pins: a restyled blank after an edited paragraph keeps its mark with a pPrChange; equal-count and unequal-count interior regions; a one-sided blank region; and the earlier tail-chain pin returns to the fused shape.

…ribute The output keeps the left document's styles part, so an updated style's current payload has to cancel any docDefaults value the right document does not share. That check was per element: a right whose defaults declared only line=276 against a left declaring after=160 line=278 produced a Normal style saying just line=276, and every accepted paragraph inherited the left's 160 twips of space below — the accepted view of a web-authored document came out double-spaced against the reference output. Spacing attributes inherit one by one, and Word's compare output treats them that way: the updated Normal there reads after=0 line=276. The neutralizer now resets each left-declared attribute the right's effective spacing lacks to its built-in on the same spacing element, and the right-defaults materialization merges its attributes into a spacing element the neutralizer already created instead of skipping it. Pinned with a right whose defaults declare only line/lineRule.

The 1:N split/merge containment scan ran the LCS scorer on every adjacent window whose content-token total fell inside the coverage/slack bounds. With short singular paragraphs those bounds admit almost everything, and once the empty-paragraph anchors that used to fragment a whole-document rewrite were released, one 50-by-1100 region (a small template against a 120-page document) scored hundreds of thousands of windows: 49 seconds where the comparison had taken 1.3. TrimAndGate's first gate needs at least two window members that each carry two or more LCS-matched words, and a member can never match more words than its Word-key multiset shares with the singular paragraph. Both facts give exact necessary conditions that cost a cached multiset intersection: a singular with fewer than four words never splits or merges, and a window with fewer than two members sharing two words with it never qualifies. Neither can change a result; the same pair now aligns in 3 seconds and the full-corpus alignment outputs are byte-for-byte the same sequences.

A clean --no-incremental build of Docxodus.Tests on main reports 707 warnings, not the 694 recorded here; the library's 113 is unchanged. Measured on the merge-base of this branch and on the branch itself (no movement).

…le, not a count gate The previous commit gated an interior region's trailing blank chain on equal paragraph counts. That fit the reference corpus only because the miner had counted a fused paragraph on both sides; with correct accounting Word chains a 3-vs-2 region ([title, blank, blank] against [title', blank] before a shared table: title' fuses into the deleted title and both blanks share their marks) and keeps every mark in a 3-vs-3 one. The rule Word's output actually follows — in 331 of the 334 interior regions that end with a blank on both sides — is about what each pair joins: the chain opens on the trailing blank pair and walks backwards while the BASE member is empty (a wordful next member is fine; its runs fuse into the first deleted paragraph), a wordful base member facing an empty next member cancels the whole chain, a wordful-against-wordful stop keeps it only when the paragraphs left before it balance, and a fused next member needs a deleted paragraph to host its runs. The story-end grammar is unchanged. With that rule in the renderer, the aligner no longer needs to take trailing blank pairs on the region's behalf: next-side support is limited to blank runs that reach straight back to the preceding pair on both sides (identical blank runs, which must stay Unchanged), and the seeded-chain hand-off is gone.

…lts declare The attribute-wise neutralizer decided "left-only" from the right style's effective properties, which by design exclude the right document's docDefaults. So a right whose defaults said line=278 against a left saying line=276 had every updated heading's line reset to the built-in 240 instead of materialized at 278, and a heading-heavy document lost most of a page of alignment against the reference output. An attribute the right's docDefaults declare is now excluded from neutralization; the materialization step already carries it over with the right's value when the left values it differently. Pinned with a style that owns before/after while the right's defaults change line.

…t defaults A paragraph style that exists only in the right document is copied into the output styles part, where it lives under the LEFT docDefaults from then on. It was copied raw, so every paragraph in it inherited the left's default spacing: a long inserted document whose own defaults said line=259 after=160 rendered a page longer than Word's output under the left's line=276 after=200, and the drift compounded across a dozen pairs that share that document. Word writes the imported style with its docDefaults delta stated: right defaults the left values differently are materialized (line=259), left defaults the right never declared are reset to their built-ins, and the run side carries the resolved right formatting — exactly what an updated shared style already gets here. The import branch now runs the same two steps, and writes the spacing and indent measures in twips as Word does (the source said line="12.95pt", which a renderer cannot read as an auto line count; 259 it can). Point-unit measures and theme-color fallbacks were also tested and make no rendering difference, so they are left alone. Pinned with a right-only heading imported under differing defaults.

… chain The previous commit stated the delta on every imported right-only paragraph style. Word states it once per chain: when the default paragraph style was itself updated, its payload carries the delta and an import based on it is written raw; only under an untouched Normal does the import restate the values. Restating them under an updated Normal double-applied nothing but did override the chain in the cases where the import's own payload had already narrowed a value (a table body style at after=60 gained a before=0 Word never writes). The import pass now runs after every style's payload is final, parents first, and only adds an attribute that no ancestor in the output chain states. Pinned with an import under an updated Normal (stays raw) beside the existing pin under an untouched Normal (carries the delta).

Major, because CommentListEntry gained a `required` init property: the record is positional, so `new CommentListEntry(...)` no longer compiles without Id. v12.0.0 is tagged at 9be2dc0, the last commit whose Playwright run was green, so the six DocxDiff arrangement entries that came in with #693 stay under [Unreleased] — that change is not in this release. It hangs the browser build: comparing WC007-Unmodified against WC007-Moved-into-Table never returns in WASM, while the same comparison takes ~25 ms natively with byte-identical output. Tracked separately. The accumulated entries had also grown a duplicate `### Changed` and `### Fixed` pair from parallel PRs each appending its own heading to [Unreleased]. They are merged into one section of each kind here; no entry text changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SxehJkp2547AyP8LSd6uFS

The release cut is the reason this merge needed hands. main emptied [Unreleased] into a 12.0.0 section, and git's auto-merge put this demo's entry back at the position that text now occupies -- inside the released section, where it would have claimed the demo shipped in a version it is not in. Moved it back under [Unreleased], and dropped the Office Math entry that came with it, because that one did ship in 12.0.0 and would otherwise appear twice. Re-pinned docs/demo/redline.html from 11.0.0 to 12.0.0. The release re-pins every sibling demo page but not this one, since this one is not on main yet; what is new is that #654's engine-pin guard globs docs/demo/*.html and requires them all to name one version, so a stale pin here is now a failing check rather than a quiet inconsistency. Confirmed jsDelivr actually serves 12.0.0 before moving it. The README conflict was the same event seen from the other side: seven pages (this branch adds one) at the new pin. #697 fixes the comparison timeout that made the last CI run red. That failure was #693's, bisected and filed as #695; the fix warms the WASM engine before the first real comparison rather than touching the aligner, so my guess at the cause in that issue was wrong even though the bisect was right. Verified here: the test that timed out at 60 seconds now passes. Build and pretest clean, 61 node checks, 14 browser assertions, and the engine-pin guard passing on all seven pages. Not re-measured: #698 changes token-stream arrangement for regions with no block correspondence, which the demo's 3 KB document does not exercise.

JSv4 added 10 commits September 2, 2026 22:16

JSv4 marked this pull request as draft September 3, 2026 11:08

JSv4 marked this pull request as ready for review September 4, 2026 00:20

Merge branch 'main' into feat/diff-fidelity-round10

6e35560

JSv4 merged commit 4652f65 into main Sep 4, 2026
13 of 14 checks passed

JSv4 deleted the feat/diff-fidelity-round10 branch September 4, 2026 12:32

This was referenced Sep 4, 2026

#693 regressed the CZ move-detection comparison from 2.7s to a 60s timeout; the Playwright test job now fails on every PR #695

Closed

Add REDLINE THEATER: an MCP-driven redline demo that proves itself #615

Open

This was referenced Sep 4, 2026

DocxDiff comparison from #693 hangs in the browser build but not natively #696

Closed

Warm the WASM comparison engine before the first real comparison #697

Merged

Navigation Menu

DocxDiff: Word-parity arrangement of unrelated regions, surplus table cells, imported paired styles, attribute-wise default spacing - #693

DocxDiff: Word-parity arrangement of unrelated regions, surplus table cells, imported paired styles, attribute-wise default spacing#693
JSv4 merged 12 commits into
mainfrom
feat/diff-fidelity-round10

JSv4 commented Sep 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

JSv4 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What changed

How it was decided

Validation

What is left

Known exceptions

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JSv4 commented Sep 3, 2026 •

edited

Loading