Proposal: decide word boundaries in docling-parse with a small learned model

[wittjeff]

Context: docling-parse#385, #358, #377, #383

Summary

7.22.1 introduced a 0.35 cap on word gaps. Since then, several issues and PRs have tried to fix how docling-parse divides glyphs into words: #358, #377, #383 and #385. Each fix adds or tunes a threshold on the gap size. Each fix repairs one class of lines and moves part of the error to a different class.

I propose a different method:

• A small model makes the decision at each ambiguous seam. It uses the geometry and the characters on each side.
• A few fixed rules handle the clear structural cases.
• A written convention defines the output for text that has no single correct answer, for example dot leaders and inline math.

This proposal does not replace #385. #385 is a clear improvement over 7.22.2 and must continue. This is a longer-term direction. I ask for feedback from the maintainers before I build it.

The problem: gap size alone cannot decide

@morten-lagabote posted a good example on #385: Innst. 240 S (2013–2014). This is justified Norwegian text with letter-spaced emphasis. The tracking is 1.65 pt per letter, and the tracked word contains no space characters. Results on the 70 tracked runs in the document:

Build	Whole words
7.20.0, 7.22.0	60 of 70
7.22.1, 7.22.2	2 of 70
#385 (9f94c75)	35 of 70

The first line of page 1 has three populations of gaps:

Gap	Size
Letters that are not tracked	≈ 0 pt
Letters in the tracked word Presidentskapet	1.65 pt
Spaces between words (justified)	2.41 pt

The letter gap is 0.69 of the space. If a threshold on gap ÷ space keeps this word whole, it also joins real words on other lines. The gate sweep on #385 showed the same result from the other direction: a gate at 0.6 of a space joins a dot leader, and a gate at 1.0 joins :, in maths.

The unit is also important. On #383 I showed that a gap in em separates seams that the current unit (the advances of the two neighbouring glyphs) cannot separate. But some lines have no geometric clue in any unit. In elsevier-00.pdf, a r t i c l e i n f o has a uniform tracking of 0.44 em and no space glyph. The geometry cannot show where article ends.

A reader settles these cases with vocabulary: Pr e s i den t is not Norwegian, and article info is two English words. A model can use the same evidence.

Proposal

1. Tag each seam

For each pair of adjacent glyphs on a text line, predict join or boundary. This is the same formulation as character-tagging word segmentation for Chinese, Japanese and Thai, with geometric features added.

The model only makes a decision at each seam. It cannot add, remove or change a character, so the text stays exact. A VLM or an OCR engine cannot give this guarantee.

Inputs for each seam:

• Geometry:

  • the gap in em (nominal_text_height)
  • the gap relative to the font space and to the explicit spaces on the line
  • the rank of the gap in the gap distribution of the line

• Producer intent:

  • an explicit space glyph (visible, suppressed or absent)
  • the raw TJ adjustment
  • the Tc and Tw values in effect
  • a flag for when get_space_width() used its flat 500 fallback

  The parser already reads Tc and Tw into the text state (char_spacing and word_spacing in pdf_states/text.h) and uses them to place glyphs. The cells do not keep them today. Tracking set with Tc is close to a direct label.

• Context: the font and size on each side, any font change, the Unicode script and category.

• Characters: approximately 8 characters on each side.

The model is a small character encoder (approximately 0.2 to 1 M parameters). A linear-chain CRF on top decodes one line at a time. Thus a tracked run gets one decision and does not get partial joins (rada, oriz, ー病 in the #385 sweep). The #383 measurement made approximately 2,000 seam decisions per page. At a few microseconds per seam on a CPU, this cost is small compared with layout analysis or TableFormer.

2. Keep a few fixed rules around the model

Use rules where the answer is clear and structural:

• Never put a boundary before a combining mark (Mn/Me), for example the Arabic sukun.
• Always put a boundary at a gap above approximately 1.5 em, at a change of baseline, or at a large change of font size. The model can never join across a table gutter.
• Find leader runs before the model (four or more identical punctuation glyphs at a constant pitch). Emit them by the convention below, and remove them from the model input.
• To discuss: always put a boundary at a visible explicit space glyph.

3. Write a convention for text that is not words

Much of the discussion on #377, #385 and #358 is about convention, not recognition. Are Θ=, α∈ and :, regressions? At present, the published groundtruth decides this implicitly, and nobody wrote down its rules. I propose that we agree these points in this thread:

• Dot leaders: one token, no token, or one token for each dot? For a TOC, the title and the page number are the useful content.
• Inline math: a boundary where TeX puts medium or thick glue (binary operators and relations), and a join at thin glue or less? Or no segmentation of math, and leave math to the formula model?
• CJK: join all seams except gutters. Put a boundary at a change between CJK and Latin text only if there is a gap.
• Punctuation: attach it to the word before it (Ref., hoeveelheden),).

The written convention can be reviewed. It also becomes the training labels. With thresholds, the convention exists only as a side effect of the tuning.

4. Fall back to geometry when the characters give no signal

For math fonts, GLYPH<n> text, mis-encoded text and scripts outside the training data, the model must not use the character features and must use geometry only. The target for these classes is "no worse than the current rules".

Expected effect by case

These are predictions, not measurements.

Case	Example	Expected result
Tracking in justified text	Presidentskapet (0.69 of a space)	Fixed: vocabulary, and the same Tc across lines
Wide tracking with no space glyph	a r t i c l e i n f o (0.44 em)	Fixed. Only vocabulary can find the boundary.
Per-glyph TJ corrections	Inf lammation (#377)	Fixed: the TJ value and vocabulary
Joins next to a wide glyph	Related Work (#383)	Fixed: the gap in em
Partial joins	rada, oriz, ー病	Fixed: one decision for each run (CRF)
Dot leaders	. . . . .	Rule plus convention. No gain from the model.
Inline math	Θ=, :,, TeX thin spaces	Convention. Geometry only. Must be no worse than now.
Display-tracked CJK	文 責 ： 事 務 室	Likely fixed. The risk is separate terms on one line.
Table gutters	Adjacent cells on one line	Rule: no join above approximately 1.5 em
Wrong characters	#362, #371, #394, #4492	Out of scope. These are decoding bugs. They must be fixed first, because they corrupt the character input.

Training and evaluation data

1. Space restoration on plain text. Remove the spaces from multilingual text, and train the character part of the model to restore them. The supply of this data has no practical limit.

2. Synthetic PDFs. Render known text with Chromium, LuaLaTeX and a word processor. Use random letter-spacing, justification, kerning, per-glyph TJ corrections, fonts without a space glyph, suppressed spaces, CJK, Arabic and maths. The labels are exact, and they follow the convention.

3. Real PDFs, for evaluation only:

   • the 940 regression pages
   • the PDFs from the issues above
   • Morten's document
   • tagged PDFs with ActualText

   The pdfium text layer is a weak reference. It is not reliable for justified CJK text or for TeX thin spaces (see #383).

The model must not train on the issue PDFs. They are the test of whether it generalizes.

Where the model can be

Option	For	Against
A. docling-parse exports seam features (opt-in, no change in behavior). docling or docling-ibm-models runs an optional model stage.	The same pattern as layout and TableFormer. The current rules stay the default and the fallback.	Users of docling-parse alone do not get the model.
B. Inference in docling-parse C++ (small embedded weights or ONNX)	All users get the model.	It adds an ML runtime to a C++ library that has none today.
C. A VLM at runtime	It already knows the words.	Too slow for every line. It reads pixels, so the text is no longer exact. It can still be useful offline, to label uncertain seams.

I recommend option A first. We can consider option B after a measured result.

Plan for a first measurement

1. Add an opt-in seam export: the gap, the em, Tc, Tw, TJ, the space-glyph state, the space-width fallback flag, a leader flag and a math-font flag.
2. Agree the convention (section 3) in this thread.
3. Make approximately 50 k synthetic lines, and train the model.
4. Report results by class (Latin, tracked text, leaders, inline math, CJK, RTL, no character signal) against 7.22.0, 7.22.2, #385 and #358.

Success criterion: better than all current builds on the recognition classes (tracking, TJ corrections, wide glyphs), and no worse on leaders, math and text with no character signal.

Limits

• Determinism. The weights must be pinned and the inference must be deterministic, so that the regression groundtruth stays stable.
• Identifiers. Vocabulary must not override strong geometric evidence in part numbers or codes (ID A 12). The CRF and the explicit-space features help, but this needs a specific test.
• Character bugs. The model cannot repair wrong or missing characters (#362, #371, #394).
• Dependency. Option A adds a model download for users who enable the stage.

Questions for the maintainers

1. Do you accept an optional learned stage for word segmentation, if the current rules stay the default?
2. Where must it be: docling, docling-ibm-models or docling-parse?
3. Can we agree the convention for leaders, inline math, CJK and punctuation in this thread?
4. Does docling-parse accept an opt-in seam feature export that does not change any output?

Thanks to @seanphopkins for the sweeps and traces on #385, @morten-lagabote for the Norwegian data, and @rasokiwayami for the verification on #377.

[wittjeff]

Follow-up: score whole candidate words, not only seams

Section 1 above describes a seam classifier with approximately 8 characters of context on each side. That description does not state the main idea clearly. A reader does not judge each gap alone. A reader recognizes the words, and a confident word makes the reader tolerant of irregular spacing inside it. I propose to make this part of the design, not something that the model must learn by chance.

Formulation. Score each candidate segmentation of a line with two independent terms, and find the best one with a Viterbi search over a lattice of candidate chunks (up to approximately 30 characters):

score(segmentation) =  Σ over words   log P_lex(word | neighbouring words)
                     + Σ over seams   log P_geo(gap | join or boundary, line statistics)

• P_lex is an open-vocabulary character language model. It scores a whole chunk as a word in its context. It is not a dictionary: a dictionary misses compounds, names and technical terms. A character model scores Presidentskapet as plausible from parts such as president, skap and et, even if it never saw the full word.
• P_geo is the geometric model from section 1: the gap in em, the explicit spaces, TJ, Tc and Tw.

Example: @morten-lagabote's tight line (letter gap 0.69 of the line's space):

Candidate	Lexical score	Geometric score	Result
Presidentskapet viser til	high: real words	medium: 14 joins at 0.69 of a space are unusual but possible	selected
Pr e s i den t skape t viser til	very low: Pr, e, s, i are not words	medium: the splits match the gap size	rejected

Consequences:

1. The tolerance comes from the score. A strong word lets the search accept irregular spacing. When the gap evidence is strong (2 em, or a visible space), the geometric term dominates, and vocabulary cannot override it.
2. Most of the fallback in section 4 needs no switch. For math, GLYPH<n> text or an unknown script, the character model has no preference. Its scores are flat across the candidates, so geometry decides. I would still flag these classes, to evaluate them separately.
3. Text that is good either way is decided by the gap. For notable against no table, the lexical term is close to neutral.
4. No fixed context window. Long words such as representantforslaget (21 characters, same document) are scored as a whole.

Implementation. For version 1, I recommend two separate models: a multilingual character LM of a few MB, and the small geometric model. One weight balances the two terms. Each decision then shows its lexical and geometric scores, so a maintainer can see why a line was split. The LM trains on plain text, and the geometric model and the weight train on synthetic PDFs. A neural semi-CRF that scores spans and seams together can come later, if it measures better.

Sections 2 and 3 and the four questions do not change. The convention questions now concern only the geometric term, because the lexical term is flat in math.

[wittjeff]

And for full disclosure I'll note that we'll be experimenting with models having deeper context, as I think that's needed to deal with real-world variability even just within lines.

[seanphopkins]

Thank you, @wittjeff, for the kind mention, and for taking the time to turn what we found on docling-parse#385 into a proposal. I can't judge the model design, but a written convention for dot leaders, inline math and CJK would help anyone reading these tables later, so I'm glad that is part of it.

This is my first attempt to contribute to an open source project. I'm a clinical pharmacist, and I came to Docling because I rely on it to read drug monographs. Finding that a table cell was wrong, tracing why, and having it taken seriously by people who know the code has been rewarding. It is a different way of contributing from what I'm used to, and I'm grateful for how welcoming this community has been.