Letter.TextSequence is finer than words since 0.1.14: still a word grouping hint?

[ilCosmico]

Since 0.1.14 (#1241) TextSequence increments on Tj as well as TJ. I know the change is deliberate, and that 0.1.15 already fixed the TJ over-increment (#1298), so this is not about that.

BaseStreamProcessor.TextSequence is documented as a counter "used to determine if letters are likely to be in the same word/group", which "may correspond to the intended grouping of letters into words". We took that literally in a CAD library that imports PDF drawings: group letters by TextSequence, then run NearestNeighbourWordExtractor within each group.

Measured on one page of a CAD-exported PDF with nothing else changed (0.1.14 matches 0.1.15):

	0.1.13	0.1.15
page.Letters.Count	6783	6783
after DuplicateOverlappingTextProcessor.Get	6783	6783
words from NearestNeighbourWordExtractor.Instance	2221	2221
distinct TextSequence	1	6728

The file only uses Tj, one call per glyph, which is common for CAD exporters. Before 0.1.14 that meant one sequence for the whole page and grouping by it did nothing. Now every glyph gets its own, so partitioning by TextSequence before word extraction gives one word per character.

Two questions:

1. Is TextSequence still meant as a "likely same word" signal, or is it now just the rendering order of ShowText calls? If the latter, the doc comment above is worth rewording.
2. Is there a supported way to get a coarser "same text run" grouping? The old counter effectively marked TJ boundaries, and I cannot see how to tell the two operators apart from a Letter now.

[BobLd]

@ilCosmico I think this really depends on the pdf producer, unfortunately.

Is TextSequence still meant as a "likely same word" signal, or is it now just the rendering order of ShowText calls? If the latter, the doc comment above is worth rewording.

I believe this is now more rendering order of ShowText indeed

Is there a supported way to get a coarser "same text run" grouping? The old counter effectively marked TJ boundaries, and I cannot see how to tell the two operators apart from a Letter now.

I think you'd need to create your own implementation of BaseStreamProcessor (one possible issue is that ShowText is not virtual at the moment).

Also, the change in behaviour might have broken RenderGlyph

Long story short, properly using TextSequence might be trickier than it seems. Where you getting better results when doing "group letters by TextSequence, then run NearestNeighbourWordExtractor within each group"?

[ilCosmico]

Yes, and an 878 byte file makes it concrete. Three lines of text, same font, same page:

1. Hello World drawn as a single TJ
2. Hello World drawn as one Tj per glyph, each positioned with its own Td
3. Part and Number drawn as two separate TJ runs, butted together with no gap

NearestNeighbourWordExtractor.Instance, whole page against grouping by TextSequence first:

	0.1.13	0.1.15
distinct TextSequence	3	13
whole page	Hello, World, Hello, World, PartNumber	identical
grouped by sequence	Hello, World, Hello, World, Part, Number	Hello, World, H, e, l, l, o, W, o, r, l, d, Part, Number

Line 3 is where the grouping earned its place. Geometry alone merges the two runs into PartNumber, and the run boundary is the only thing keeping them apart. That is the better result you asked about, and it is why the extractor on its own is not a substitute.

Line 2 is what broke. On 0.1.13 it shared one sequence with the rest of the page, on 0.1.15 it is one sequence per glyph. Downstream that turns 6 text entities into 14, one per character on that line.

What leaves us stuck is that on 0.1.15 neither strategy gets all three lines right: whole page loses Part and Number, grouping loses Hello World. On 0.1.13 grouping got all three. Our real files behave the same way, this is just the smallest thing that shows it.

---

On RenderGlyph you were right, and it is measurable. A second small file draws a base letter plus a combining mark five times, varying only how the two glyphs are emitted. page.Letters on the official packages:

glyphs emitted as	0.1.13	0.1.14	0.1.15
(e) Tj then (´) Tj	combined	separate	separate
(e´) Tj	combined	combined	combined
[(e) 0 (´)] TJ	combined	separate	combined
total Letters	5	8	7

The guard is attachTo.TextSequence == TextSequence, so a mark can only attach when both glyphs come from the same show operation. fee5ce1f repaired the TJ row, the two separate Tj case is still open. When it fails the mark becomes its own Letter, sitting to the right of the base and 6 points wide against 13.3 for the base, and the base keeps its own box instead of being replaced by the composed letter.

page.Text and GetWords() are unaffected on that file, so only consumers that walk Letters see it. Worth noting that Diacritics_export.pdf cannot catch this: it uses precomposed U+00ED and U+00F1, 24 letters on both versions, so it never reaches that branch.

Happy to open a separate issue with the generator if you would rather track it there.

combining_separate_tj.pdf
tj-vs-TJ.pdf

[BobLd]

@ilCosmico thanks a lot for the analysis, do you mind opening an issue as I think we lost something good between 0.1.13 and 0.1.15.

Please attached the documents so that I can add unit tests too.

I will release the official 0.1.16 shortly (this weekend probably) and will work on your issue after.

EDIT: I had a look at your tj-vs-TJ.pdf and I believe PartNumber is more correct than Part Number. Still believe addressing your remarks is worth it.

[ilCosmico]

Both opened: #1401 for the TextSequence run boundaries, #1402 for the combining marks. Both link the files already attached here, so you have the repros for the unit tests.

Thanks for looking at the sample and pushing back on the third line, that saved me from building the case on a bad example :)