Explore logical Unicode PDF encoding with exact ToUnicode

[Sovichea]

I am sharing an Enhanced Unicode PDF architecture developed for Typsastra. This is not a request for fpdf2 to adopt it, but a working reference that may be useful for future Unicode PDF encoding work, particularly as an alternative to the current per-glyph /ToUnicode mapping.

The implementation was motivated by Khmer support but is script-independent and has been tested with multilingual complex-script content.

Implementations

Typst adapter

Converts already-shaped text into source-ordered logical units:

• Fork: https://github.com/Sovichea/typst
• Tested revision:
  https://github.com/Sovichea/typst/tree/75202cf09a26a5ef5dfd0f26ab7a4fe007e1be39
• Logical-unit preparation:
  https://github.com/Sovichea/typst/blob/75202cf09a26a5ef5dfd0f26ab7a4fe007e1be39/crates/typst-pdf/src/text/logical.rs

Krilla PDF backend

Implements logical CID allocation, synthetic TrueType glyphs, exact /ToUnicode, font sharding, and logical-order TJ output:

• Fork: https://github.com/Sovichea/krilla
• Tested revision:
  https://github.com/Sovichea/krilla/tree/d05158cf3ebead248745f846d0397e84dfb9f2d0
• Architecture document:
  https://github.com/Sovichea/krilla/blob/d05158cf3ebead248745f846d0397e84dfb9f2d0/PDF_LOGICAL_UNITS.md

Architecture

The boundary between shaping and PDF generation is a source-ordered logical unit:

STRUCT LogicalUnit
    text: exact logical Unicode
    glyphs: already-shaped positioned glyphs
    visual_position: independent x and y position
END STRUCT

The contract is:

• logical text is authoritative for extraction and accessibility;
• glyphs are authoritative for appearance;
• units remain in source order;
• the PDF backend does not reshape the text.

flowchart LR
    A[Logical Unicode]
    B[Existing shaping engine]
    C[Positioned glyphs]
    D[Logical units]
    E[Semantic and visual key]
    F[Unique logical CID]
    G[Synthetic glyph]
    H[Exact ToUnicode]
    I[Logical-order TJ]
    J[PDF]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    F --> H
    G --> I
    H --> I
    I --> J

Loading

A CID is reused only when both the logical text and visual construction are identical:

semantic_key := exact Unicode + positioned glyph construction

IF semantic_key already exists
    reuse its CID
ELSE
    allocate a new CID
    create a synthetic composite glyph
    map the CID to exact Unicode through ToUnicode
END IF

Each synthetic glyph contains the unit's shaped source glyphs as positioned TrueType components. Logical CIDs are then emitted in source order, with TJ adjustments reproducing visual placement.

Unsupported units fall back to ordinary glyph output rather than failing the document.

Possible relevance to fpdf2

fpdf2's pipeline in fpdf/fonts.py already contains most of the surrounding machinery: HarfBuzz shaping with cluster tracking, Glyph records carrying source codepoint tuples, SubsetMap allocating per-glyph character codes, positioned TJ output, and fontTools subsetting with synthetic glyph construction (TTGlyphPen, already used for the .notdef fallback).

The smallest integration boundary would be to change the unit of identity from "glyph" to "logical unit" at the shaping-to-font boundary:

INPUT
    shaped glyph run from uharfbuzz
    source Unicode text

OUTPUT
    logical unit per source cluster
    unique logical CID per unit
    synthetic composite glyph per unit
    exact ToUnicode mapping per CID

This would not require replacing fpdf2's shaping engine or redesigning its layout. The main new code would be:

1. cluster-span reconstruction, so each unit carries the full source sequence (e.g. ក្ស as three codepoints, not a partial fragment);
2. CID allocation keyed by logical unit rather than by glyph ID in SubsetMap;
3. composite-glyph synthesis appended after fontTools subsetting;
4. /ToUnicode emitted from the unit's full source text.

fpdf2's existing per-glyph unicode tuple and SubsetMap already point in this direction, so the gap is smaller than in most PDF generators.

Test binary

A prebuilt Typst-compatible binary is available for testing:

https://github.com/Sovichea/typsastra/releases/tag/enhanced-unicode-v0.3.1

It supports Windows, Linux, and macOS. After downloading the appropriate archive:

typst compile input.typ output.pdf
pdftotext -enc UTF-8 output.pdf output.txt

Relevant fixture:

https://github.com/Sovichea/typsastra/blob/enhanced-unicode-v0.3.1/tests/fixtures/enhanced-unicode/unicode-selection.typ

There is no expectation that fpdf2 adopt this implementation. I am sharing it as a tested architecture that may provide useful ideas for exact Unicode /ToUnicode encoding without depending on /ActualText.

[andersonhc]

Thank you for sharing this thoughtful and detailed research. I’ll definitely keep it in mind and revisit it when we’re ready to refactor our text-rendering code.

fpdf2 has its roots in a legacy PHP library that could only handle simple text rendering. We’ve worked hard to develop the text-shaping capabilities available in fpdf2 today, while preserving the legacy rendering path - especially given the performance cost that text shaping introduces.

A refactoring of our text-shaping code is overdue, and I’ll definitely take your analysis into consideration when we undertake it.

[Sovichea]

Thanks, Anderson. That context makes a lot of sense, especially the need to preserve the legacy rendering path and avoid imposing shaping costs on simple text.

I’ll keep this as experimental research rather than proposing changes to the current architecture.

For reference, I also have an experimental fpdf2 branch, enhanced-unicode-v3, where I ported the logical-unit model far enough to test the idea against the existing pipeline. It includes source-ordered logical units, synthetic TrueType composites, /ToUnicode, v3 TJ serialization, font sharding, the v0.3.1 composite-width guard, and multilingual benchmark results:

Experimental enhanced-unicode-v3 branch

The experiment also helped identify the tradeoff we discussed: the current fragment-local integration is a reasonable low-impact compromise, while a fragment-spanning logical-run model would only become relevant if a future refactor needs stronger cross-fragment semantics or more aggressive PDF-size optimization.

No need to review it now, but it may be useful as a reference when the text-rendering refactor comes up.