Scrambled Content Stream Problem

Scrambled Content Stream Problem

This page documents a critical issue we discovered and solved: PDF content streams can contain text in any order.

The Problem

User Report

Redaction was failing for certain PDFs. The user could select text correctly (text selection worked), but redaction didn't remove the selected text.

Observed Behavior

CharacterMatcher logging showed:

CharacterMatcher: Searching for 'FIRST' in 150 candidate letters.
Candidates text: 'R PLEEQAUSEES TPR FIONRT OLNAMFEI RLASMT IANDADMLREE S...'
CharacterMatcher: Operation text NOT FOUND in candidates.

The expected text "PLEASE PRINT FIRST..." was scrambled!

Root Cause Analysis

The PDF content stream contained text in storage order (order written to file), not visual order (left-to-right reading).

# Content stream (simplified)
(R) Tj at x=150     # 'R' in "FIRST"
( ) Tj at x=155     # space
(P) Tj at x=10      # 'P' in "PLEASE"
(L) Tj at x=20      # 'L' in "PLEASE"
...

Each letter has a transformation matrix that positions it correctly on the page, but the order in the file is arbitrary.

Why This Happens

Optimization

PDF generators may:

1. Group by font: All Helvetica-Bold text first, then Helvetica regular
2. Group by color: All black text, then blue text
3. Group by layer: Background elements, then foreground
4. Process order: Order text was added during document creation

It's Valid PDF

The PDF specification allows any order. The viewer must use transformation matrices to position glyphs correctly.

The Solution

Key Insight

PdfPig returns letters sorted by reading order (top-to-bottom, left-to-right), not content stream order.

Our content stream operations were in storage order, but PdfPig letters were in reading order.

Fix: Sort Candidates by Position

// CharacterMatcher.cs - BEFORE (broken)
var candidateLetters = letters.Where(letter => /* spatial filter */).ToList();

// CharacterMatcher.cs - AFTER (working)
var candidateLetters = letters
    .Where(letter => /* spatial filter */)
    // CRITICAL: Sort by position to get reading order!
    .OrderBy(l => pageHeight - l.GlyphRectangle.Top)  // Top to bottom
    .ThenBy(l => l.GlyphRectangle.Left)               // Left to right
    .ToList();

Now candidates are in reading order, matching what we search for.

The Test Case

Problematic PDF

birth-certificate-request-scrambled.pdf - A public government form with scrambled content order.

Test

[Fact]
public void Redact_ScrambledContentStream_FindsWord()
{
    var pdfPath = "Resources/sample-pdfs/birth-certificate-request-scrambled.pdf";

    // This PDF has scrambled content stream order
    var textBefore = PdfTestHelpers.ExtractAllText(pdfPath);
    textBefore.Should().Contain("FIRST");

    // Redaction should still work
    var redactedPath = Path.Combine(_tempDir, "redacted.pdf");
    RedactionService.RedactText(pdfPath, redactedPath, "FIRST");

    var textAfter = PdfTestHelpers.ExtractAllText(redactedPath);
    textAfter.Should().NotContain("FIRST");
}

Lessons Learned

Lesson 1: Content Stream Order ≠ Visual Order

Never assume text appears in reading order in the content stream. Always sort by position when reconstructing reading order.

Lesson 2: Test with Real-World PDFs

Simple test PDFs generated by our code always have correct order. Real-world PDFs from various sources expose this issue.

Lesson 3: PdfPig Already Handles This

PdfPig's text extraction works because it sorts letters by position. We should follow the same approach.

Lesson 4: Log the Intermediate State

The debug logging in CharacterMatcher was crucial:

_logger.LogDebug("Candidates text: '{CandText}'", candidatesText);

This immediately showed the scrambled order.

Detection

How to identify if a PDF has scrambled content:

// Compare stream order vs position order
var streamOrder = GetLettersInStreamOrder(page);
var positionOrder = page.Letters
    .OrderBy(l => -l.GlyphRectangle.Top)
    .ThenBy(l => l.GlyphRectangle.Left)
    .ToList();

bool isScrambled = !streamOrder.SequenceEqual(positionOrder);

References

• PDF 32000-1:2008, Section 9.4 - Text objects can appear in any order
• PdfPig source code - See how they sort letters

---

Issue discovered: December 2024 Fix committed: dbdc7b5