Skip to content

PDF Content Streams

Jump to bottom
Marc Jones edited this page Dec 22, 2025 · 1 revision

PDF Content Streams

Content streams are the heart of PDF rendering. This page documents their structure and how we parse/modify them.

References

What is a Content Stream?

A content stream is a sequence of operators and their operands that describe page content:

q                           % Save graphics state
1 0 0 1 72 720 cm           % Translate to (72, 720)
BT                          % Begin text
  /F1 12 Tf                 % Set font
  (Hello World) Tj          % Show text
ET                          % End text
Q                           % Restore graphics state

Key characteristics:

  • Postfix notation: operands come before operators
  • Whitespace separates tokens
  • Comments start with %
  • Operators are keywords (no leading /)
  • Names start with / (e.g., /F1)

Page Content Structure

Single Content Stream

/Page <<
  /Type /Page
  /Contents stream          % Single stream object
  /Resources <<...>>
>>

Multiple Content Streams

/Page <<
  /Type /Page
  /Contents [stream1 stream2 stream3]   % Array of streams
  /Resources <<...>>
>>

When multiple streams exist, they are concatenated for rendering (with implicit space between).

Getting Content in PdfSharpCore

// Access page content
var page = document.Pages[0];
var contents = page.Contents;

// Contents may have multiple elements
foreach (var content in contents.Elements)
{
    // Each is a PdfReference to a stream
}

Operator Categories

Graphics State Operators

Operator Purpose Our Handling
q Save graphics state Push to state stack
Q Restore graphics state Pop from state stack
cm Modify CTM Multiply with current matrix
w Set line width Track for paths
J Set line cap Ignore
j Set line join Ignore
M Set miter limit Ignore
d Set dash pattern Ignore
ri Set rendering intent Ignore
i Set flatness Ignore
gs Set graphics state dict Partial support

Color Operators

Operator Purpose
CS / cs Set color space (stroking/nonstroking)
SC / sc Set color
SCN / scn Set color (extended)
G / g Set gray level
RG / rg Set RGB color
K / k Set CMYK color

Text Operators

See PDF-Text-Operators for detailed documentation.

Operator Purpose
BT / ET Begin/end text object
Tf Set font
Td, TD, Tm, T* Position text
Tj, TJ, ', " Show text
Tc, Tw, Tz, TL, Tr, Ts Text state

Path Operators

Operator Purpose
m Move to
l Line to
c Curve to (cubic Bézier)
v, y Curve shortcuts
h Close subpath
re Rectangle
S Stroke
s Close and stroke
f, F Fill (nonzero winding)
f* Fill (even-odd)
B, B* Fill and stroke
b, b* Close, fill, and stroke
n End path (no-op)

XObject Operators

Operator Purpose
Do Invoke XObject (image or form)

Marked Content Operators

Operator Purpose
BMC Begin marked content
BDC Begin marked content with properties
EMC End marked content

Our ContentStreamParser

Overview

ContentStreamParser.cs parses content streams into structured PdfOperation objects:

public List<PdfOperation> Parse(byte[] contentBytes, double pageHeight)

Operation Types

public abstract class PdfOperation
{
    public Rect BoundingBox { get; }
    public bool IntersectsWith(Rect area);
}

public class TextOperation : PdfOperation
{
    public string Text { get; }
    public string OriginalOperator { get; }  // "Tj", "TJ", etc.
    public byte[] OriginalBytes { get; }
}

public class PathOperation : PdfOperation
{
    public List<PathSegment> Segments { get; }
    public bool IsFilled { get; }
    public bool IsStroked { get; }
}

public class ImageOperation : PdfOperation
{
    public string XObjectName { get; }
}

public class StateOperation : PdfOperation
{
    // q, Q, cm, etc.
}

State Tracking

The parser maintains:

private PdfGraphicsState _currentState;           // CTM, colors, line width
private Stack<PdfGraphicsState> _stateStack;      // For q/Q
private PdfTextState _textState;                   // Font, size, matrix

Bounding Box Calculation

For text, we use TextBoundsCalculator:

var bbox = _textBoundsCalculator.CalculateBounds(
    text,
    _textState,
    _currentState,
    pageHeight
);

ContentStreamBuilder

Rebuilds content streams after filtering operations:

public byte[] Build(List<PdfOperation> operations)

Serialization

private void SerializeOperation(PdfOperation op, StringBuilder sb)
{
    switch (op)
    {
        case TextOperation textOp:
            SerializeTextOperation(textOp, sb);
            break;
        case PathOperation pathOp:
            SerializePathOperation(pathOp, sb);
            break;
        // ...
    }
}

Text Serialization

private void SerializeTextOperation(TextOperation op, StringBuilder sb)
{
    switch (op.OriginalOperator)
    {
        case "Tj":
            sb.Append('(');
            sb.Append(EscapeString(op.Text));
            sb.Append(") Tj\n");
            break;
        case "TJ":
            // Rebuild TJ array
            SerializeTJArray(op, sb);
            break;
        // ...
    }
}

Stream Compression

Content streams are typically compressed:

Common Filters

Filter Description
/FlateDecode zlib compression (most common)
/LZWDecode LZW compression (legacy)
/ASCII85Decode ASCII encoding
/ASCIIHexDecode Hex encoding

Decompression in PdfSharpCore

// Get raw bytes (decompressed)
var stream = contentObject as PdfDictionary;
byte[] bytes = stream.Stream.Value;  // Already decompressed

Recompression

// When replacing content
var newContent = new PdfDictionary();
newContent.CreateStream(newBytes);
// PdfSharpCore handles compression

The Scrambled Order Problem

Symptom

Content stream order ≠ Reading order:

% Content stream order:
(W) Tj    % 'W' at x=200
(H) Tj    % 'H' at x=100
(o) Tj    % 'o' at x=150

Visual result: "HoW" (sorted by x position)

Why This Happens

  1. Optimization: PDF generators optimize for output efficiency
  2. Font changes: Text may be grouped by font
  3. Layer order: Background first, foreground last

Solution: Use PdfPig's Reading Order

PdfPig sorts letters by position to produce reading order:

var letters = page.Letters.OrderBy(l => l.GlyphRectangle.Top)
                          .ThenBy(l => l.GlyphRectangle.Left);

Our CharacterMatcher does similar sorting:

.OrderBy(l => pageHeight - l.GlyphRectangle.Top)  // Top to bottom
.ThenBy(l => l.GlyphRectangle.Left)               // Left to right

Form XObjects

What They Are

Reusable content streams, like a "subroutine":

% Page content
q
1 0 0 1 100 500 cm    % Position the form
/Form1 Do              % "Call" the form
Q

% Form XObject (separate stream)
/XObject <<
  /Type /XObject
  /Subtype /Form
  /BBox [0 0 200 100]
  /Resources <<...>>
  stream
    BT /F1 12 Tf (Form Text) Tj ET
  endstream
>>

Challenges for Redaction

  1. Coordinate transformation: Form has own coordinate space
  2. Resource inheritance: May use page resources
  3. Multiple invocations: Same form used multiple times
  4. Nested forms: Forms can invoke other forms

Current Limitation

Our parser doesn't currently handle Form XObjects. This is tracked in the implementation plan.

Inline Images

Images embedded directly in content stream:

BI                    % Begin inline image
  /W 100
  /H 100
  /CS /RGB
  /BPC 8
ID                    % Begin image data
[raw image bytes]
EI                    % End inline image

Parsing Challenge

The image data can contain any bytes, including sequences that look like operators. You must:

  1. Read the image dictionary to find dimensions and color space
  2. Calculate exact byte length
  3. Skip that many bytes

Current Limitation

We don't fully parse inline images. They pass through unchanged.

Practical Example: Parsing a Simple Page

Input content stream:

q
BT
/F1 12 Tf
100 700 Td
(Hello) Tj
50 0 Td
(World) Tj
ET
Q

Parsed operations:

[0] StateOperation { Operator = "q" }
[1] StateOperation { Operator = "BT" }
[2] TextStateOperation { Font = "F1", Size = 12 }
[3] TextOperation { Text = "Hello", BBox = (100, 92, 45, 12) }
[4] TextOperation { Text = "World", BBox = (150, 92, 42, 12) }
[5] StateOperation { Operator = "ET" }
[6] StateOperation { Operator = "Q" }

Note: Text state operators (Tf, Td) are absorbed into TextOperation context; only text-showing operators produce TextOperation.

Codebase Locations

File Purpose
Services/Redaction/ContentStreamParser.cs Parse content streams
Services/Redaction/ContentStreamBuilder.cs Rebuild content streams
Services/Redaction/PdfOperation.cs Operation data models
Services/Redaction/PdfGraphicsState.cs Graphics state tracking
Services/Redaction/PdfTextState.cs Text state tracking

Last updated: 2024-12-22

Clone this wiki locally