Skip to content

Atomic Testing with veraPDF Corpus

Jump to bottom
Marc Jones edited this page Dec 24, 2025 · 1 revision

Atomic Testing with veraPDF Corpus

Using the veraPDF corpus correctly for PDF redaction testing

What veraPDF Actually Is

Based on research from the veraPDF project and PDF Association:

veraPDF is "a purpose-built, open source, file-format validator covering all PDF/A and PDF/UA parts and conformance levels" designed for digital preservation. It's the first complete open source PDF/A validator.

The veraPDF Philosophy: Atomic Testing

The key insight: veraPDF uses an "atomic test corpus, covering each clause in the PDF/A specifications".

What "atomic" means:

  • Each test file targets one specific clause in the PDF/A specification
  • Tests are synthetic (purpose-built) not real-world documents
  • Each file is designed to test one thing and one thing only
  • Tests include both valid (pass) and invalid (fail) variants

From the PDF Association article on veraPDF validation:

"A good test should be atomic if possible, examining the codes handling of a specific validation issue."

History of the Corpus

From veraPDF 0.14 announcement:

  • 2008: Isartor Test Suite created (323 atomic test files for PDF/A-1b)
  • Later: Bavaria and additional corpora added
  • veraPDF project: Added 112 atomic tests for PDF/A-2/3, 65 for PDF/A-2 specific requirements
  • Now: Comprehensive corpus covering all PDF/A specifications

How veraPDF Corpus is INTENDED to be Used

From veraPDF documentation and corpus README:

1. Validator Development

The corpus is designed to help developers build PDF/A validators:

  • Each test file exercises a specific clause (e.g., "6.1.2 File header")
  • Developers test if their validator correctly identifies pass vs fail cases
  • Example: Does your validator correctly reject a PDF with invalid header?

2. Comprehensive Coverage Testing

Run the validator against the entire corpus:

# Validate all files recursively
verapdf -r --flavour 1b corpus_directory/

# Process 1,500 files in ~1 minute 40 seconds

From veraPDF testing:

"The software took one minute and forty seconds to process one thousand and five hundred files"

3. Clause-by-Clause Validation

Test each specification section systematically:

  • 6.1 File structure: Does your code handle all file structure requirements?
  • 6.2 Graphics: Does your code handle all graphics requirements?
  • 6.3 Fonts: Does your code handle all font requirements?

How This Applies to PDF Redaction Testing

What We're Actually Testing

We're not building a PDF/A validator - we're building a PDF redactor.

Our questions are different:

veraPDF Question Our Question
"Does this PDF pass PDF/A validation?" "Can we redact content without corrupting the PDF?"
"Does our validator correctly identify clause 6.3.5 violations?" "Can we preserve fonts after redacting text?"
"Can we detect non-compliant PDFs?" "Can we process diverse PDF features without crashing?"

The Correct Way to Use veraPDF Corpus

✅ DO: Use for Feature Coverage Testing

Goal: Ensure redaction doesn't break PDF features

Approach: Atomic testing by category

Example Test:

Test: "Can we redact from PDFs with CIDFonts without corrupting them?"
Corpus: 6.3.3.2 CIDFonts/*.pdf files
Method:
  1. Take each CIDFont test file (pass and fail variants)
  2. Redact arbitrary area
  3. Verify output PDF is still valid (structure intact)
  4. Verify we didn't introduce NEW errors

Success: Output PDF has same validity as input

✅ DO: Use for Specification Compliance

Goal: Ensure redaction maintains PDF/A compliance

Example Test:

Test: "Does redaction break PDF/A-1b compliance?"
Corpus: PDF_A-1b/*-pass-*.pdf files
Method:
  1. Take valid PDF/A-1b file
  2. Redact content
  3. Run veraPDF validator on output
  4. Check: Does it still validate as PDF/A-1b?

Success: Valid input → Valid output

✅ DO: Use for Edge Case Discovery

Goal: Find features that break redaction

Example Test:

Test: "Which PDF features cause redaction to fail?"
Corpus: All categories
Method:
  1. Redact from each category systematically
  2. Track failures by category
  3. Identify problematic features

Analysis: "Redaction fails on 6.2.5 Form XObjects - investigate why"

❌ DON'T: Use for Content Redaction Validation

Problem: Most corpus PDFs have no text content

From our corpus analysis:

  • Many files are graphics-only
  • Many have 0-1 words total
  • Designed to test structure, not content

Wrong approach:

❌ Test: "Can we redact 'CONFIDENTIAL' from corpus files?"
Problem: Most files don't contain "CONFIDENTIAL" or any text!

Right approach:

✅ Test: "Can we redact text from PDFs that have text?"
Use: Your own demo PDFs with known text content

❌ DON'T: Expect High "Text Redaction" Pass Rate

Wrong expectation:

"90% of corpus files should successfully redact text"

Why wrong: Most files have no text to redact!

Right expectation:

"95% of corpus files should process without crashing, maintaining structural validity"

Revised Testing Strategy

Phase 0: Content Redaction (Use Demo PDFs)

Goal: Prove text redaction works

Test Set: Your demo PDFs (01_simple_original.pdf, etc.)

  • Text-rich documents
  • Known content
  • Real-world structure

Tests:

  • ✅ Can redact specific text
  • ✅ Text is removed (pdftotext verification)
  • ✅ File size decreases
  • ✅ Non-redacted content intact

Success: Both CLI and GUI successfully remove text

Phase 1: Atomic Feature Testing (Use veraPDF Corpus)

Goal: Ensure redaction doesn't break PDF features

Approach: Test each category atomically

Test 1.1: File Structure Integrity

Category: 6.1 File structure (68 files)

What we're testing: Can we redact without corrupting PDF structure?

Method:

for pdf in 6.1*/*-pass-*.pdf; do
  redact "$pdf" output.pdf
  validate_pdf_structure output.pdf
  # Success: Structure still valid
done

Success Criteria:

  • 95% process without errors
  • No structure corruption (headers, trailers, xref intact)
  • Invalid input → clearly reported error (don't crash)

Test 1.2: Graphics/Content Stream Handling

Category: 6.2 Graphics (58 files), especially 6.2.10 Content streams

What we're testing: Can we parse/rebuild content streams correctly?

Method:

for pdf in 6.2.10*/*-pass-*.pdf; do
  redact "$pdf" output.pdf
  compare_content_streams input.pdf output.pdf
  # Success: Content stream structure maintained
done

Success Criteria:

  • 90% process successfully
  • Content streams parse without errors
  • Rebuilt content streams render correctly
  • No visual corruption of non-redacted content

Test 1.3: Font Preservation ← CRITICAL

Category: 6.3 Fonts (15 files)

What we're testing: Do we preserve font information correctly?

Method:

for pdf in 6.3*/*-pass-*.pdf; do
  redact "$pdf" output.pdf
  compare_fonts input.pdf output.pdf
  # Success: Font dictionaries intact
done

Success Criteria:

  • 95% process successfully
  • Font information preserved
  • No font corruption errors
  • Text renders in correct fonts

Current Status: 🔴 FAILING - Issue #137 (font bug)

Test 1.4: Form Fields

Category: 6.9 Interactive Forms (4 files)

What we're testing: Can we handle form field structures?

Method:

for pdf in 6.9*/*-pass-*.pdf; do
  redact "$pdf" output.pdf
  validate_form_structure output.pdf
  # Success: Form structure intact
done

Success Criteria:

  • 90% process successfully
  • Form structure not corrupted
  • Can handle form XObjects

Phase 2: PDF/A Compliance Preservation

Goal: Ensure redaction maintains standards compliance

Test Set: All *-pass-*.pdf files from corpus

Method:

for pdf in */*-pass-*.pdf; do
  # 1. Verify input is valid PDF/A
  verapdf "$pdf" > input_validation.txt

  # 2. Redact
  redact "$pdf" output.pdf

  # 3. Verify output is still valid PDF/A
  verapdf output.pdf > output_validation.txt

  # 4. Compare
  if [ input_valid ] && [ output_invalid ]; then
    echo "REGRESSION: Redaction broke compliance"
  fi
done

Success Criteria:

  • Valid input → Valid output (95% of cases)
  • We don't introduce NEW compliance errors
  • Known issues are documented

Phase 3: Failure Mode Testing

Goal: Ensure we handle invalid PDFs gracefully

Test Set: *-fail-*.pdf files from corpus

Method:

for pdf in */*-fail-*.pdf; do
  redact "$pdf" output.pdf
  # Success: Doesn't crash, reports error clearly
done

Success Criteria:

  • No crashes (100%)
  • Clear error messages
  • Doesn't corrupt files further

Phase 4: Real-World Documents

Goal: Test with actual use cases

Test Set:

  • Birth Certificate form
  • Invoices
  • Government documents
  • Business contracts

Method: Manual testing with real documents

Success Criteria:

  • Can redact sensitive information
  • Documents remain usable
  • Compliance maintained if required

Test Metrics That Matter

For veraPDF Corpus Testing

Metric Target What It Means
Structural Integrity 95% PDFs don't get corrupted
Feature Support 90% Can handle diverse PDF features
Graceful Degradation 100% Never crashes
Compliance Preservation 95% Valid input → Valid output

For Content Redaction Testing

Metric Target What It Means
Text Removal 100% pdftotext can't find redacted text
Visual Confirmation 100% Black box appears
File Size Reduction 100% Content actually removed
Surrounding Preservation 100% Non-redacted content intact

Atomic Test Creation

For features not covered by veraPDF corpus, create our own atomic tests:

Example: Redaction-Specific Atomic Tests

Test: Multiple sequential redactions

File: test_sequential_redactions.pdf
Content: "AAA BBB CCC DDD"
Redact: "AAA", then "CCC"
Verify: "BBB DDD" remains, fonts preserved

Test: Overlapping redaction areas

File: test_overlapping_redactions.pdf
Content: "ABCDEFGH" at position X
Redact: (X, Y, 100, 50) then (X+25, Y, 100, 50)
Verify: Overlaps handled correctly

Test: Redaction at page boundaries

File: test_boundary_redaction.pdf
Content: Text at (0, 0) and (pageWidth, pageHeight)
Redact: Edge areas
Verify: No coordinate overflow

Implementation: Atomic Test Runner

Create test infrastructure that mirrors veraPDF approach:

[Theory]
[MemberData(nameof(GetCorpusFiles), "6.3 Fonts")]
public void RedactionPreservesFonts_Atomic(string corpusFile)
{
    // Atomic test: One file, one assertion

    // 1. Get baseline
    var inputFonts = ExtractFontInfo(corpusFile);

    // 2. Redact
    var output = RedactArbitraryArea(corpusFile);

    // 3. Verify atomic requirement
    var outputFonts = ExtractFontInfo(output);
    outputFonts.Should().BeEquivalentTo(inputFonts,
        "Redaction must preserve font information (PDF/A clause 6.3)");
}

Each category gets its own atomic test method testing ONE requirement.

Resources

Research Sources

Key Quotes

"veraPDF is accompanied by a comprehensive, atomic test corpus, covering each clause in the PDF/A specifications."

"A good test should be atomic if possible, examining the codes handling of a specific validation issue."

"The goal is to create standardized documents that deal with different aspects of PDF/A. The strategy is to then purposely introduce breaches to the specification."

Related Pages

Clone this wiki locally