Skip to content

Progressive Testing Strategy

Jump to bottom
Marc Jones edited this page Dec 24, 2025 · 1 revision

Progressive Testing Strategy

Systematic approach to testing PDF redaction: start simple, expand incrementally

Philosophy

Start Small, Build Up

Don't jump directly to complex PDFs. Build confidence progressively:

  1. Simple, known-good PDFs (demo files)
  2. Structural validation (file integrity)
  3. Feature-specific testing (content streams, fonts)
  4. Real-world documents (complex forms)
  5. Stress testing (large corpus)

Why Progressive Testing?

Problem with jumping to complex cases:

  • Hard to debug when basics don't work
  • Unclear if issue is fundamental or edge case
  • Wastes time on wrong problem

Benefits of progressive approach:

  • Validates fundamentals first
  • Isolates issues to specific features
  • Builds systematic understanding
  • Catches regressions early

6-Phase Testing Plan

Phase 0: Core Functionality Validation

Goal: Prove redaction works at all

Test PDFs: Your demo files (known-good, text-rich)

  • Simple PDFs with actual text content
  • Controlled, minimal complexity
  • Easy to validate results

Test Strategy:

  1. Redact known text (e.g., "CONFIDENTIAL")
  2. Verify with external tool (pdftotext)
  3. Validate file size decreases
  4. Check visual black box appears

Success Criteria:

  • pdftotext output.pdf | grep "REDACTED_TEXT" returns nothing
  • Output PDF is smaller than input
  • Output PDF is valid (opens without errors)

Time: 5 minutes

Exit Criteria: Must pass before proceeding to Phase 1

Phase 1: Structural Validation

Goal: Ensure redaction doesn't corrupt PDF structure

Test Categories:

  • 6.1 File structure (68 PDFs)
  • 6.7 Metadata (414 PDFs - sample 20)

Test Strategy:

  1. Redact arbitrary areas (may not contain text)
  2. Validate output PDF structure is intact
  3. Check headers, trailers, xref tables not corrupted
  4. Verify metadata preserved or safely modified

Success Criteria:

  • 95% of PDFs process without errors
  • Output PDFs are valid
  • No structural corruption

Time: 30 minutes

Exit Criteria: High pass rate on structural tests

Phase 2: Content Stream Testing

Goal: Validate content stream parsing and rebuilding

Test Categories:

  • 6.2 Graphics (58 PDFs)
  • Focus on 6.2.10 Content streams

Test Strategy:

  1. Test content stream parsing doesn't crash
  2. Validate rebuilt content streams render correctly
  3. Check for visual artifacts after redaction
  4. Verify text extraction still works for non-redacted text

Success Criteria:

  • 90% of PDFs process successfully
  • No visual corruption of non-redacted content
  • Content streams parse and rebuild correctly

Time: 1 hour

Exit Criteria: Content stream handling is stable

Phase 3: Font Testing

Goal: Validate font handling during redaction

Test Categories:

  • 6.3 Fonts (15 PDFs)
  • All subcategories (General, CIDFonts, subsets, metrics)

Test Strategy:

  1. Verify font information is preserved
  2. Test that non-redacted text uses correct fonts
  3. Validate font embedding remains intact
  4. Check font metrics are correct

Success Criteria:

  • 95% of PDFs process successfully
  • No font corruption in output
  • Non-redacted text renders in correct font/size
  • No "tofu" (missing glyph boxes)

Time: 2 hours

Exit Criteria: Font handling is correct, no font-related bugs

Phase 4: Form Testing

Goal: Validate form field redaction

Test Categories:

  • 6.9 Interactive Forms (4 PDFs)

Test Strategy:

  1. Test redacting form field text
  2. Verify other fields remain intact
  3. Check form structure not corrupted
  4. Test with real-world forms (Birth Certificate, etc.)

Success Criteria:

  • 90% of form PDFs process successfully
  • Form field text can be redacted
  • Form structure preserved
  • Real-world forms redact correctly

Time: 1 hour

Exit Criteria: Can reliably redact form documents

Phase 5: Edge Case Testing

Goal: Handle transparency, annotations, and other edge cases

Test Categories:

  • 6.4 Transparency (4 PDFs)
  • 6.5 Annotations (2 PDFs)
  • 6.6 Actions (3 PDFs)

Test Strategy:

  1. Test PDFs with transparency groups
  2. Distinguish annotations from content
  3. Ensure interactive features not corrupted
  4. Validate redaction boxes appear correctly

Success Criteria:

  • 80% of edge case PDFs process successfully
  • Transparency handled correctly
  • Annotations preserved or safely removed

Time: 2 hours

Exit Criteria: Edge cases handled gracefully

Phase 6: Stress Testing

Goal: Validate at scale, catch unexpected issues

Test Strategy:

  1. Test 500-1000 random PDFs from full corpus
  2. Look for crashes, hangs, memory leaks
  3. Validate performance is acceptable
  4. Test diverse PDF features simultaneously

Success Criteria:

  • 85% of corpus processes successfully
  • No crashes or data corruption
  • Performance acceptable (<5 seconds per page average)
  • Known failures are documented

Time: 4+ hours (can run overnight)

Exit Criteria: Ready for production use

Testing Tools

External Validation

Always use external tools to validate redaction:

pdftotext (poppler-utils):

pdftotext output.pdf - | grep "REDACTED_TEXT"
# Should return nothing if redaction worked

pdfinfo (poppler-utils):

pdfinfo output.pdf
# Validate PDF metadata and structure

veraPDF (validation):

verapdf output.pdf
# Check PDF/A compliance

Internal Tests

Use automated test suites:

  • Unit tests (220+ tests)
  • Integration tests (74+ tests)
  • GUI simulation tests (8+ tests)

Automation Scripts

Use provided scripts for corpus testing:

  • test-category.sh - Test specific category
  • validate-all-categories.sh - Test all categories

Success Metrics

Pass Rate Thresholds

Phase Minimum Pass Rate Ideal Pass Rate
Phase 0 100% 100%
Phase 1 95% 98%
Phase 2 90% 95%
Phase 3 95% 98%
Phase 4 90% 95%
Phase 5 80% 90%
Phase 6 85% 90%

What "Pass" Means

A test passes if:

  • ✅ Code doesn't crash or hang
  • ✅ Output PDF is valid (opens without errors)
  • ✅ Redacted text is removed (verified externally)
  • ✅ Non-redacted content is intact
  • ✅ File size decreases (content removed, not just hidden)

A test fails if:

  • ❌ Code crashes or hangs
  • ❌ Output PDF is corrupted
  • ❌ Redacted text still extractable
  • ❌ Non-redacted content is corrupted
  • ❌ File size increases or stays same

Common Pitfalls

Don't Skip Phases

Temptation: Jump to complex PDFs (Birth Certificate)

Problem: When it fails, you don't know if it's a fundamental issue or edge case

Solution: Complete each phase before moving to next

Don't Trust Internal Logs Only

Temptation: Trust your own logging that says "redaction succeeded"

Problem: Code can lie, bugs can hide

Solution: Always verify with external tools (pdftotext)

Don't Test Only Text-Rich PDFs

Temptation: Only test PDFs with lots of text

Problem: Miss structural and edge case issues

Solution: Test across all categories, even those with minimal text

Don't Test Only *pass*.pdf Files

Temptation: Skip *fail*.pdf files entirely

Problem: Miss opportunities to test error handling

Solution: Test a few *fail*.pdf files to ensure graceful failure

Regression Testing

After fixing bugs, re-run all previous phases:

  1. Fix identified issue
  2. Re-run failing phase
  3. Re-run all earlier phases (ensure no regression)
  4. Continue to next phase if all pass

Example:

  • Phase 3 fails due to font bug
  • Fix font bug
  • Re-run Phase 3 → should pass now
  • Re-run Phases 0-2 → ensure fix didn't break basics
  • Proceed to Phase 4

Performance Testing

Track performance metrics during testing:

Metrics to track:

  • Time per page
  • Memory usage
  • File size reduction percentage
  • Success rate per category

Acceptable performance:

  • Simple PDF (1 page, 50 ops): <100ms
  • Complex PDF (1 page, 500 ops): <500ms
  • Form (10 pages): <5 seconds
  • Large document (100 pages): <1 minute

Documentation During Testing

Document findings as you test:

GitHub Issues: For bugs found

  • Create issue immediately when test fails
  • Include PDF that caused failure
  • Include error logs and stack traces

GitHub Discussions: For patterns and insights

  • "Phase 2 findings: content stream edge cases"
  • "Font handling observations"
  • Share test results and statistics

Wiki Updates: For learned knowledge

  • Update this page with new insights
  • Add new edge cases to relevant pages
  • Document workarounds and solutions

Related Pages

External Resources

Clone this wiki locally