Skip to content

Testing Strategies

Jump to bottom
Marc Jones edited this page Dec 30, 2025 · 2 revisions

Testing Strategies

This page documents our testing infrastructure, best practices, and lessons learned about testing PDF operations.

Test Framework

  • xUnit 2.5.3 - Test framework
  • FluentAssertions 6.12.0 - Readable assertions
  • Moq - Mocking for unit tests
  • Avalonia.Headless.XUnit - UI testing without display

Test Organization

PdfEditor.Tests/
├── Integration/           # End-to-end tests
│   ├── ComprehensiveRedactionTests.cs
│   ├── GuiRedactionSimulationTests.cs
│   ├── CoordinateConversionTests.cs
│   └── BatchProcessingTests.cs
├── Unit/                  # Isolated component tests
│   ├── CoordinateConverterTests.cs
│   ├── PdfOperationTests.cs
│   └── MainWindowViewModelTests.cs
├── UI/                    # UI integration tests
│   ├── HeadlessUITests.cs
│   └── ViewModelIntegrationTests.cs
├── Security/              # Security-focused tests
│   └── ContentRemovalVerificationTests.cs
└── Utilities/             # Test helpers
    ├── TestPdfGenerator.cs
    └── PdfTestHelpers.cs

Key Test Patterns

Pattern 1: Create-Modify-Verify

The standard pattern for redaction tests:

[Fact]
public void Redact_SimpleText_RemovesFromContentStream()
{
    // 1. CREATE: Generate a PDF with known content
    var pdfPath = Path.Combine(_tempDir, "test.pdf");
    TestPdfGenerator.CreateSimpleTextPdf(pdfPath, "Hello REDACTED World");

    // Verify text is present
    var textBefore = PdfTestHelpers.ExtractAllText(pdfPath);
    textBefore.Should().Contain("REDACTED");

    // 2. MODIFY: Perform redaction
    var redactedPath = Path.Combine(_tempDir, "redacted.pdf");
    RedactionService.RedactText(pdfPath, redactedPath, "REDACTED");

    // 3. VERIFY: Check text is removed
    var textAfter = PdfTestHelpers.ExtractAllText(redactedPath);
    textAfter.Should().NotContain("REDACTED",
        "Text must be REMOVED from PDF structure, not just hidden");
    textAfter.Should().Contain("Hello");
    textAfter.Should().Contain("World");
}

Pattern 2: External Tool Verification

Use external tools to verify true content removal:

[Fact]
public void Redact_VerifyWithPdftotext()
{
    // ... create and redact ...

    // Verify with pdftotext (independent extraction)
    var process = Process.Start(new ProcessStartInfo
    {
        FileName = "pdftotext",
        Arguments = $"\"{redactedPath}\" -",
        RedirectStandardOutput = true,
        UseShellExecute = false
    });

    var output = process.StandardOutput.ReadToEnd();
    process.WaitForExit();

    output.Should().NotContain("REDACTED",
        "pdftotext should not find redacted text");
}

Pattern 3: GUI Simulation Tests

Test the exact workflow users follow:

[Fact]
public void GuiSimulation_SelectAndRedact()
{
    // 1. Simulate loading document
    var viewModel = new MainWindowViewModel();
    await viewModel.LoadDocumentAsync(pdfPath);

    // 2. Simulate rendering at user's DPI
    var renderDpi = 150;
    var pageImage = viewModel.RenderPage(0, renderDpi);

    // 3. Simulate user selection (in image pixels)
    var selection = new Rect(100, 200, 50, 20);  // Image coordinates

    // 4. Perform redaction through ViewModel
    await viewModel.RedactSelectionAsync(selection, renderDpi);

    // 5. Verify
    var text = PdfTestHelpers.ExtractAllText(viewModel.CurrentDocument);
    text.Should().NotContain("expected_redacted_word");
}

TestPdfGenerator

Our utility for creating test PDFs with known content.

Available Methods

// Simple single-line text
TestPdfGenerator.CreateSimpleTextPdf(path, "Hello World");

// Multi-line text
TestPdfGenerator.CreateMultiLinePdf(path, new[] {
    "Line one",
    "Line two",
    "Line three"
});

// Text with specific positioning
TestPdfGenerator.CreatePositionedTextPdf(path, new[] {
    (100, 700, "Top text"),
    (100, 400, "Middle text"),
    (100, 100, "Bottom text")
});

// PDF with TJ operator (kerned text)
TestPdfGenerator.CreateKernedTextPdf(path, "WATER");  // Uses TJ

// PDF with specific font
TestPdfGenerator.CreatePdfWithFont(path, "Text", FontType.Helvetica);

Why Custom Generator?

  • Known content: Exact text positions and operators
  • Reproducible: Same PDF every time
  • Minimal: No unnecessary complexity
  • Controlled: Test specific operators/features

PdfTestHelpers

Utility methods for PDF inspection.

Text Extraction

// Extract all text from PDF
var text = PdfTestHelpers.ExtractAllText(pdfPath);

// Extract text from specific page
var pageText = PdfTestHelpers.ExtractPageText(pdfPath, pageNumber: 1);

// Get letter details
var letters = PdfTestHelpers.GetLetters(pdfPath, pageNumber: 1);
foreach (var letter in letters)
{
    Console.WriteLine($"'{letter.Value}' at ({letter.X}, {letter.Y})");
}

Content Stream Access

// Get raw content stream
var content = PdfTestHelpers.GetContentStream(pdfPath, pageNumber: 1);
var contentString = Encoding.UTF8.GetString(content);

// Check for specific operator
contentString.Should().Contain("Tj");

PDF Validation

// Verify PDF is valid
PdfTestHelpers.ValidatePdf(pdfPath).Should().BeTrue();

// Check page count
PdfTestHelpers.GetPageCount(pdfPath).Should().Be(3);

Testing Coordinate Conversions

Coordinates are a common source of bugs. Test explicitly:

[Theory]
[InlineData(0, 0, 792, 0)]           // Bottom-left corner
[InlineData(0, 792, 792, 792)]       // Top-left corner (PDF Y=792)
[InlineData(100, 100, 792, 692)]     // Arbitrary point
public void ConvertPdfToAvalonia_CorrectY(
    double pdfX, double pdfY, double pageHeight, double expectedAvaloniaY)
{
    var result = CoordinateConverter.PdfToAvalonia(pdfX, pdfY, pageHeight);

    result.Y.Should().BeApproximately(expectedAvaloniaY, 0.001);
}

Testing with Real-World PDFs

Collecting Problem PDFs

When a user reports a bug:

  1. Get a redacted sample (remove sensitive data)
  2. Document the issue
  3. Add to Resources/sample-pdfs/
  4. Create regression test

Sample PDF Documentation

Each sample PDF should have documentation:

### birth-certificate-request-scrambled.pdf

**Source**: Public government form (no sensitive data)
**Issue**: Letters encoded out of reading order
**Test Cases**:
- Redact "FIRST" - fails without position sorting
- Redact "MAKING" - middle of scrambled sequence

Real-World Test Pattern

[Fact]
public void RealWorld_BirthCertificate_RedactFIRST()
{
    var pdfPath = "Resources/sample-pdfs/birth-certificate-request-scrambled.pdf";

    // Verify precondition
    var textBefore = PdfTestHelpers.ExtractAllText(pdfPath);
    textBefore.Should().Contain("FIRST",
        "Test precondition: PDF should contain 'FIRST'");

    // Redact
    var redactedPath = Path.Combine(_tempDir, "redacted.pdf");
    RedactionService.RedactText(pdfPath, redactedPath, "FIRST");

    // Verify
    var textAfter = PdfTestHelpers.ExtractAllText(redactedPath);
    textAfter.Should().NotContain("FIRST");
}

UI Testing with Avalonia Headless

Setup

[assembly: AvaloniaTestApplication(typeof(TestAppBuilder))]

public class TestAppBuilder
{
    public static AppBuilder BuildAvaloniaApp() =>
        AppBuilder.Configure<App>()
            .UseHeadless(new AvaloniaHeadlessPlatformOptions());
}

Headless UI Test

[AvaloniaFact]
public async Task MainWindow_LoadDocument_ShowsPage()
{
    var window = new MainWindow();
    window.Show();

    // Simulate user action
    var viewModel = (MainWindowViewModel)window.DataContext;
    await viewModel.LoadDocumentAsync(testPdfPath);

    // Verify UI state
    viewModel.IsDocumentLoaded.Should().BeTrue();
    viewModel.PageCount.Should().BeGreaterThan(0);

    window.Close();
}

Test Categories

Use traits to categorize tests:

[Fact]
[Trait("Category", "Integration")]
[Trait("Feature", "Redaction")]
public void IntegrationTest() { }

[Fact]
[Trait("Category", "Unit")]
public void UnitTest() { }

Run specific categories:

dotnet test --filter "Category=Integration"
dotnet test --filter "Feature=Redaction"

Performance Testing

For operations that should be fast:

[Fact]
public void Parse_LargePdf_CompletesInReasonableTime()
{
    var stopwatch = Stopwatch.StartNew();

    // Parse a complex PDF
    var operations = parser.Parse(complexPdfContent, pageHeight);

    stopwatch.Stop();

    stopwatch.ElapsedMilliseconds.Should().BeLessThan(500,
        "Parsing should complete in under 500ms");
}

Pre-Check PDF Content Before Testing

CRITICAL: Before running any redaction test on a PDF, ALWAYS verify the PDF actually contains the content type you're testing. This applies especially to corpus tests.

The Problem

Running text redaction tests on PDFs with no text, or graphics redaction tests on PDFs with no graphics, wastes time and produces misleading results. This is a common mistake when iterating on corpus tests.

Pre-Check Pattern

[Fact]
public void Redact_TextFromCorpusPdf()
{
    // STEP 1: Pre-check content exists
    var text = PdfTestHelpers.ExtractAllText(pdfPath);

    if (string.IsNullOrWhiteSpace(text))
    {
        _output.WriteLine($"SKIP: {pdfPath} has no extractable text");
        return; // Or use Skip.If() with xUnit
    }

    // STEP 2: Only now proceed with redaction test
    var textToRedact = FindFirstWord(text);
    // ... perform redaction ...
}

Corpus Test Pre-Filtering

When testing against a corpus like veraPDF, pre-filter files:

public static IEnumerable<object[]> GetTextContainingPdfs()
{
    var corpusPath = "/path/to/corpus";

    foreach (var pdf in Directory.GetFiles(corpusPath, "*.pdf"))
    {
        var text = PdfTestHelpers.ExtractAllText(pdf);
        if (!string.IsNullOrWhiteSpace(text))
        {
            yield return new object[] { pdf };
        }
    }
}

[Theory]
[MemberData(nameof(GetTextContainingPdfs))]
public void Redact_CorpusPdfWithText(string pdfPath)
{
    // Now guaranteed to have text
}

Content Type Checks

Test Type Pre-Check Skip If
Text redaction ExtractAllText() Empty or whitespace
Image redaction Check for /XObject in resources No XObjects
Graphics redaction Parse content stream for path operators No paths
Form XObject redaction Check /XObject with /Subtype /Form No Form XObjects
PDF/A compliance Check for XMP metadata with pdfaid Not a PDF/A file

Utility Method

Add to PdfTestHelpers:

public static class PdfContentChecker
{
    public static bool HasExtractableText(string pdfPath)
    {
        var text = ExtractAllText(pdfPath);
        return !string.IsNullOrWhiteSpace(text);
    }

    public static bool HasFormXObjects(string pdfPath)
    {
        // Check resources for Form XObjects
        using var doc = PdfReader.Open(pdfPath);
        var page = doc.Pages[0];
        var xobjects = page.Resources?.Elements.GetDictionary("/XObject");
        // ... check for /Subtype /Form entries
    }

    public static string? GetFirstWord(string pdfPath)
    {
        var text = ExtractAllText(pdfPath);
        if (string.IsNullOrWhiteSpace(text)) return null;
        return text.Split(' ', StringSplitOptions.RemoveEmptyEntries).FirstOrDefault();
    }
}

Test Output for Skipped Files

Always log why a file was skipped:

_output.WriteLine($"SKIP: {Path.GetFileName(pdfPath)} - no extractable text");
_output.WriteLine($"SKIP: {Path.GetFileName(pdfPath)} - not a PDF/A file");
_output.WriteLine($"SKIP: {Path.GetFileName(pdfPath)} - no Form XObjects");

This makes test runs transparent and helps identify corpus coverage gaps.

Common Test Mistakes

Mistake 1: Testing Implementation, Not Behavior

// BAD: Tests implementation details
[Fact]
public void Parser_CallsTextBoundsCalculator()
{
    mockCalculator.Verify(x => x.Calculate(...), Times.Exactly(3));
}

// GOOD: Tests behavior
[Fact]
public void Parse_TextOperators_ReturnsCorrectBoundingBoxes()
{
    var operations = parser.Parse(content, pageHeight);
    operations[0].BoundingBox.X.Should().BeApproximately(100, 1);
}

Mistake 2: Insufficient Assertion Messages

// BAD: No context on failure
textAfter.Should().NotContain("SECRET");

// GOOD: Clear failure message
textAfter.Should().NotContain("SECRET",
    "Redacted text 'SECRET' must be removed from PDF structure, not just hidden. " +
    "If this fails, check that content stream is being modified, not just overlaid.");

Mistake 3: Not Testing Failure Cases

// Also test what happens when things go wrong
[Fact]
public void Redact_EmptySelection_DoesNotModifyDocument()
{
    var emptyRect = new Rect(0, 0, 0, 0);

    // Should not throw and should not modify
    var hash1 = GetFileHash(pdfPath);
    redactionService.RedactArea(page, emptyRect);
    var hash2 = GetFileHash(pdfPath);

    hash1.Should().Be(hash2, "Empty selection should not modify document");
}

Debugging Test Failures

Output Test Data

[Fact]
public void DebugFailingTest()
{
    var letters = PdfTestHelpers.GetLetters(pdfPath, 1);

    _output.WriteLine("Letters found:");
    foreach (var letter in letters)
    {
        _output.WriteLine($"  '{letter.Value}' at ({letter.X:F1}, {letter.Y:F1})");
    }

    // ... assertions ...
}

Save Intermediate Files

[Fact]
public void SaveIntermediateFiles()
{
    // Save for manual inspection
    var debugDir = Path.Combine(Path.GetTempPath(), "pdf-debug");
    Directory.CreateDirectory(debugDir);

    File.Copy(originalPath, Path.Combine(debugDir, "original.pdf"));
    File.Copy(redactedPath, Path.Combine(debugDir, "redacted.pdf"));

    _output.WriteLine($"Debug files saved to: {debugDir}");
}

Test Coverage Goals

Category Coverage Target
Coordinate conversion 100%
Text operators (Tj, TJ, ', ") 100%
Redaction pipeline 90%+
ViewModel commands 80%+
UI rendering 60%+

Run coverage:

dotnet test --collect:"XPlat Code Coverage"

Last updated: 2024-12-22

Clone this wiki locally