BlackBox Testing from Doxygen

Black-Box Testing from Doxygen Documentation

Some teams do not want to expose production source code to external tools or services for automated test generation. This chapter describes a disciplined approach to writing black-box unit tests from Doxygen-documented headers, without reading the implementation.

Companion code lives in workshop/08_blackbox_testing/.

---

What Black-Box Testing Means Here

In this context, black-box means the test author works only from the public interface and its documentation. The implementation file is compiled and linked but never read. The test is a verification of the documented contract, not of the code that fulfils it.

This has three practical consequences:

• Tests cannot depend on internal state, private fields, or implementation order
• A passing test confirms that the implementation honours its documentation
• A failing test is unambiguous: either the implementation is wrong, or the documentation is wrong

---

Why Doxygen Is the Right Specification

Doxygen comments on a public header form a machine-readable, reviewable specification that lives in the same repository as the code. When every public method documents its preconditions, postconditions, and state transitions, the documentation contains everything needed to write a complete test suite.

The workflow is:

Developer writes                    Tester reads
public header.h  ────────────────▶  header + Doxygen HTML
with full Doxygen                   writes tests from docs only
                  ────────────────▶
                  compiled .o file  (implementation, not read)

The Doxygen HTML can be generated and handed to the tester as the specification document. The source file never needs to leave the team.

---

Writing Good Doxygen for Testable Contracts

A header that supports black-box testing must document more than parameter types. Each method needs:

• Preconditions: what state must the object and its inputs be in?
• Postconditions: what is guaranteed to be true after the call?
• State transitions: what state does the object move to?
• Edge cases and boundaries: what happens at limits?
• Error conditions: what observable effect does a failure produce?

Compare the two documentation styles below.

Insufficient: describes what, not when or why

/// Opens the valve.
void open();

A tester reading this knows the method exists. They do not know when it should be called, what state results, or what happens if it is called at the wrong time.

Sufficient: documents the contract completely

/// @brief Open the valve to allow fluid flow.
///
/// Calling open() while the regulator is in RegulatorState::Fault
/// has no effect. The valve position is unchanged.
///
/// @post isOpen() returns true if called in any state other than Fault.
virtual void open() = 0;

Now the tester can derive two tests: one confirming the valve opens in the normal case, and one confirming it does not open in the Fault state.

---

Deriving Tests from Documentation: The Process

For each documented item in the header, apply this three-step process:

Step 1: Identify a documentated clause

Read the Doxygen comment and find one specific, verifiable claim. One test per claim.

Doc clause: "Transitions from Idle to active regulation."
            [from PressureRegulator::start()]

Step 2: Express it as Arrange / Act / Assert

Arrange:  regulator is in Idle state (initial state after construction)
Act:      call start(), then update() with a sensor reading in the deadband
Assert:   state() returns Regulating

Step 3: Write the test and reference the clause

/// Doc: start(): "Transitions from Idle to active regulation."
TEST_F(PressureRegulatorTest, StartTransitionsFromIdleToRegulating)
{
    sensor.setReading(1000.0f); // within deadband
    regulator.start();
    regulator.update();
    EXPECT_EQ(regulator.state(), RegulatorState::Regulating);
}

The comment citing the documentation clause makes the test reviewable: a reviewer can check the claim in the header, read the test, and confirm they match, without understanding the implementation.

---

What to Test: A Checklist from the Header

When reading a Doxygen-documented header, work through this checklist to ensure full coverage of the documented contract.

Lifecycle

• Initial state after construction matches the documented default
• Each documented state transition occurs when the corresponding method is called
• Methods documented as having no effect in certain states produce no observable change

Boundaries

• Values at documented lower bounds behave as documented
• Values at documented upper bounds behave as documented
• Values just outside documented bounds behave as documented

Postconditions

• Every @post clause in the documentation has a corresponding test
• Observable state after each method call matches what the docs promise

Fault and error conditions

• Each documented fault or error condition can be triggered
• The documented recovery sequence (e.g. reset()) works as described
• Methods documented as no-ops in error states produce no changes

Return values

• Default return values match the documented defaults
• Return values after state changes match the documented postconditions

---

The Pimpl Idiom: Keeping Internals Hidden

To make the implementation genuinely inaccessible from the test, use the pimpl (pointer-to-implementation) idiom. The public header declares only the interface. All private state lives in a nested Impl struct defined in the .cpp file.

// In the public header (all the tester sees)
class PressureRegulator {
public:
    PressureRegulator(IPressureSensor&, IValve&, RegulatorConfig);
    ~PressureRegulator(); // destructor required by pimpl
    void start();
    // ... other public methods

private:
    struct Impl;                   // forward declaration only
    std::unique_ptr<Impl> impl_;   // tester cannot access Impl
};

// In the .cpp file (never read by the tester)
struct PressureRegulator::Impl {
    RegulatorState state = RegulatorState::Idle;
    float          lastPressure = 0.0f;
    // ... full private state
};

The tester cannot declare a variable of type Impl, cannot inspect its fields, and cannot observe anything the documentation does not expose through a public method.

---

Automated Test Generation with the Claude API

Writing tests manually from the documentation is practical for small headers. For larger APIs, the same workflow can be automated: run Doxygen to extract the contract, send it to an LLM API, and receive a complete GTest file in return. The implementation file is never involved.

The script generate_blackbox_tests.py in workshop/08_blackbox_testing/ supports two providers: Anthropic (Claude) and Mistral, selectable via a --provider flag.

How it works

pressure_regulator.h
        │
        ▼ doxygen (XML output)
  contract.xml
        │
        ▼ python parser
  plain-text contract
        │
        ▼ anthropic or mistral Python SDK
  generated_tests.cpp

The .cpp implementation file is never read, never parsed, and never sent to any API. Only the Doxygen-extracted documentation is transmitted.

Prerequisites

pip install anthropic mistralai
# doxygen must be on PATH: https://www.doxygen.nl

Only the package for the provider you intend to use is strictly required.

Usage

# Anthropic (default)
export ANTHROPIC_API_KEY=sk-ant-...
python generate_blackbox_tests.py

# Mistral
export MISTRAL_API_KEY=...
python generate_blackbox_tests.py --provider mistral

# Custom header, output, and model
python generate_blackbox_tests.py \
    --provider anthropic \
    --header path/to/my_component.h \
    --output my_component_tests.cpp \
    --model claude-sonnet-4-6

The script accepts four arguments:

Argument	Default	Description
--header	pressure_regulator.h	Path to the header file to document and test
--output	pressure_regulator_blackbox_tests.cpp	Path for the generated test file
--provider	anthropic	anthropic or mistral
--model	Provider default (see below)	Model name to use

Default models per provider:

Provider	Default model
anthropic	claude-sonnet-4-6
mistral	mistral-large-latest

Provider comparison

Both providers receive the same system prompt and contract. Claude and Mistral Large both handle structured technical prompts well. In practice the main reason to choose one over the other is which API key you already have, or a preference for keeping all AI usage within a single vendor.

What is sent to the API

The script sends a plain-text rendering of the Doxygen XML: class names, method briefs, parameter documentation, return values, postconditions, and state descriptions. It does not send file paths, raw source code, or anything outside the documented public interface. This is true for both providers.

Reviewing the output

The generated file is a starting point, not a final artefact. Before committing:

• Check that every documented clause has a corresponding test
• Verify that the test doubles match the interfaces declared in the header
• Compile and run the suite; fix any issues the generator introduced
• Apply the What to Test checklist to confirm coverage

---

Trade-offs of the Black-Box Approach

Benefit	Cost
Tests verify the documented contract, not the implementation	Gaps in documentation become gaps in test coverage
Implementation can change freely without breaking tests	Implementation-specific edge cases may go untested
Tests remain valid after a complete rewrite	More effort required to document the contract thoroughly
No source code exposure to external tools or testers	Test author cannot use code coverage as a completeness check

The most important implication of the last point: the quality of the test suite is bounded by the quality of the documentation. Incomplete or vague Doxygen comments produce incomplete tests. Writing good Doxygen is not optional when using this approach.

---

Coverage Without Reading the Code

Since branch coverage cannot be measured from the black-box perspective, use documentation coverage as the completeness metric instead:

• Every @brief, @param, @return, and @post clause has at least one test
• Every documented state transition is covered
• Every documented edge case, boundary, or fault condition is covered

A review pass over the header comparing it line by line against the test file is the practical way to assess this.

---

See also: Testing Processes, Test Architecture Patterns, Effective Use of Test Doubles