Generated Reports

After tests complete, the following files are written to the Reports folder inside bin\Debug\net10.0\ (configurable via ReportsFolderPath).

Output Files Summary

File	Purpose	Contains Diagrams?	Machine-Readable?	Always Generated?
Specifications.html	Feature specifications with embedded sequence diagrams	Yes	No	Yes
Specifications.yml	Machine-readable scenario specifications	No	Yes (YAML)	Yes
TestRunReport.html	Full test run report with diagrams, CI metadata, failure details	Yes	No	Yes
TestRunReport.json	Machine-readable test run results with PlantUML diagrams and HTTP interactions	Yes (PlantUML source)	Yes (JSON)	Yes
ComponentDiagram.html	C4-style component diagram showing service relationships	Yes	No	When GenerateComponentDiagram = true
CiSummary.md	CI-friendly markdown summary with failure details	Inline PlantUML	Partially	When WriteCiSummary = true

Data format: The .yml extension is the default for Specifications and TestRunReport data files. Set SpecificationsDataFormat or TestRunReportDataFormat to ReportDataFormat.Json to produce .json files instead.

File names: All file names are configurable via HtmlSpecificationsFileName, HtmlTestRunReportFileName, and YamlSpecificationsFileName on ReportConfigurationOptions.

---

Specifications.yml

A YAML specification document listing all features and scenarios. The ReqNRoll and BDDfy adapters additionally include the BDD steps (Given/When/Then/And/But) for each scenario.

Specifications.html

An HTML specification file with embedded sequence diagrams for each test scenario. Features and scenarios are grouped hierarchically.

Behaviour: If any test in the run has failed, this file is written as blank. This prevents publishing stale or misleading specifications when the test suite is not fully passing.

TestRunReport.html

A full test run report including:

• Execution summary — Total pass/fail/skip counts and run duration
• Sequence diagrams — Embedded for each scenario
• BDD steps — Included when using ReqNRoll or BDDfy
• Failure details — Error messages and stack traces for failed tests

TestRunReport.json (Machine-Readable Test Run Data)

A machine-readable structured data file containing the complete test run results, including PlantUML diagram source and HTTP interaction traces for each scenario. This file is designed for automated debugging, CI tooling, and programmatic analysis.

Data format: The default format is JSON (TestRunReportDataFormat = DataFormat.Json). Set to DataFormat.Yaml or DataFormat.Xml to produce .yml or .xml files instead.

Schema

The top-level structure contains the test run time window and a list of features with nested scenarios:

{
  "startTime": "2026-01-01T10:00:00Z",
  "endTime": "2026-01-01T10:05:00Z",
  "features": [
    {
      "name": "Order Management",
      "endpoint": "/api/orders",
      "description": "...",
      "labels": ["api"],
      "scenarios": [
        {
          "id": "test-runtime-id",
          "name": "Place order successfully",
          "result": "Passed",
          "durationSeconds": 1.234,
          "isHappyPath": true,
          "errorMessage": null,
          "errorStackTrace": null,
          "labels": ["smoke"],
          "categories": ["Integration"],
          "steps": [
            {
              "keyword": "Given",
              "text": "a valid order request",
              "status": "Passed",
              "durationSeconds": 0.1,
              "subSteps": []
            }
          ],
          "diagrams": [
            "@startuml\nTestClient -> OrderService : POST /api/orders\nOrderService --> TestClient : 201 Created\n@enduml"
          ],
          "httpInteractions": [
            {
              "type": "Request",
              "method": "POST",
              "uri": "https://api.example.com/orders",
              "serviceName": "OrderService",
              "callerName": "TestClient",
              "content": "{\"item\":\"widget\",\"quantity\":1}",
              "headers": [
                { "key": "Content-Type", "value": "application/json" }
              ],
              "statusCode": null,
              "traceId": "guid",
              "requestResponseId": "guid",
              "timestamp": "2026-01-01T10:00:01.000Z"
            },
            {
              "type": "Response",
              "method": "POST",
              "uri": "https://api.example.com/orders",
              "serviceName": "OrderService",
              "callerName": "TestClient",
              "content": "{\"id\":42}",
              "headers": [
                { "key": "Content-Type", "value": "application/json" }
              ],
              "statusCode": "Created",
              "traceId": "guid",
              "requestResponseId": "guid",
              "timestamp": "2026-01-01T10:00:01.050Z"
            }
          ]
        }
      ]
    }
  ]
}

Key Fields

Field	Description
diagrams	Array of PlantUML source text strings for this scenario. A scenario may have multiple diagrams (e.g. when using focused views). Empty array when no diagrams are available.
httpInteractions	Array of tracked HTTP request/response log entries for this scenario, in chronological order. Each entry includes the full method, URI, headers, content body, service names, status code, and timestamps.
traceId	Groups a request and its corresponding response together.
requestResponseId	Unique identifier for each individual log entry.

Note: The diagrams and httpInteractions fields are only present when diagram/tracking data is available (i.e. when CreateStandardReportsWithDiagrams is used). When calling GenerateTestRunReportData directly without passing diagrams or logs, these fields are omitted.

Report Features

Both HTML reports include:

• Search — A search bar that filters features and scenarios by keyword. Supports quoted phrases for exact matching. Supports @tag syntax for Cucumber-style tag expression filtering (e.g. @smoke and @api, not @flaky, (@ui or @api) and not @slow).
• Happy path filter — A toggle to show only scenarios marked as happy path.
• Click-to-reveal PlantUML — Click any diagram image to reveal the raw PlantUML code behind it.
• Feature grouping — Tests are grouped by feature (test class / story title / feature file).
• Context menu — Right-click any diagram (SVG, flame chart, or call tree) for copy/save options including PNG, SVG, and diagram source. See Inline SVG Rendering for details.
• Status filter — Toggle buttons for each ExecutionResult value to show/hide scenarios by status.
• Category filter — When scenarios have categories, filter by category with AND/OR toggle (LightBDD, ReqNRoll).
• Pie chart — A donut chart showing the pass/fail/skip/bypass breakdown with a pass rate percentage in the centre (TestRunReport only).
• Duration columns — Feature summary table includes Duration, Avg, and Longest columns when scenario durations are available.
• CI metadata — When running in CI (GitHub Actions or Azure DevOps), the report displays build number, branch, commit SHA, and a link to the pipeline run.
• Step numbering — Configurable hierarchical step numbers (e.g. 1., 2., 2.1.) via SpecificationsShowStepNumbers and FeaturesReportShowStepNumbers.
• Custom branding — Inject custom CSS, favicon, and logo HTML via ReportConfigurationOptions.
• DocString rendering — Steps with a DocString property render a <pre> code block below the step text.
• Rules grouping — Scenarios with a Rule property are grouped under a collapsible rule heading.
• Examples / Scenario Outline tables — Scenarios sharing an OutlineId are rendered as examples tables with parameter columns and per-row status icons.

Execution Result Statuses

Each scenario and step is assigned an ExecutionResult status. The report renders each status with a colour-coded circular icon. See Framework Status Mappings below for full details on what each status renders as, its tooltip text, and which native framework statuses produce it.

Status	Icon	Colour	Meaning
Passed	✓	Green (#1daf26)	Ran and succeeded
Failed	✗	Red (#cc0000)	Assertion or runtime failure
Skipped	Ø	Grey (#949494)	Deliberately excluded (attribute, filter, inconclusive)
Bypassed	↷	Blue (#2e7bff)	Step body ran, developer opted out; execution continues (LightBDD only)
SkippedAfterFailure	!	Yellow (#fbc800)	Cascading non-execution after a prior step failure

Combined icons: When a parent step has Passed status but contains descendant sub-steps with Bypassed or Skipped status, the parent's icon changes to reflect this:

Descendant statuses	Parent icon	Background	Meaning
Any Bypassed (no Skipped)	✓	Blue (#2e7bff)	Passed overall, but some sub-steps were bypassed
Any Skipped (regardless of Bypassed)	✓	Grey (#949494)	Passed overall, but some sub-steps were skipped

This bubbles up recursively through all ancestor levels. Skipped takes priority over Bypassed because skipping is a bigger deal — it is cascading and prevents execution of subsequent steps, whereas bypassed steps still allow subsequent steps to run. Note that SkippedAfterFailure does not trigger this behaviour (it has its own distinct status).

Framework Status Mappings

Each framework adapter maps its native statuses to TTD's ExecutionResult enum. The tables below show which native statuses from each framework produce each TTD status, split by scenario level and step level.

Scenario-Level Mappings

These mappings determine the overall status icon shown next to each scenario in the report. Every framework provides scenario-level statuses.

TTD Status	Icon	Colour	Description	xUnit v2	xUnit v3	NUnit 4	MSTest	TUnit	BDDfy	ReqNRoll	LightBDD
Passed	✓	Green	All assertions passed	ITestPassed	Passed	Passed, Warning	Passed	Passed	Passed	OK	Passed
Failed	✗	Red	An assertion or runtime failure occurred	ITestFailed	Failed	Failed	Failed, Error, Timeout, Aborted	Failed, Timeout	Failed	TestError, BindingError	Failed
Skipped	Ø	Grey	Either the entire test did not run (e.g. a skip attribute or filter excluded it), or a step was skipped at runtime which also prevented all subsequent steps from executing	ITestSkipped	Skipped, NotRun	Skipped, Inconclusive	Inconclusive, InProgress, NotRunnable, Unknown	Skipped, Cancelled	Inconclusive, NotImplemented, NotExecuted	UndefinedStep, StepDefinitionPending, Skipped	Ignored, NotRun
Bypassed	↷	Blue	Some or all of the logic in a step was intentionally skipped over at runtime without preventing execution of subsequent steps	—	—	—	—	—	—	—	Bypassed

Note: SkippedAfterFailure does not apply at scenario level — it is a step-level status only. Bypassed is exclusive to LightBDD.

Step-Level Mappings

These mappings determine the status icon shown next to each BDD step (Given/When/Then) within a scenario. Only BDDfy, ReqNRoll, and LightBDD capture step-level statuses; the other frameworks have no step concept.

TTD Status	Icon	Colour	Description	BDDfy	ReqNRoll	LightBDD
Passed	✓	Green	All assertions in this step passed	Passed	OK	Passed
Failed	✗	Red	This step threw an exception or an assertion failed	Failed	TestError, BindingError	Failed
Skipped	Ø	Grey	This step did not execute because it was intentionally skipped, either at the scenario level, or at the step level. In the latter case the skip also prevented execution of subsequent steps	Inconclusive, NotImplemented, NotExecuted¹	UndefinedStep, StepDefinitionPending, Skipped³	Ignored⁵, NotRun⁶
Bypassed	↷	Blue	Some or all of the logic in this step was intentionally skipped over without preventing execution of subsequent steps	—	—	Bypassed
SkippedAfterFailure	!	Yellow	This step was never reached because an earlier step failed	NotExecuted²	Skipped⁴	Ignored⁷, NotRun⁸

BDDfy footnotes — BDDfy's NotExecuted maps to different statuses depending on whether a prior step failed:

1. NotExecuted → Skipped: When no prior step in the scenario has failed
2. NotExecuted → SkippedAfterFailure: When a prior step in the same scenario has already failed

ReqNRoll footnotes — ReqNRoll's Skipped maps to different statuses depending on whether a prior step failed:

3. Skipped → Skipped: When no prior step in the scenario has failed
4. Skipped → SkippedAfterFailure: When a prior step in the same scenario has already failed

LightBDD footnotes — LightBDD's Ignored and NotRun map to different statuses depending on both scenario-level skip state and prior step failure:

5. Ignored → Skipped: When the entire scenario is skipped, or when no prior step has failed
6. NotRun → Skipped: Only when the entire scenario was skipped at scenario level
7. Ignored → SkippedAfterFailure: When the scenario was not skipped and a prior step has already failed
8. NotRun → SkippedAfterFailure: When the scenario was not skipped (i.e. a prior failure prevented this step from running)

---

Integration-Specific Report Details

Each integration framework adds its own enhancements to the reports. The sections below describe what each integration contributes beyond the core report structure.

xUnit

The xUnit integration uses the core report generator directly. Features are derived from test class names and scenarios from test display names. Reports are generated when the test run fixture is disposed.

No additional report content is added beyond the core features.

NUnit 4

The NUnit integration uses the core report generator directly. Features are derived from NUnit's TestContext, with scenario names from test method names. Reports are generated in the GlobalTeardown method after all tests complete.

No additional report content is added beyond the core features.

BDDfy (xUnit v3)

The BDDfy integration provides a custom report generator that enhances both HTML reports with:

• BDD steps — Each scenario displays its Given/When/Then steps in a styled list with colour-coded keywords.
• Story descriptions — If a BDDfy story has a description, it is shown below the feature heading in italics.

The YAML specifications also include the BDD steps for each scenario.

Scenario data is collected automatically via BDDfyScenarioCollector during test execution. Reports are typically triggered from an [AssemblyFixture].

ReqNRoll (xUnit v2, xUnit v3 & TUnit)

The ReqNRoll integration provides a custom report generator that enhances both HTML reports with:

• Gherkin steps — Each scenario displays its Given/When/Then steps in a styled list with colour-coded keywords.
• Feature descriptions — If a Gherkin feature file has a description, it is shown below the feature heading in italics.

The YAML specifications also include the Gherkin steps for each scenario.

Additionally, the ReqNRoll integration automatically enhances ReqNRoll's native Cucumber HTML report by injecting sequence diagram image attachments and raw PlantUML code into the Cucumber Messages JSON. This means diagrams appear inside the standard Cucumber report alongside the existing Gherkin steps and results, with no extra configuration needed.

Scenario data is collected via ReqNRollScenarioCollector during test execution. Reports are triggered from a [AfterTestRun] hook.

LightBDD (xUnit v2, xUnit v3 & TUnit)

The LightBDD integration takes a different approach — rather than generating its own HTML, it plugs into LightBDD's native report pipeline. Report generation is configured via an extension method on ReportWritersConfiguration:

configuration.CreateStandardReportsWithDiagrams(typeof(MyTest).Assembly, options);

This registers LightBDD's built-in formatters (CustomisableHtmlReportFormatter and YamlReportFormatter) with diagram data injected via the ExampleDiagramsAsCode option. LightBDD handles the HTML rendering, BDD step display, and report lifecycle automatically.

The same three output files are produced, but LightBDD controls the HTML layout and styling natively.

---

CI Summary (Optional)

When WriteCiSummary = true is set on ReportConfigurationOptions, two additional files may be produced:

CiSummary.md

A Markdown summary of the test run, written to the Reports folder and also to the CI platform:

• GitHub Actions — Appended to $GITHUB_STEP_SUMMARY (appears inline in the job summary)
• Azure DevOps — Uploaded via ##vso[task.uploadsummary] (appears in the Extensions tab)

The summary is failure-focused: when tests fail, it shows up to MaxCiSummaryDiagrams failed scenarios with error details and sequence diagram images. When all tests pass, it shows diagrams for the first N scenarios.

---

Browser-Rendered Reports

When using PlantUmlRendering.BrowserJs, the standard HTML reports (TestRunReport and Specifications) also render PlantUML diagrams client-side in the browser using the same PlantUML TeaVM JS engine. Diagrams are lazily rendered as the user scrolls, keeping initial page load fast even with thousands of diagrams. See PlantUML Browser Rendering for details.

See CI Summary Integration for full configuration and CI platform setup guides.

---

CI Artifact Upload (Optional)

When PublishCiArtifacts = true is set on ReportConfigurationOptions, all report files in the Reports directory (.html, .yml, .md) are published as CI build artifacts:

• Azure DevOps — Each file is uploaded via ##vso[artifact.upload] logging commands during test execution. No additional pipeline configuration needed.
• GitHub Actions — The reports directory path and retention days are written to $GITHUB_OUTPUT. Add a single upload-artifact step to your workflow.

This works alongside the CI summary — the summary appears inline in the build page while the full reports are available as downloadable artifacts.

See CI Artifact Upload for full configuration and workflow examples.

---

Diagram Storage Format

Sequence diagrams in HTML reports are stored as gzip-compressed base64 in data-plantuml-z attributes. The browser-side JavaScript decompresses and renders them dynamically. This means you cannot find diagram content by searching the HTML source with plain text — the content is opaque base64 strings.

To inspect diagram content programmatically:

# PowerShell: Decompress a diagram from the report
$html = Get-Content "Reports/TestRunReport.html" -Raw
$m = [regex]::Match($html, 'data-plantuml-z="([^"]+)"')
$bytes = [Convert]::FromBase64String($m.Groups[1].Value)
$ms = [System.IO.MemoryStream]::new($bytes)
$gz = [System.IO.Compression.GZipStream]::new($ms, 'Decompress')
$sr = [System.IO.StreamReader]::new($gz)
$sr.ReadToEnd()  # → @startuml ... @enduml

Tip: Use the Click-to-reveal PlantUML feature in the HTML report instead — click any diagram image to see the raw PlantUML source directly in the browser.

---

Troubleshooting

Entries logged but not appearing in report

If you are calling RequestResponseLogger.Log() (e.g. from a custom tracker) and the entries don't appear in the generated report:

1. Check TestId matches — Entries are grouped by TestId. If the TestId you pass to RequestResponseLogger.Log() doesn't match what BDDfy/xUnit/ReqNRoll uses for the scenario, the entries will be orphaned.
2. Check TrackingIgnore — Entries with TrackingIgnore = true are excluded from diagram rendering.
3. Pair your Request and Response — Each interaction needs both a Request and Response entry with matching TraceId and RequestResponseId. An unpaired entry produces an incomplete arrow.
4. Check timing — Ensure logging happens before report generation. If you log entries after CreateStandardReportsWithDiagrams() runs, they won't appear.
5. Don't search HTML with plain text — Diagrams are stored as gzip-compressed base64 in data-plantuml-z attributes (see Diagram Storage Format above). Plain text search for participant names or URIs will return zero results.
6. YAML/JSON data files don't contain diagrams — The Specifications.yml and TestRunReport.yml files contain scenario metadata only, not PlantUML diagram content. Diagrams are only in the .html reports.