Skip to content

Add option to create markdown formatted docs for each scenario. - #59

Merged
RomanPszonka merged 16 commits into
openutm:mainfrom
atti92:scenario-markdown-docs
Dec 7, 2025
Merged

Add option to create markdown formatted docs for each scenario.#59
RomanPszonka merged 16 commits into
openutm:mainfrom
atti92:scenario-markdown-docs

Conversation

@atti92

@atti92 atti92 commented Nov 29, 2025

Copy link
Copy Markdown
Contributor

Fixes #47
Closes #48

Visible on github: https://github.com/atti92/utm_verification/blob/scenario-markdown-docs/docs/index.md

image

Description:

This PR introduces the ability to attach Markdown documentation to verification scenarios. This documentation is automatically read during execution and rendered as HTML within the generated verification reports. This allows for better context and explanation of test scenarios directly in the output.

Key Changes:

Documentation Support:

  • Added markdown library dependency.

  • Updated ScenarioResult model to include an optional docs field.

  • Modified the HTML report template (report_template.html) to render the docs field using a custom Jinja2 markdown filter.
    Registry & Execution:

  • Updated register_scenario decorator to accept an optional docs argument (filename).

  • Enhanced dependencies.py to resolve and read the documentation file relative to the scenario function's location.

  • Propagated documentation content through RunContext to the final ScenarioResult.

Example:

  • Added add_flight_declaration.md as documentation for the add_flight_declaration scenario.
  • Updated test_add_flight_declaration.py to register this documentation file.

How to Test:

  • Run the verification tool with the default configuration:
  • uv run openutm-verify --config config/default.yaml
  • Open the generated HTML report in reports.
  • Expand the "add_flight_declaration" scenario.
  • Verify that the "Documentation" section is visible and correctly renders the content from src/openutm_verification/scenarios

/add_flight_declaration.md.

@atti92
atti92 marked this pull request as ready for review November 29, 2025 13:13
Comment thread src/openutm_verification/core/execution/dependencies.py Outdated
Comment thread src/openutm_verification/scenarios/add_flight_declaration.md Outdated
@hrishiballal

Copy link
Copy Markdown
Contributor

A sample documentation can be like the following image, source, Section 8.3.2, it describes in text and maps what is happening in the scenario the PDF has additional examples, we should be able to build this type of description (among others):
Screenshot from 2025-12-02 10-56-19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for attaching markdown documentation to verification scenarios, which is automatically rendered in HTML reports. The implementation introduces a documentation discovery system that searches for markdown files in multiple locations (docs/scenarios directory or side-by-side with scenario files) and integrates them into the reporting pipeline.

Key Changes:

  • Added markdown documentation support with automatic discovery and HTML rendering
  • Introduced docs path resolution utilities and updated scenario registry to track documentation paths
  • Extended the execution context and reporting models to propagate documentation content through the system

Reviewed changes

Copilot reviewed 18 out of 21 changed files in this pull request and generated 16 comments.

Show a summary per file
File Description
src/openutm_verification/utils/paths.py New utility to locate docs directory in both installed and development environments
src/openutm_verification/scenarios/registry.py Updated to discover and track documentation files for each scenario
src/openutm_verification/scenarios/test_fire_response.py New placeholder scenario with accompanying markdown documentation
src/openutm_verification/core/reporting/reporting_models.py Extended ScenarioResult and ReportData models with optional docs fields
src/openutm_verification/core/templates/report_template.html Added documentation section rendering with markdown-to-HTML conversion
src/openutm_verification/core/reporting/reporting.py Added markdown filter and image copying functionality for docs
src/openutm_verification/core/execution/execution.py Integrated docs directory path and propagated docs content to results
src/openutm_verification/core/execution/dependency_resolution.py Updated CONTEXT default to include docs field
src/openutm_verification/core/execution/dependencies.py Added get_scenario_docs function and updated scenarios generator
src/openutm_verification/core/execution/config_models.py Extended RunContext TypedDict with docs field
src/openutm_verification/core/execution/scenario_runner.py Added ScenarioRegistry TypedDict definition
pyproject.toml Added markdown dependency and configured docs packaging with force-include
config/default.yaml Added fire_response scenario to execution list
docs/scenarios/fire_response.md New markdown documentation for fire response scenario
docs/index.md New index page for scenario documentation
Dockerfile Changed to editable install mode
.gitattributes Added binary file declarations for images
docs/.DS_Store macOS system file (should not be committed)
Comments suppressed due to low confidence (3)

src/openutm_verification/core/execution/dependencies.py:1

  • Import of 'base64' is not used.
import base64

src/openutm_verification/core/execution/dependencies.py:2

  • Import of 'mimetypes' is not used.
import mimetypes

src/openutm_verification/core/execution/dependencies.py:3

  • Import of 're' is not used.
import re

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/openutm_verification/core/execution/dependencies.py Outdated
Comment thread src/openutm_verification/utils/paths.py Outdated
Comment thread src/openutm_verification/core/reporting/reporting.py Outdated
Comment thread src/openutm_verification/core/execution/dependencies.py Outdated
Comment thread src/openutm_verification/scenarios/registry.py Outdated
Comment thread src/openutm_verification/core/execution/dependencies.py Outdated
Comment thread src/openutm_verification/core/reporting/reporting.py Outdated
Comment thread Dockerfile Outdated
Comment thread src/openutm_verification/core/execution/dependencies.py Outdated
Comment thread src/openutm_verification/scenarios/test_fire_response.py Outdated
atti92 and others added 5 commits December 5, 2025 14:53
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@RomanPszonka
RomanPszonka merged commit bba57fb into openutm:main Dec 7, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scenario Definition: Integrate requirements with test runner Scenario Definition (YAML / Markdown)

4 participants