diff --git a/text/0082-toggle-coverage.md b/text/0082-toggle-coverage.md new file mode 100644 index 0000000..5ec562c --- /dev/null +++ b/text/0082-toggle-coverage.md @@ -0,0 +1,230 @@ +- Start Date: 2025-08-27 +- RFC PR: [amaranth-lang/rfcs#0000](https://github.com/amaranth-lang/rfcs/pull/0000) +- Amaranth Issue: [amaranth-lang/amaranth#0000](https://github.com/amaranth-lang/amaranth/issues/0000) + +# Toggle Coverage + +## Summary +[summary]: #summary + +Introduce toggle coverage, a feature that tracks the number of individual signal transitions from 0→1 and 1→0. It generates human-readable reports that list each signal's bitwise toggle by its full hierarchical path. + +## Motivation +[motivation]: #motivation + +Toggle coverage is one of the fundamental coverage metrics in hardware verification. It helps identify signals that remain constant during tests, revealing insufficient stimulus, missing corner cases, or unreachable logic. It is also used in activity/power analysis and is expected in most coverage flows. +Today, Amaranth users who need toggle coverage must either post-process VCDs or rely on simulator-specific options (e.g., Verilator’s --coverage-toggle) with no consistent integration. Adding toggle coverage to Amaranth provides a unified, easy-to-use API across simulators and improves test quality by highlighting dead or under-exercised parts of designs. + +## Guide-level explanation +[guide-level-explanation]: #guide-level-explanation +Toggle coverage is currently implemented for Amaranth’s built-in Python simulator, which is most useful for lightweight unit tests and quick functional checks. In this environment, users can attach an observer to collect per-bit toggle counts without modifying their testbench code. + +##### Example: UART Peripheral with Toggle Coverage + +A user writes a testbench for a simple `UARTPeripheral`. By attaching a `ToggleCoverageObserver` to the simulator, they can collect per-bit transition counts for all relevant signals. + +```python +from amaranth.sim import Simulator +from amaranth.sim.coverage import ToggleCoverageObserver +from tests.test_utils import get_signal_full_paths +from chipflow_digital_ip.io import UARTPeripheral + +dut = UARTPeripheral(init_divisor=8, addr_width=5) +sim = Simulator(dut) + +# Attach toggle coverage +design = sim._engine._design +signal_path_map = get_signal_full_paths(design) +toggle_cov = ToggleCoverageObserver(sim._engine.state, signal_path_map=signal_path_map) +sim._engine.add_observer(toggle_cov) + +# Add clock, processes, and testbench (omitted for brevity) +sim.add_clock(1e-6) +# sim.add_process(uart_rx_proc) +# sim.add_process(uart_tx_proc) +# sim.add_testbench(testbench) + +with sim.write_vcd("uart.vcd"): + sim.run() +toggle_cov.close(0) +``` + +When the simulation completes, Amaranth prints a toggle coverage report: + +``` +=== Toggle Coverage Report === +bench/top/_uart/rx/bridge/mux/bus__r_stb: +Bit 0: 0→1=2, 1→0=3 +bench/top/_uart/rx/bridge/Status/element__r_stb: +Bit 0: 0→1=1, 1→0=2 +bench/top/_phy/rx/lower/fsm_state: +Bit 0: 0→1=0, 1→0=1 +Bit 1: 0→1=1, 1→0=1 +bench/top/_phy/rx/lower/timer: +Bit 0: 0→1=39, 1→0=38 +Bit 1: 0→1=20, 1→0=19 +Bit 2: 0→1=10, 1→0=10 +Bit 3: 0→1=0, 1→0=0 +Bit 4: 0→1=0, 1→0=0 +Bit 5: 0→1=0, 1→0=0 +... +``` +Each signal is listed with its full hierarchical path (e.g. `bench/top/_uart/rx/bridge/Status/element__r_stb`). Each bit records how many times it transitioned from `0→1` and from `1→0`. + +From a programmer’s perspective, toggle coverage should be thought of as a completeness check: it does not prove correctness, but it highlights which parts of the design have remained idle across the test suite. This makes it easier to spot insufficient test stimulus or unreachable conditions. + +For new users, the feature provides an approachable structural coverage metric that requires no special setup beyond enabling the collector. Reports identify under-exercised signals by their hierarchical paths, making it straightforward for developers to locate the relevant logic in source code. As testbenches evolve, coverage output offers immediate feedback on whether changes have reduced or improved test completeness. + +Finally, no deprecations or migration steps are required: toggle coverage is additive, opt-in, and does not affect existing simulation behavior. + + +## Reference-level explanation +[reference-level-explanation]: #reference-level-explanation +Toggle coverage is implemented as an observer attached to the simulation engine. For each signal, it stores the previous value, then on each tick it compares the old and new values bit by bit. If a bit changes from 0→1 or 1→0, it increments the corresponding counter. Each counter is stored in a dictionary keyed by signal identifier and bit index. Signal names are resolved into full hierarchical paths, ensuring results can be traced directly back to source code. + +#### API: `ToggleCoverageObserver` + +```python +class ToggleCoverageObserver(Observer): + def __init__(self, state, signal_path_map: Optional[Dict[int, str]] = None, **kwargs): ... + def update_signal(self, timestamp: int, signal: Signal) -> None: ... + def update_memory(self, timestamp: int, memory, addr) -> None: ... + def get_results(self) -> Dict[str, Dict[int, Dict[ToggleDirection, int]]]: ... + def close(self, timestamp: int) -> None: ... +``` +#### `ToggleDirection` Enum +Represents the direction of a signal toggle: +- **`ZERO_TO_ONE`** — transition from logic 0 → 1 +- **`ONE_TO_ZERO`** — transition from logic 1 → 0 + +#### Constructor + +- **`state`**: simulation state object, used to query signal values. +- **`signal_path_map`**: optional dict mapping `id(signal)` → hierarchical path string. +- **`**kwargs`**: forwarded to the base `Observer`. + +#### Fields + +- **`_prev_values: Dict[int, int]`** — last observed values, keyed by signal ID. +- **`_toggles: Dict[int, Dict[int, Dict[ToggleDirection, int]]]`** — per-signal, per-bit toggle counters. +- **`_signal_names: Dict[int, str]`** — human-readable names for reporting. +- **`_signal_path_map: Dict[int, str]`** — mapping of signal IDs to full paths. + +#### Methods + +- **`update_signal(timestamp, signal)`** + Updates toggle counters by comparing current vs. previous values, bit by bit. + +- **`update_memory(timestamp, memory, addr)`** + Currently a placeholder with no effect. + +- **`get_results()`** Returns structured results: + ```python + { + "signal_name": { + bit_index: { + ToggleDirection.ZERO_TO_ONE: count, + ToggleDirection.ONE_TO_ZERO: count + } + } + } +- **`close(timestamp)`** Prints a formatted toggle coverage report to stdout. +#### Helper functions +##### `collect_all_signals(obj)` Helper +Recursively traverses a design object to collect all `Signal` instances. +- Skips private attributes (those starting with `_`). +- Follows `submodules` if present (supports both dicts and iterables). +- Returns: a list of discovered `Signal` objects. + +##### `get_signal_full_paths(design)` Helper +Builds hierarchical names for all signals in a design. +- Iterates through `design.fragments` and their signal names. +- Returns: a dictionary mapping `id(signal)` → full hierarchical path string. + +#### Example Workflow + +Using the `UARTPeripheral` testbench (see Guide-level explanation): + +1. The user attaches a `ToggleCoverageObserver` to the simulator. +2. During execution, each clock tick updates the observer with new signal values. +3. On a transition (e.g. `prev=0, curr=1`), the appropriate counter is incremented. +4. At the end of simulation, `toggle_cov.close()` produces a report. + +#### Corner Cases +- **First sample of a signal**: when a signal is seen for the first time, its current value is stored but no toggle is recorded, since there is no previous value to compare against. +- **Bitwise handling**: multi-bit signals are tracked per bit; if only some bits change, only those bit counters are incremented. +- **No toggles**: signals that never change during simulation remain in the report with zero counts for both directions. + +#### Interaction with Other Features + +- Works alongside other coverage observers (e.g. statement, block coverage). +- The observer is read-only and has no side effects on simulation behavior. +- Backward compatible: the feature is strictly opt-in. Existing designs and testbenches require no modification and continue to run unchanged. + +## Drawbacks +[drawbacks]: #drawbacks +- **Increased complexity**: Introducing another coverage type adds new APIs, configuration options, and reporting paths that need to be maintained in Amaranth. +- **Simulation overhead**: Recording per-bit transitions adds bookkeeping at each tick, which may slow down large simulations if many signals are tracked. +- **Performance trade-offs**: Even with filtering, toggle coverage introduces extra work each cycle. Users who do not need it may see slower simulations if they enable it unnecessarily. + +## Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +This design is the best choice because it builds on Amaranth’s existing simulator observer interface, keeping coverage separate from design and testbench code. By counting bit-level 0→1 and 1→0 transitions, it provides exactly the information verification engineers expect from toggle coverage, while staying lightweight and easy to extend. Unlike external-only solutions (e.g. relying solely on Verilator flags or post-processing VCDs), this approach gives Amaranth users a consistent API across simulators, immediate feedback during testing, and reports tied directly to hierarchical signal paths in their designs. Having toggle coverage in the Python simulator helps new users understand the concept without needing Verilator or extra tools. + +#### Alternatives considered +- **Post-processing waveforms (VCD/FSDB) to compute toggles** + *Pros:* simulator-agnostic; no runtime overhead in the simulator. + *Cons:* slow and memory-heavy on large traces; direction counts must be inferred; requires extra tooling and a second pass. + *Why not:* increases friction and resource usage; less immediate feedback. + +- **Simulator-specific coverage only (e.g., “just use Verilator flags”)** + *Pros:* leverages mature backend features; high performance for big designs. + *Cons:* fractured user experience; different flags, formats, and reports per backend; difficult to teach and document. + *Why not:* Amaranth aims for consistent tooling and ease of learning, but today there is no unified API across simulators. This makes toggle coverage harder to use for both new learners and advanced users, since it requires backend-specific commands and extra tooling. + +- **Functional coverage libraries** + *Pros:* expressive, user-defined coverage points; great for protocols, corner cases, and cross-conditions. + *Cons:* they don’t automatically tell you if a particular signal or bit ever changed state. Writing functional coverage requires extra effort to define bins and events - higher authoring cost. + *Why not:* functional coverage complements, but cannot substitute toggle coverage. Toggle coverage ensures structural activity (bits move at least once), while functional coverage ensures behavioral activity (scenarios were exercised). + +- **Inline instrumentation (manually adding counters into testbench, using macros/ library utilities)** + *Pros:* no simulator support required; explicit in tests. + *Cons:* pollutes design/testbench with counters and boilerplate; easy to miss signals; harder to maintain. + *Why not:* observer does nto change the DUT or clutter test code. This separation is cleaner, less error-prone, and easier to maintain. + +### Impact of not doing this +- **Fragmented workflows**: Without a unified toggle coverage feature, users are stuck with a mix of ad-hoc methods (e.g. post-process VCD files, Verilator flags). This results in inconsistent behavior across simulators, more effort for users, and more support questions for maintainers. +- **Lower test effectiveness**: Missing an essential structural metric makes it harder to detect unexercised logic early. + +Toggle coverage could technically be implemented as a standalone library, providing an observer and reporting scripts outside of Amaranth itself. However, this approach is suboptimal: without upstream support, such a library would rely on unstable simulator hooks, invent its own naming schemes and formats, and remain disconnected from Amaranth’s documentation and tutorials. By integrating toggle coverage directly, Amaranth can guarantee consistent APIs across backends, stable hierarchical naming, and a unified reporting surface. + +## Prior art +[prior-art]: #prior-art + +Toggle coverage is a long-established feature in hardware simulators. Commercial tools such as Questa/ModelSim, Xcelium, and VCS provide it alongside line, branch, and FSM coverage. They record results into UCIS databases, which makes it easy to merge results from different runs and display them in coverage dashboards. These tools are mature and polished, with graphical interfaces and robust reporting, but they come with downsides: they are proprietary, require expensive licenses, and each vendor uses its own command lines and formats. In the open-source space, Verilator includes toggle coverage counters and utilities for post-processing reports. This is fast and accessible, but the options and report formats are tool-specific, and users need to learn Verilator’s workflow separately. + +Other open-source flows compute toggle counts by post-processing waveforms such as VCD files. This has the advantage of working with any simulator, but it is often slow and memory-intensive on large designs, and it does not provide a unified API. Functional coverage libraries, such as SystemVerilog covergroups or Python packages, are widely used for protocol scenarios. These are powerful and flexible, but they answer different questions: they check if a protocol behaved as expected, not whether every bit in the design ever toggled. They complement toggle coverage but do not replace it. + +There is no direct equivalent of toggle coverage in software ecosystems. Languages like C, Java, or Python rely on statement, branch, or path coverage (via tools like gcov, LCOV, or JaCoCo). Software does not have per-bit signal activity, so toggle coverage is a hardware-specific structural metric. + +Amaranth takes a slightly different approach. Its goal is to provide consistent tooling and an approachable learning experience. By integrating toggle coverage directly, Amaranth can avoid backend-specific flags or offline scripts and instead present a single, unified API. Reports would be tied to hierarchical signal paths and work consistently across simulators, while remaining optional and non-intrusive to user RTL or testbenches. + +## Unresolved questions +[unresolved-questions]: #unresolved-questions +- **Output formats.** Is a plain text report sufficient for the first version, or should JSON export be included? Should UCIS/LCOV output be in scope for this RFC or deferred to later proposals? +- **Signal filtering.** Should include/exclude lists and a maximum bit-width cap be defined in this RFC, or added as extensions after the basic observer is merged? +- **Interaction with other coverage types.** Since work is also underway on statement and block coverage, should toggle coverage eventually share a common reporting interface with these metrics? +- **Scope of implementation.** Should this RFC cover only the Python simulator (current implementation) or also include Verilator integration? If not, how should future work on Verilator support be tracked? + + +## Future possibilities +[future-possibilities]: #future-possibilities + +The most natural next step is to integrate toggle coverage with other coverage metrics currently under development, such as statement, block, assertion, unreachable code, and expression coverage. Toggle coverage will evenually share a common reporting interface with these other metrics. This would allow users to collect, merge, and visualize different coverage types through a single entry point, making Amaranth’s coverage ecosystem more cohesive. + +Toggle coverage could also participate in hierarchical aggregation, where reports summarize coverage per module, instance, or subsystem rather than just per signal bit. + +Over time, toggle coverage might evolve into a more configurable feature. Examples include filtering (include/exclude lists, maximum bus width), or selective depth of hierarchy in reports. + +Ultimately, toggle coverage can serve as a foundation for more advanced metrics. While the initial implementation focuses on simple per-bit transitions, future work can unify it with broader coverage reporting, add richer configuration options, and extend support across backends. This evolution would position toggle coverage not only as a basic completeness check but also as an integral part of a comprehensive verificatiosn framework. \ No newline at end of file diff --git a/text/0083-statement-coverage.md b/text/0083-statement-coverage.md new file mode 100644 index 0000000..ae6e905 --- /dev/null +++ b/text/0083-statement-coverage.md @@ -0,0 +1,192 @@ +- Start Date: 2025-09-17 +- RFC PR: [amaranth-lang/rfcs#0000](https://github.com/amaranth-lang/rfcs/pull/0000) +- Amaranth Issue: [amaranth-lang/amaranth#0000](https://github.com/amaranth-lang/amaranth/issues/0000) + +# Statement Coverage + +## Summary +[summary]: #summary +Introduce statement coverage, a feature that tracks execution of individual HDL statements: assignments and conditional branches (switch / switch_case). The report indicates whether each statement was HIT or MISS (and how many times), along with the file name and line number of the Python source that generated the HDL. + +## Motivation +[motivation]: #motivation +Statement coverage is a fundamental metric in hardware verification. It ensures that every line of code in the design has been exercised at least once during simulation. Uncovered statements highlight dead code, incomplete testbenches, or untested logic branches, guiding engineers to strengthen their stimulus and improve test completeness. +Without built-in support, Amaranth users must rely on simulator-specific options or manual inspection of waveforms/logs to approximate this metric. Integrating statement coverage directly into Amaranth provides a unified solution across simulators — aligning with standard verification practices and improving overall design quality. + +## Guide-level explanation +[guide-level-explanation]: #guide-level-explanation +Statement coverage in Amaranth’s Python simulator works by tagging each HDL statement with its type (assignment or conditional), a unique ID (hierarchical path, clock domain, numeric counter), and a human-readable name (file and line number plus a short summary such as `comb:sda = rhs or sync:switch_case(pattern)`), making results both machine-traceable and easy to interpret in reports. A coverage signal is inserted so that when the statement executes, the signal goes high for one cycle. The StatementCoverageObserver records these hits, and at the end results are merged into a report showing the amount of times it was HIT or MISS. + +##### Example: I²C Peripheral with Statement Coverage +Below is a minimal example showing how to enable statement coverage for an I²C Peripheral. Some code in the original testbench omitted for brevity. + +```python +from amaranth.sim import Simulator +from amaranth.sim.coverage import StatementCoverageObserver +from tests.test_utils import * +from chipflow_digital_ip.io import I2CPeripheral + +dut = _I2CHarness() + +# Build a simulator with statement-coverage instrumentation +sim, stmt_cov, stmt_info, fragment = mk_sim_with_stmtcov(dut, verbose=True) + +# Add clock, processes, and testbench (omitted for brevity) +sim.add_clock(1e-6) +# sim.add_testbench(testbench) + +with sim.write_vcd("i2c_stmtcov.vcd", "i2c_stmtcov.gtkw"): + sim.run() + +# Merge and emit a JSON summary across tests +merge_stmtcov(results, stmt_info) +emit_agg_summary("i2c_statement_cov.json", label="tests/test_i2c.py") +``` + +When the simulation finishes, Amaranth prints a statement coverage report: + +``` +[Statement coverage for tests/test_i2c.py] 315/378 = 83.3% + +HIT (3x) | chipflow-digital-ip/tests/test_i2c.py:51 | comb:sda = (& (~ (sig i2c_pins__sda__oe)) (sig sda_i)) +HIT (3x) | chipflow-digital-ip/tests/test_i2c.py:55 | comb:i2c_pins__scl__i = scl +HIT (8x) | .../amaranth_soc/csr/bus.py:564 | i2c/bridge/mux | comb:switch_case(('10000',)) +HIT (3x) | .../amaranth_soc/csr/bus.py:564 | i2c/bridge/mux | sync:switch_case(('00001',)) +... +``` +HIT means that a statement executed at least once during the run. The observer not only marks it as covered but also keeps a counter, so you can see how many times that statement was executed in total (ex. (3x) means three times). MISS indicates code that never executed. This highlights untested branches, dead code, or insufficient stimulus in the testbench. +Each entry in the report includes the hierarchical path (e.g. i2c/bridge/mux) and the source location (src_loc), allowing you to quickly find the exact statement in the source code. For conditional logic, entries such as switch_case(('00001',)) show which case patterns were taken, while default cases are explicitly labeled. +There are some important trade-offs to keep in mind. Instrumentation adds a small amount of extra logic and can modestly increase simulation time, though this is typically negligible for unit tests. Hit counts represent statement execution events, not per-cycle residency; a signal toggling frequently does not increase the count unless the statement itself re-executes. Statement coverage only shows which code was exercised, not whether behavior was correct. MISS items should be treated as guidance to improve stimulus or revisit design intent. Finally, there is zero migration cost: statement coverage is opt-in and additive; enabling it does not affect existing simulation behavior. + +## Reference-level explanation +[reference-level-explanation]: #reference-level-explanation +Similar to toggle coverage, statement coverage also has an observer attached to the simulation engine. Each HDL statement is tagged during elaboration with a unique ID, a type (assignment or conditional), and a human-readable name. A coverage signal is injected so that when the statement executes, the signal goes high for one cycle. The StatementCoverageObserver records these pulses, and each execution increments that statement’s entry in the hit counter dictionary `AGG_STMT_HITS`. Finally, the unique IDs and names are mapped back to their file name and line number, and the report prints both the source location and the number of times each statement executed. + +#### API: `StatementCoverageObserver` + +```python +class StatementCoverageObserver(Observer): + def __init__(self, coverage_signal_map, state, stmtid_to_info=None, **kwargs): ... + def update_signal(self, timestamp: int, signal: Signal) -> None: ... + def update_memory(self, timestamp: int, memory, addr) -> None: ... + def get_results(self) -> Dict[StmtID, int]: ... + def close(self, timestamp: int) -> None: ... + +``` + +#### Constructor +- **`coverage_signal_map`**: maps id(signal) → statement ID. +- **`state`**: simulation state object, used to track signal values. +- **`stmtid_to_info`**: optional dictionary mapping statement IDs → (name, type). + +#### Fields +- **`_statement_hits: Dict[StmtID, int]`** — counters of how many times each statement executed.. +- **`coverage_signal_map: Dict[int, StmtID]`** — reverse lookup from signal ID to statement ID. +- **`stmtid_to_info: Dict[StmtID, Tuple[str, str]]`** — statement metadata (name, type). + +#### Methods +- **`update_signal(timestamp, signal)`** + Checks whether a signal corresponds to a tagged statement; if so, increments its hit counter. +- **`update_memory(timestamp, memory, addr)`** + Currently a placeholder with no effect. +- **`get_results()`** + Returns a dictionary of `{stmt_id: hit_count}`. +- **`close(timestamp)`** + Currently a no-op; results are typically collected via `get_results()` and merged externally. + +#### Helper functions +##### `tag_all_statements(fragment)` +Recursively traverses an Amaranth fragment, assigning each `Assign` and `Switch` statement a unique coverage ID, name, and type. The header of each switch statement is tagged (type = "switch") and each case inside switch is also tagged separately (type = "switch_case"). Every branch (e.g. case 0:, case 1:) gets its own unique ID and name, allowing coverage to distinguish which branches of the conditional were executed. +##### `insert_coverage_signals(fragment)` +Walks through every HDL statement in the design and inserts a special coverage signal assignment just before it. These coverage signals don’t affect the design logic; instead, they act as breadcrumbs. Each time a statement executes, its breadcrumb signal is written for one cycle, and the `StatementCoverageObserver` detects this write, increments the statement’s hit counter, and records the execution. This way, every Assign, Switch, and switch_case can be tracked precisely during simulation. +##### `mk_sim_with_stmtcov(dut, verbose=False)` +Elaborates the DUT into an Amaranth IR fragment, then calls `tag_all_statements(fragment)` and `insert_coverage_signals(fragment)`. Next, it builds a simulator from the instrumented fragment and attaches a `StatementCoverageObserver`, which maps those coverage signals back to statement IDs and increments hit counters whenever they fire. Function returns `(sim, stmt_cov, stmtid_to_info, fragment)`. +##### `merge_stmtcov(results, stmtid_to_info)` +Merges results from multiple simulations into global aggregators `AGG_STMT_HITS` and `AGG_STMT_INFO`. It first registers all statements, then accumulates statement hits. After merging all tests, the globals contain everything needed to compute coverage percentages and to print a final report with HIT (Nx) or MISS (0x) for every statement. +##### `emit_agg_summary(json_path, label)` +Emits a console report and a JSON file showing each statement’s source location, type, and hit count. It calculates the overall coverage percentage by dividing the number of statements executed at least once by the total number of instrumented statements. + +#### Example Workflow +Using the `I²C peripheral` testbench (see Guide-level explanation): + +1. The user builds a simulator with `mk_sim_with_stmtcov(dut)`. +2. The helper injects coverage signals into all `Assign` and `Switch` statements. +3. During simulation, when a statement executes, its signal pulses high for one cycle. The `StatementCoverageObserver` records the hit for that statement ID. +4. After simulation, results are merged, and emit_agg_summary prints a HIT/MISS report and writes a JSON file. + +#### Corner Cases +- **Default switch cases:**: labeled as `switch_case(default)` when patterns is `None` (catch-all case). +- **Unreachable code**: statements that never execute appear as MISS (0x) in the report. +- **Nested fragments**: recursion ensures statements inside submodules or cases are covered. +- **Hit count granularity**: counter is incremented only when a statement actually executes, not for every cycle it remains active. + +#### Interaction with Other Features + +- Designed to work alongside other coverage observers (toggle, block, expression, assertion). Especially unreachable code coverage ... +- The observer is read-only and does not affect simulation behavior. +- Backward compatible: opt-in and additive; existing designs and testbenches run unchanged if not enabled. + +## Drawbacks +[drawbacks]: #drawbacks +- **Increased complexity**: Adding statement coverage introduces new helper functions, observers, and reporting logic, which increases maintenance cost in Amaranth. +- **Instrumentation overhead**: Each statement gets an extra coverage signal and assignment, slightly expanding the design and simulation workload. +- **Simulation slowdown**: The observer must check and count every coverage signal write. For designs with thousands of statements, this can noticeably impact simulation speed. +- **Limited granularity**: Coverage only records whether a statement executed and how many times; it does not capture path conditions, correctness of values, or deeper functional coverage. + +## Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives +This design builds on Amaranth’s existing simulator observer interface and works entirely at the IR (Fragment) level, so coverage stays separate from the DUT and testbench code. By tagging each Assign/Switch (and each switch_case) with a unique ID and injecting a one-cycle coverage pulse, it captures exactly what engineers expect from statement coverage: whether each statement executed and how many times. Results are mapped back to source locations, giving immediate, actionable feedback during testing with a consistent API across simulators. Keeping it in the Python simulator makes the concept approachable for new users without requiring external tools or backend-specific flags. + +#### Alternatives considered +- **Post-processing waveforms (VCD/FSDB) to infer statement hits** + *Pros:* simulator-agnostic; no runtime overhead in the simulator. + *Cons:* inferring which statement executed from signals is error-prone; slow on large traces; needs extra tooling/passes. + *Why not:* fragile inference and higher friction; less immediate feedback. + +- **Simulator-specific statement coverage (e.g., backend flags only)** + *Pros:* can leverage mature features in some tools; good performance on large designs. + *Cons:* fragmented UX (different flags/formats), inconsistent reports/naming, harder to teach and document. + *Why not:* Amaranth aims for a unified, portable coverage surface; backend-only solutions don’t provide that. + +- **Functional coverage libraries** + *Pros:* expressive, user-defined coverage points; great for protocols, corner cases, and cross-conditions. + *Cons:* do not automatically show whether each HDL statement executed; authoring bins/events adds effort. + *Why not:* functional coverage complements but does not replace statement coverage. We still need a structural baseline to expose dead/untouched code. + +- **Inline instrumentation (manually adding counters into testbench, using macros/ library utilities)** + *Pros:* no simulator hooks required; explicit control. + *Cons:* clutters code, easy to miss statements, harder to maintain, not standardized across projects. + *Why not:* the observer approach keeps coverage out of the DUT, minimizes boilerplate, and standardizes reporting. + +### Impact of not doing this +- **Fragmented workflows**: Users would bounce between ad-hoc scripts, backend flags, or waveform post-processing to approximate statement hits, leading to inconsistent results and more support burden. +- **Lower test effectiveness**: Without a built-in structural metric, unexecuted assignments/branches remain hidden longer, reducing confidence in test completeness. + +Statement coverage could technically live as a standalone library, but that would rely on unstable hooks, ad-hoc naming, and disconnected documentation. Integrating directly with Amaranth guarantees stable IDs/names, consistent APIs, and a unified reporting surface—making coverage both reliable and easy to adopt. + +## Prior art +[prior-art]: #prior-art +Statement coverage is one of the most established structural metrics in hardware verification. Commercial simulators such as Questa/ModelSim, Xcelium, and VCS provide statement, branch, and FSM coverage alongside toggle coverage. Results are usually stored in UCIS databases, which makes it easy to merge multiple runs and display progress in polished dashboards. These tools are mature, with graphical browsers and sophisticated reporting, but they are proprietary, expensive, and vendor-specific: each simulator has its own switches, report formats, and workflows. + +In the open-source space, Verilator supports line and toggle coverage and can emit LCOV-compatible reports. This is fast and integrates well with existing software coverage tooling, but it is still simulator-specific, and users must learn Verilator’s flags and reporting pipeline separately. Other open flows attempt to infer statement coverage by post-processing waveforms (VCD/FSDB), but this is slow and memory-hungry for large designs, and results are fragile since waveforms only show signal values, not which HDL statement executed. + +From the software side, statement coverage is a direct parallel to tools like gcov, LCOV, or JaCoCo, which measure whether each line or branch in C, C++, Java, or Python has executed. This makes the concept familiar to many engineers, but unlike software, hardware IRs often lower high-level if/else into switch/case, so structural coverage must map back carefully to source constructs. + +Amaranth’s approach is to integrate statement coverage directly into its Python simulator observer framework, avoiding reliance on backend-specific flags or heavyweight post-processing. This provides a consistent API, immediate feedback during testing, and reports tied to hierarchical paths and source locations — all while remaining optional and keeping DUT/testbench code uncluttered. + +## Unresolved questions +[unresolved-questions]: #unresolved-questions +- **Statement types** At present only Assign and Switch/Case are tagged. Should additional constructs (e.g. assertions, covers, or more granular branch/path coverage) be included, or left for follow-up work? +- **Scope of implementation** Should this RFC cover only the Python simulator (current implementation) or also include Verilator integration? If not, how should future work on Verilator support be tracked? +- **Cross-run merging** Is the current global AGG_STMT_HITS sufficient, or should there be a more formal UCIS/LCOV-like database for merging across regressions? +- **Branch vs statement coverage** Should coverage distinguish between branches of conditionals (like if / else if / else) separately from generic switch_case hits, or is statement-level enough? + +## Future possibilities +[future-possibilities]: #future-possibilities +The next natural step is to integrate statement coverage with other coverage metrics such as block, toggle, assertion, unreachable code, and expression coverage. Statement coverage could eventually share a common reporting interface with these metrics, allowing users to collect, merge, and visualize different coverage types through a single entry point. This would make Amaranth’s coverage ecosystem more unified and easier to adopt. + +Another possibility is hierarchical aggregation, where reports summarize coverage not only per statement but also per module, instance, or subsystem. This would let users spot which parts of a design are poorly exercised without sifting through every individual assignment or case. + +Over time, statement coverage could also become more configurable. For example, reports might filter by domain (sync vs comb), include/exclude certain files or submodules, or allow adjustable verbosity (summary-only vs. full detail with hit counts). + +Ultimately, statement coverage could serve as a foundation for higher-level metrics. While the initial implementation focuses on recording executions of assignments and conditionals, future work could extend to branch/path coverage, standardized output formats (UCIS/LCOV), and integration with Verilator or other backends. This evolution would make statement coverage not just a basic completeness check, but a key part of a comprehensive verification framework in Amaranth. \ No newline at end of file