Enable doctests (COMPLETE): add benchmark script and docs

[leynos]

Summary

• Enables Rust doctests as a first-class quality gate
• Adds doctest benchmark tooling and an execution script
• Updates docs and policy to reflect runnable/no_run balance
• Aligns doctest examples with compilation constraints by adjusting fences and explicit types
• ExecPlan status updated to COMPLETE; artifacts and progress documented

Changes

• Build and testing tooling
  • Cargo.toml: enable doctests for lib (doctest = true)
  • Makefile: added test-doc and doctest-benchmark targets
  • Added scripts/doctest-benchmark.sh
• Documentation
  • docs/execplans/enable-and-resolve-doctests.md: status updated to COMPLETE; progress items and artifacts documented
  • docs/rust-doctest-dry-guide.md: policy updated to reflect doctest workflow and benchmarks
• Doctest corrections
  • Adjusted doctest fences and explicit types across multiple modules to satisfy doctest runner (e.g., changed ignore to text, added explicit type annotations)
  • Examples updated in several modules to use concrete types for doctest stability (e.g., WireframeApp::new() with explicit type, etc.)
• Artifacts
  • Introduced logs and outputs related to doc preflight, doc pass, and benchmark runs (e.g., /tmp/doc-pre.log, /tmp/doctest-benchmark.log)

Test plan

• Run cargo test --doc --all-features to validate doctests locally
• Run make test-doc to execute all doctests with warnings enabled
• Run make doctest-benchmark to verify runnable/no_run balance against policy thresholds
• Expected results per plan: doctests pass (e.g., 179 passed, 0 failed, 4 ignored, 2 compile_fail)

Notes

• This PR does not introduce user-facing API changes. Changes are focused on doctest enablement, benchmark tooling, and documentation policy and guidance.
• Private/test-only helpers have been adjusted to use non-doctest fences where appropriate to maintain separation of concerns between user-facing docs and internal test helpers.

◳ Generated by DevBoxer ◰

---

ℹ️ Tag @devboxerhub to ask questions and address PR feedback

📎 Task: https://www.devboxer.com/task/05b7b8a6-295c-47bb-af96-c1521bdbc929

Summary by Sourcery

Enable Rust doctests as a enforced quality gate, add benchmarking for doctest coverage, and align documentation and examples with the new doctest policy.

Enhancements:

• Add Makefile targets for running doctests and benchmarking runnable vs no_run doctests.
• Introduce a doctest benchmark script to enforce minimum runnable and maximum no_run thresholds.
• Clarify and extend the doctest execution plan with progress, decisions, outcomes, and retained artifacts.
• Adjust server, client, builder, connection, session, and response documentation examples for stable doctest execution, including explicit types and corrected imports.
• Reclassify examples that require runtime context or are test-only helpers to use no_run or non-doctest text fences as appropriate.
• Update doctest ownership and balance policy in the Rust doctest guide to reflect Wireframe-specific workflow and commands.

Build:

• Enable doctests in Cargo.toml for the main library crate.
• Add test-doc and doctest-benchmark make targets to integrate doctests into the standard quality gates.

Documentation:

• Mark the doctest enablement exec plan as complete and document the achieved runnable/no_run ratios and quality gate results.
• Document project-level doctest policy, including how to run doctests and the benchmark, and where doctests should live in the public API surface.

Tests:

• Stabilize doctest coverage by converting suitable examples from no_run to runnable and ensuring all doctests pass across all features.

[sourcery-ai]

Reviewer's Guide

Enables Rust doctests as a quality gate, adds a doctest benchmark script and Makefile targets, and adjusts documentation and Rustdoc examples (including fences and explicit types) to satisfy doctest execution and runnable/no_run ratio policy while recording artifacts and updating project guidance.

Sequence diagram for running doctest quality gates

sequenceDiagram
  actor Developer
  participant Makefile
  participant Cargo as Cargo_test
  participant Rustdoc as Rustdoc_runner
  participant Script as DoctestBenchmarkScript

  Developer->>Makefile: make test-doc
  Makefile->>Cargo: RUSTFLAGS="-D warnings" test --doc --all-features
  Cargo->>Rustdoc: run doctests for public API
  Rustdoc-->>Cargo: results 179 passed, 0 failed, 4 ignored, 2 compile_fail
  Cargo-->>Makefile: exit status 0 or nonzero
  Makefile-->>Developer: report doctest success or failure

  Developer->>Makefile: make doctest-benchmark
  Makefile->>Script: ./scripts/doctest-benchmark.sh
  Script->>Script: rg list src/*.rs
  Script->>Script: awk classify doctest fences
  Script->>Script: compute runnable_pct and no_run_pct
  Script-->>Makefile: exit 0 if policy met
  Script-->>Developer: printed benchmark summary

Loading

Flow diagram for doctest-benchmark.sh policy evaluation

flowchart TD
  A[start doctest-benchmark.sh] --> B[set MIN_RUNNABLE=70 and MAX_NO_RUN=30]
  B --> C[collect rust_files with rg --files src -g *.rs]
  C --> D[awk scans rust_files for Rustdoc ``` fences]

  D --> D1[if info has no_run flag\nincrement no_run]
  D --> D2[if info has ignore or compile_fail\nincrement ignored]
  D --> D3[if info empty or starts with rust\nincrement runnable]
  D --> D4[else increment non_rust]

  D1 --> E[after processing all files]
  D2 --> E
  D3 --> E
  D4 --> E

  E[compute eligible = runnable + no_run] --> F{eligible == 0?}
  F -->|yes| F1[print error no runnable/no_run doctest blocks] --> Z_fail[exit 1]
  F -->|no| G[compute runnable_pct and no_run_pct]

  G --> H[print benchmark summary counts and percentages]
  H --> I{runnable_pct < MIN_RUNNABLE?}
  I -->|yes| I1[print error runnable ratio below threshold] --> Z_fail
  I -->|no| J{no_run_pct > MAX_NO_RUN?}
  J -->|yes| J1[print error no_run ratio exceeds threshold] --> Z_fail
  J -->|no| Z_pass[exit 0]

Loading

File-Level Changes

Change	Details	Files
Enable doctests and add dedicated commands for running doctests and enforcing runnable/no_run ratios.	Set [lib] doctest = true in Cargo.toml to enable doctests for the crate. Added test-doc Makefile target running RUSTFLAGS="-D warnings" cargo test --doc --all-features. Added doctest-benchmark Makefile target that invokes a new benchmark script.	Cargo.toml Makefile
Introduce a doctest benchmark script that inventories Rustdoc code fences and enforces runnable vs no_run thresholds.	Added scripts/doctest-benchmark.sh that scans Rust source files, classifies Rustdoc code blocks as runnable/no_run/ignored/non-Rust, and computes ratios. Script exits non‑zero when runnable percentage falls below 70% or no_run exceeds 30%, printing a summary of counts and percentages.	scripts/doctest-benchmark.sh
Update the doctest ExecPlan and Rust doctest guide to reflect the completed rollout, policy, and artifacts.	Marked the doctest enablement ExecPlan status as COMPLETE and filled in Progress, Surprises & Discoveries, Decision Log, Outcomes & Retrospective, and Artifacts sections with concrete counts and log paths. Documented final runnable vs no_run inventory, quality gate commands, and where benchmark and preflight logs are stored. Extended the Rust doctest guide with Wireframe-specific policy, including make test-doc, make doctest-benchmark, public-API-only doctest ownership, and benchmark thresholds.	docs/execplans/enable-and-resolve-doctests.md docs/rust-doctest-dry-guide.md
Stabilize server-side doctests by making type inference explicit and correctly classifying runtime-dependent examples as no_run.	Updated server configuration and runtime docs to call WireframeServer::new with an explicit closure return type `
Promote deterministic, side-effect-free examples to runnable doctests and add explicit typing where needed for generic APIs.	Changed various rust,no_run fences in session and extractor docs to rust where the examples are pure and deterministic. Added explicit type annotations in doctests (e.g., let _app: WireframeApp = WireframeApp::new()..., ConnectionActor<u8, ()>, Response::<u8, ()>::MultiPacket) to satisfy the doctest compiler. Adjusted builder and frame metadata examples to import and use the correct traits/types explicitly.	src/session.rs src/extractor/request.rs src/app/builder/protocol.rs src/app/builder/core.rs src/connection/mod.rs src/frame/metadata.rs src/response.rs src/server/runtime/backoff.rs
Ensure private/test-only helpers and internal runtime code use non-doctest fences to keep doctests focused on public APIs.	Converted examples in server test utilities and internal connection/runtime modules from doctest-capable fences (no_run/ignore) to text to avoid executing internal helper docs. Clarified imports and usage in these examples while keeping them as documentation-only snippets.	src/server/test_util.rs src/connection/frame.rs src/connection/state.rs src/server/runtime/accept.rs

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Summary by CodeRabbit

• Tests

  • Enabled doctests for the library, ensuring documentation examples are automatically validated.
  • Added benchmarking tool to monitor doctest coverage ratios and enforce documentation quality standards.

• Documentation

  • Updated code examples throughout documentation to be compatible with executable doctests.
  • Established project policy for doctest ownership and documentation example standards.

Walkthrough

Enable crate doctests in Cargo.toml, add Makefile targets and a doctest-benchmark script, update policy and execplan docs to require runnable/no_run ratios, and convert many inline doc examples to runnable Rust or plaintext with explicit type/closure annotations.

Changes

Cohort / File(s)	Summary
Doctest infra Cargo.toml, Makefile, scripts/doctest-benchmark.sh	Enable lib doctests; add test-doc and doctest-benchmark Make targets; add script that scans fenced examples and enforces runnable/no_run thresholds (70%/30%).
Policy & planning docs docs/execplans/enable-and-resolve-doctests.md, docs/rust-doctest-dry-guide.md	Mark execplan COMPLETE; add project policy wireframe specifying doctest ownership, commands, and benchmark acceptance criteria.
App & protocol examples src/app/builder/core.rs, src/app/builder/protocol.rs	Update documentation examples to include explicit WireframeApp type annotations.
Client & connection examples src/client/messaging.rs, src/connection/frame.rs, src/connection/mod.rs, src/connection/state.rs	Convert various fences from ignore/no_run to plaintext/runnable Rust; add explicit type annotations and an example assertion.
Extractor, frame & response examples src/extractor/request.rs, src/frame/metadata.rs, src/response.rs	Convert examples to executable Rust; add explicit use statements and type qualifications in patterns.
Server config examples src/server/config/binding.rs, src/server/config/mod.rs, src/server/config/preamble.rs	Standardise factory closures in examples to `
Server runtime & tests src/server/mod.rs, src/server/runtime.rs, src/server/runtime/accept.rs, src/server/runtime/backoff.rs, src/server/test_util.rs	Update doc examples: explicit closure return types, change fence languages, correct import paths, and expand await chains in examples.
Session & metrics examples src/session.rs, src/metrics.rs	Convert many no_run/text fences to runnable Rust or plaintext for non-executable snippets.

Sequence Diagram(s)

sequenceDiagram
  participant Dev as Developer
  participant Make as Makefile
  participant Cargo as Cargo/Rustdoc
  participant Script as doctest-benchmark.sh
  participant FS as Source Files
  participant CI as CI

  Dev->>Make: run `make test-doc`
  Make->>Cargo: run `RUSTFLAGS="-D warnings" cargo test --doc --all-features`
  Cargo->>FS: load crate doc examples
  FS-->>Cargo: provide doctest blocks
  Cargo->>Dev: report doctest results (pass/fail)
  Dev->>Make: run `make doctest-benchmark`
  Make->>Script: execute `scripts/doctest-benchmark.sh`
  Script->>FS: scan fenced code blocks
  Script->>Make: report runnable/no_run/ignored counts and enforce thresholds
  Script->>CI: exit non-zero on threshold violation
  Make->>CI: propagate success/failure

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• Fix footnotes and list formatting #211 — Overlaps documentation and doctest policy changes; strongly related to the guide updates.

Poem

🧪 Enable the tests, let examples run,
Annotate types until inference is done,
A script will tally each block in the land,
Keep runnable high and no_run well‑planned,
Celebrate docs that compile — clap your hand!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title clearly and specifically describes the main change: enabling doctests and adding benchmark scripting and documentation to support the new doctest quality gate.
Description check	✅ Passed	The PR description comprehensively explains the changeset, covering build tooling, documentation updates, doctest corrections, and the rationale for the changes.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch action-execplan-doctests-nz59h4

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The doctest benchmark script hard-depends on rg being installed and on src as the only search root; consider either checking for rg with a clear error or falling back to find, and making the search path configurable (e.g., via an argument or env var) if you ever want to include doctests from docs/ or other crates.
• scripts/doctest-benchmark.sh hard-codes the runnable/no_run thresholds; it may be more maintainable to read these from environment variables (with the current values as defaults) so that policy changes don’t require editing the script.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The doctest benchmark script hard-depends on `rg` being installed and on `src` as the only search root; consider either checking for `rg` with a clear error or falling back to `find`, and making the search path configurable (e.g., via an argument or env var) if you ever want to include doctests from `docs/` or other crates.
- `scripts/doctest-benchmark.sh` hard-codes the runnable/`no_run` thresholds; it may be more maintainable to read these from environment variables (with the current values as defaults) so that policy changes don’t require editing the script.

## Individual Comments

### Comment 1
<location> `docs/execplans/enable-and-resolve-doctests.md:129-131` </location>
<code_context>
+  `cargo test --doc --all-features` passes (`179 passed`, `0 failed`,
+  `4 ignored` + `2 compile_fail` checks).
+- Runnable-vs-`no_run` benchmark met:
+  runnable 128 (71%), `no_run` 51 (28%), ignored/compile_fail 6, non-rust/text
+  7.
+- Scope policy enforced for private/test-only helpers by converting their code
</code_context>

<issue_to_address>
**nitpick (typo):** Align capitalization of "Rust" in "non-Rust/text" for consistency with earlier usage.

In the Progress section this appears as `non-Rust/text 7.`, but here it’s `non-rust/text`. Please capitalize “Rust” here as well for consistency.

```suggestion
- Runnable-vs-`no_run` benchmark met:
  runnable 128 (71%), `no_run` 51 (28%), ignored/compile_fail 6, non-Rust/text
  7.
```
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[leynos]

@coderabbitai Have the following now been resolved?

Please address the comments from this code review:

## Overall Comments
- The doctest benchmark script hard-depends on `rg` being installed and on `src` as the only search root; consider either checking for `rg` with a clear error or falling back to `find`, and making the search path configurable (e.g., via an argument or env var) if you ever want to include doctests from `docs/` or other crates.
- `scripts/doctest-benchmark.sh` hard-codes the runnable/`no_run` thresholds; it may be more maintainable to read these from environment variables (with the current values as defaults) so that policy changes don’t require editing the script.

## Individual Comments

### Comment 1
<location> `docs/execplans/enable-and-resolve-doctests.md:129-131` </location>
<code_context>
+  `cargo test --doc --all-features` passes (`179 passed`, `0 failed`,
+  `4 ignored` + `2 compile_fail` checks).
+- Runnable-vs-`no_run` benchmark met:
+  runnable 128 (71%), `no_run` 51 (28%), ignored/compile_fail 6, non-rust/text
+  7.
+- Scope policy enforced for private/test-only helpers by converting their code
</code_context>

<issue_to_address>
**nitpick (typo):** Align capitalization of "Rust" in "non-Rust/text" for consistency with earlier usage.

In the Progress section this appears as `non-Rust/text 7.`, but here it’s `non-rust/text`. Please capitalize “Rust” here as well for consistency.

```suggestion
- Runnable-vs-`no_run` benchmark met:
  runnable 128 (71%), `no_run` 51 (28%), ignored/compile_fail 6, non-Rust/text
  7.
```
</issue_to_address>

[coderabbitai]

✅ This comment has been addressed.

[coderabbitai]

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@docs/execplans/enable-and-resolve-doctests.md`:
- Around line 94-96: Update the sentence containing "converted deterministic
state/extractor examples from `no_run` to runnable to restore the target
balance." (in docs/execplans/enable-and-resolve-doctests.md) to remove the
duplicated "to" and correct grammar and spelling to en-GB-oxendict by changing
it to "converted deterministic state/extractor examples from `no_run` to
runnable, restoring the target balance." Ensure the replacement preserves the
inline code formatting around `no_run` and `runnable`.

[leynos]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@docs/execplans/enable-and-resolve-doctests.md`:
- Around line 93-96: The observation sentence in
docs/execplans/enable-and-resolve-doctests.md currently has a duplicated "to" —
locate the sentence "converted deterministic state/extractor examples from
`no_run` to runnable to restore the target balance." and replace it with
"converted deterministic state/extractor examples from `no_run` to runnable,
restoring the target balance." to fix the grammar and comply with en-GB-oxendict
spelling/grammar.