Skip to content

Formal verification methods#510

Merged
leynos merged 8 commits intomainfrom
formal-verification-methods
Mar 22, 2026
Merged

Formal verification methods#510
leynos merged 8 commits intomainfrom
formal-verification-methods

Conversation

@leynos
Copy link
Copy Markdown
Owner

@leynos leynos commented Mar 22, 2026

Summary by Sourcery

Document a recommended approach for integrating formal verification into Wireframe, including tooling choices, repository layout, and phased adoption strategy.

Documentation:

  • Add a comprehensive guide describing how to use Kani, Stateright, and Verus for formal verification in Wireframe, including priorities, repository structure, Makefile targets, and CI integration.
  • Extend the roadmap with a new milestone that tracks establishing formal verification tooling, defining key protocol contracts, and adding initial Kani, Stateright, and Verus checks.
  • Update the docs contents to link to the new formal verification methods guide.

leynos added 2 commits March 22, 2026 01:40
Add `docs/formal-verification-methods-in-wireframe.md` as the formal
verification strategy guide for Wireframe.

Sweep the document for style-guide conformance by removing personal
pronouns, expanding acronyms on first use, normalizing British English,
and converting references to numbered GFM footnotes in appearance order.

Update `docs/contents.md` and add roadmap work item `9.8` so the
workspace, Kani, Stateright, Verus, and CI follow-up tasks are tracked
explicitly.
@sourcery-ai
Copy link
Copy Markdown
Contributor

sourcery-ai Bot commented Mar 22, 2026

Reviewer's Guide

Adds a new formal verification roadmap item and comprehensive design document describing how to integrate Kani, Stateright, and Verus into the Wireframe project, wires that document into the docs contents, and leaves code behaviour unchanged.

Flow diagram for formal verification workflow in development and CI

flowchart TD
    Dev[Developer_edits_wireframe_code] --> DecideFormal[Decide_whether_change_affects_formal_targets]

    DecideFormal -->|No| RunStdTests[Run_standard_make_test_flow]
    DecideFormal -->|Yes| RunFormalLocally[Run_local_formal_targets]

    subgraph Local_formal_workflow
        RunFormalLocally --> L1[make_test_verification]
        RunFormalLocally --> L2[make_kani]
        RunFormalLocally --> L3[make_verus]
        L1 --> FixModel[Refine_connection_actor_model_or_properties]
        L2 --> FixKaniHarness[Adjust_Kani_harness_bounds_or_assumptions]
        L3 --> FixVerusProof[Update_Verus_specs_or_triggers]
    end

    RunStdTests --> OpenPR[Open_pull_request]
    FixModel --> OpenPR
    FixKaniHarness --> OpenPR
    FixVerusProof --> OpenPR

    subgraph CI_PR_pipeline
        OpenPR --> CIbuild[CI_job_build_test]
        OpenPR --> CIKani[CI_job_kani_smoke]
        OpenPR --> CIStateright[CI_job_stateright_models]
        OpenPR --> CIVerus[CI_job_verus_proofs]

        CIbuild --> PRStatus[PR_status_checks]
        CIKani --> PRStatus
        CIStateright --> PRStatus
        CIVerus --> PRStatus
    end

    PRStatus -->|All_pass| Merge[Merge_change]
    PRStatus -->|Any_fail| Investigate[Investigate_formal_or_test_failure]
    Investigate --> RunFormalLocally

    subgraph Nightly_pipeline
        NightlyTrigger[Scheduled_or_manual_trigger] --> NightlyFormal[make_formal_nightly]
        NightlyFormal --> N1[Deeper_Stateright_BFS_or_DFS]
        NightlyFormal --> N2[make_kani_full]
        NightlyFormal --> N3[Full_Verus_proof_suite]
    end
Loading

File-Level Changes

Change Details Files
Document a new roadmap phase for formal verification infrastructure and checks.
  • Introduce roadmap section 9.8 describing formal verification workspace setup, protocol contract decisions, Kani/Stateright/Verus checks, and their CI integration.
  • Specify concrete tasks for Kani harnesses on frame/fragment/message logic, Stateright models for ConnectionActor, and Verus proofs for message assembly invariants.
  • Clarify dependencies between verification tasks and preceding roadmap items.
docs/roadmap.md
Expose the new formal verification methods guide in the documentation navigation.
  • Add a "Formal verification methods" entry to the main docs contents list.
  • Describe the entry as recommended use of Kani, Stateright, and Verus in Wireframe.
docs/contents.md
Add a detailed design document describing how to adopt Kani, Stateright, and Verus in Wireframe.
  • Provide an in-depth analysis of current testing (proptest, loom) and where formal methods add value.
  • Define recommended repository layout: hybrid Cargo workspace, internal wireframe-verification crate for Stateright, Kani harnesses colocated with production modules, and a separate verus/ tree for proofs.
  • Specify concrete Kani, Stateright, and Verus targets (frame/fragment/message-series/message-assembler invariants and ConnectionActor models), including example harness shapes and proof obligations.
  • Recommend Makefile targets and CI jobs for running Kani smoke/full, Stateright model checks, and Verus proofs, plus phased adoption steps and anti-patterns to avoid.
docs/formal-verification-methods-in-wireframe.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 22, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b09826dc-2e20-47d0-8a19-6e814ea1f8b8

📥 Commits

Reviewing files that changed from the base of the PR and between 77dc3a7 and 4d59c14.

📒 Files selected for processing (1)
  • docs/formal-verification-methods-in-wireframe.md

Summary by CodeRabbit

  • Documentation

    • Added a comprehensive guide on formal verification methods covering Kani, Stateright and Verus, recommended tool layout and phased implementation plan
    • Updated the project roadmap with a new formal verification section detailing phases, verification tasks and CI/make targets
    • Added a contents entry linking the new formal verification guide
  • Chores

    • Updated .gitignore to exclude Zone.Identifier files

Walkthrough

Summarise the pull request: add formal verification planning and documentation (Kani, Stateright, Verus), extend the roadmap with formal verification milestones, and add a .gitignore rule to exclude Windows Zone.Identifier files.

Changes

Cohort / File(s) Summary
Configuration
\.gitignore
Added a pattern to ignore *:Zone.Identifier files (Windows alternate data streams).
Documentation Index
docs/contents.md
Added a Reference Guides entry linking the new formal verification methods document.
Formal Verification Plan
docs/formal-verification-methods-in-wireframe.md
Added a comprehensive new document outlining tooling (Kani, Stateright, Verus), target modules for checks, prerequisite protocol design decisions, suggested workspace/repo layout, Makefile targets, CI job proposals, and a phased rollout plan.
Project Roadmap
docs/roadmap.md
Inserted section 9.8 “Formal verification checks” with phased tasks covering workspace/tooling setup, protocol contract decisions, bounded Kani checks, Stateright models for ConnectionActor, and Verus proof targets.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Poem

Tidy the repo, mark the streams away,
Write harnesses where proofs will play,
Kani bounds, Stateright queues align,
Verus counts the invariants fine,
Ship the roadmap — let the models sway.

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title 'Formal verification methods' directly describes the primary change—a comprehensive guide for integrating formal verification into Wireframe with tooling, structure, and strategy.
Description check ✅ Passed The description clearly outlines all changes: adding a comprehensive formal verification guide, extending the roadmap with verification milestones, and updating docs contents to link the guide.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch formal-verification-methods

Comment @coderabbitai help to get the list of available commands and usage tips.

codescene-delta-analysis[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • In the example [workspace] configuration you reference a wireframe_testing member that doesn’t appear elsewhere in the doc or tree; consider either explaining what that crate is or removing it from the snippet to avoid confusion when people copy-paste it.
  • The new formal-verification guide hardcodes many GitHub main-branch URLs to files already in this repo; you may want to switch those to relative links so they keep working if files move or the default branch changes.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In the example `[workspace]` configuration you reference a `wireframe_testing` member that doesn’t appear elsewhere in the doc or tree; consider either explaining what that crate is or removing it from the snippet to avoid confusion when people copy-paste it.
- The new formal-verification guide hardcodes many GitHub `main`-branch URLs to files already in this repo; you may want to switch those to relative links so they keep working if files move or the default branch changes.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Remove the unexplained `wireframe_testing` workspace member from the
example workspace snippet so copy-pasted configuration stays aligned
with the surrounding guidance.

Replace hardcoded GitHub `main` links for Wireframe-owned files with
repo-relative links so the references remain valid if files move or the
default branch changes.
codescene-delta-analysis[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/formal-verification-methods-in-wireframe.md`:
- Around line 572-573: Replace the second-person phrase "depending on the exact
contract you choose" in the safety-property bullet with project-neutral wording
(e.g., "depending on the exact contract chosen" or "depending on the specific
contract selected") so the documentation avoids 2nd-person phrasing; update the
safety-property bullet text where that exact phrase appears to the neutral
alternative.

In `@docs/roadmap.md`:
- Around line 446-492: Update each 9.8 checklist item so it cites the exact
section in formal-verification-methods-in-wireframe.md that defines the design
scope: for headings "9.8.1. Establish the formal verification workspace...",
"9.8.2. Resolve the protocol contracts...", "9.8.3. Add bounded Kani checks...",
"9.8.4. Add Stateright model checks...", and "9.8.5. Add Verus proofs..." add a
parenthetical/link like “(See formal-verification-methods-in-wireframe.md §X.Y)”
pointing to the corresponding section; ensure the section numbers map to the
correct design subsections and update the checklist lines (e.g., the sub-bullets
under each 9.8.* item) to include that reference so each roadmap task directly
links to its source design scope.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: dcd62bc0-6d92-431a-b4d5-38ffb170289f

📥 Commits

Reviewing files that changed from the base of the PR and between 317a3a4 and 36ef8a9.

📒 Files selected for processing (4)
  • .gitignore
  • docs/contents.md
  • docs/formal-verification-methods-in-wireframe.md
  • docs/roadmap.md

Comment thread docs/formal-verification-methods-in-wireframe.md Outdated
Comment thread docs/roadmap.md
Replace the remaining second-person phrasing in the Stateright safety
property note with project-neutral wording.

Update roadmap item `9.8` so each checklist item points back to the
relevant section of `docs/formal-verification-methods-in-wireframe.md`,
keeping the implementation tasks tied to their source design scope.
codescene-delta-analysis[bot]

This comment was marked as outdated.

@leynos
Copy link
Copy Markdown
Owner Author

leynos commented Mar 22, 2026

@coderabbitai review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 22, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/formal-verification-methods-in-wireframe.md`:
- Line 974: Replace the phrase "anti-patterns" in the sentence "A few
anti-patterns are worth ruling out explicitly." with the single-word form
"antipatterns" to conform to the requested technical writing convention; search
for the exact string "A few anti-patterns are worth ruling out explicitly." and
update it to use "antipatterns" so the sentence reads "A few antipatterns are
worth ruling out explicitly."
- Around line 1-1089: The Markdown file's prose paragraphs exceed the 80-column
limit; reflow all narrative paragraphs to wrap at 80 characters while preserving
the original content, leaving headings (e.g., "Executive summary", "Kani
integration", "Stateright integration"), code blocks, inline code, tables,
lists, and reference footnotes untouched; ensure bullet points and list items
are wrapped at 80 columns as well and that no headings or fenced code blocks are
altered.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 91c9ca04-45dc-405f-a5c0-35c2e2c43898

📥 Commits

Reviewing files that changed from the base of the PR and between 36ef8a9 and 730b5f7.

📒 Files selected for processing (2)
  • docs/formal-verification-methods-in-wireframe.md
  • docs/roadmap.md

Comment thread docs/formal-verification-methods-in-wireframe.md
Comment thread docs/formal-verification-methods-in-wireframe.md Outdated
Use `antipatterns` in the formal-verification guide so the sentence
matches the requested technical writing convention.
codescene-delta-analysis[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/formal-verification-methods-in-wireframe.md`:
- Line 42: Update the British-Oxford spelling to use the -ize suffix in the
sentence "- **Loom** stays in place for real synchronisation interleavings on
queue" by changing "synchronisation" to "synchronization"; apply the same Oxford
-ize change to the other occurrences called out (lines referenced in the review:
794, 929, 1024) so all instances use "synchronization" consistently across the
document.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: e527ddaf-b061-4c7d-bb50-f695c1d5a21b

📥 Commits

Reviewing files that changed from the base of the PR and between 730b5f7 and f2d75af.

📒 Files selected for processing (1)
  • docs/formal-verification-methods-in-wireframe.md

Comment thread docs/formal-verification-methods-in-wireframe.md Outdated
Replace the remaining `synchronisation` spellings in the formal
verification guide with `synchronization` to match the requested
technical writing convention.
codescene-delta-analysis[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
docs/formal-verification-methods-in-wireframe.md (1)

793-795: ⚠️ Potential issue | 🟡 Minor

Normalize Oxford -ize spellings in remaining lines.

Replace the remaining non-compliant forms to match repository spelling rules.

✏️ Proposed edit
-1. `cargo test` is the better starting point for the Stateright crate because
-   it minimises the delta from current Wireframe practice. If the wider repo
+1. `cargo test` is the better starting point for the Stateright crate because
+   it minimizes the delta from current Wireframe practice. If the wider repo
@@
-That will give Wireframe the highest bug-finding return for the least
-organisational friction.
+That will give Wireframe the highest bug-finding return for the least
+organizational friction.

As per coding guidelines, "docs/**/*.md: Use British English based on the Oxford English Dictionary (en-GB-oxendict), including: suffix -ize ...".

Also applies to: 1023-1024

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@docs/formal-verification-methods-in-wireframe.md` around lines 793 - 795,
Replace non‑Oxford -ize spellings in the sentence starting "cargo test is the
better starting point for the Stateright crate because it minimises the
delta..." by changing "minimises" to "minimizes", and scan the rest of the
document for other -ise variants (e.g., any remaining "‑ise" suffixes referenced
in the review) and normalize them to the Oxford "-ize" form so all occurrences
(including the additional instances noted nearby) follow the repository
en-GB-oxendict rule.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@docs/formal-verification-methods-in-wireframe.md`:
- Around line 793-795: Replace non‑Oxford -ize spellings in the sentence
starting "cargo test is the better starting point for the Stateright crate
because it minimises the delta..." by changing "minimises" to "minimizes", and
scan the rest of the document for other -ise variants (e.g., any remaining
"‑ise" suffixes referenced in the review) and normalize them to the Oxford
"-ize" form so all occurrences (including the additional instances noted nearby)
follow the repository en-GB-oxendict rule.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 134ff5a3-3fe9-43f5-8cbf-e105cda230a9

📥 Commits

Reviewing files that changed from the base of the PR and between f2d75af and 2ba24b5.

📒 Files selected for processing (1)
  • docs/formal-verification-methods-in-wireframe.md

Update the formal-verification guide to use `minimizes` and
`organizational` so the wording matches the repository's spelling
convention.
codescene-delta-analysis[bot]

This comment was marked as outdated.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@docs/formal-verification-methods-in-wireframe.md`:
- Around line 218-262: Add a short prose description and a Mermaid diagram (with
an accessible short description immediately before it) that visualizes the
rollout and component split: show the main crate Kani harnesses, the
verification crate wireframe-verification (Stateright model), the verus/ proof
tree, and the phase ordering (e.g., build → Kani → Stateright → Verus) mapped to
CI targets; ensure diagram node/labels match the names "Kani",
"wireframe-verification", "verus/", "Stateright" and align the diagram text with
the listed phases and CI targets so readers and screen readers can correlate
phases to CI jobs. Apply the same addition to the other section mentioned (lines
935-970).
- Line 827: The workflow snippet uses the incomplete action ref string
"leynos/shared-actions/.github/actions/setup-rust@..." — replace the trailing
"@..." with a pinned commit SHA (preferred for production) or the placeholder
"<PINNED_SHA>" for documentation templates; update every occurrence of that
action ref (and any other refs using "@...") so the example workflows contain
explicit, auditable refs instead of the "@..." placeholder.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1d602121-414b-49e1-84b0-ecbe058ce55c

📥 Commits

Reviewing files that changed from the base of the PR and between 2ba24b5 and 77dc3a7.

📒 Files selected for processing (1)
  • docs/formal-verification-methods-in-wireframe.md

Comment thread docs/formal-verification-methods-in-wireframe.md
Comment thread docs/formal-verification-methods-in-wireframe.md Outdated
Document the formal-verification rollout with accessible Mermaid diagrams
in both the repository-layout and first-tasks sections so the crate split,
phase ordering, and CI targets are easier to correlate.

Replace the `@...` action placeholders in the workflow examples with the
documented `<PINNED_SHA>` placeholder so the examples use explicit,
auditable refs.
@leynos leynos merged commit 0630434 into main Mar 22, 2026
6 checks passed
@leynos leynos deleted the formal-verification-methods branch March 22, 2026 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant