v0.2.60

0.2.60

Everything since 0.2.59. For the changes from 0.2.46 on, in one page, see What changed from 0.2.46 to 0.2.60.

In short

• New, tested locally: an opt-in GitHub Actions workflow that replays one proof after each merge (opro install-github-action, opro actions-run).
• New: opro draft-test lets your own model write the new test from the brief, and checks it with opro verify-test.
• New: --static-only reads a repository and drafts tests without running any of its code.
• Page objects, fixtures and other test-support code leave the worklist, recognized by the test framework they import.
• The README's five evidence names are now the labels in rtm.md, the CLI, opro doctor, the MCP text and the agent docs. Tools that parse rtm.md need the new status names.
• The reports say what each number counts, and proof runs say what happened, time-outs included.
• @orangepro/mcp-server now pins its engine exactly.
• --proof-limit 0 means no proof attempt, and --generate-limit 0 no drafted test slots.
• The first run after upgrading is a full run: the report data format changed.

After a merge, in GitHub Actions (new, tested locally)

• opro install-github-action --profile-from <folder> shows the files it would write. With --yes it writes them:
  • .github/workflows/orangepro-proof.yml;
  • .orangepro/actions/, which holds the profile: which function, the approved expected behavior, the fault, the selected tests and the test to replay, plus the frozen fixture manifest and lockfile.
• The workflow:
  • asks for contents: read only and uses no secret;
  • pins every action to a commit and installs one pinned OrangePro version;
  • runs after a push to the base branch, or by hand for a named merged commit.
• The installer commits nothing and pushes nothing. A file you changed since it wrote it is a conflict and is never overwritten. --uninstall removes only what is still as written.
• The MCP tool orangepro_install_github_action does the same. It only previews unless you confirm. MCP tools: 19.
• opro actions-run is the workflow's step:
  • It reads the push or manual-run event.
    • It skips a first push, and a push in which every changed file is a test file a test patch may touch.
    • It blocks a rewritten history, an OrangePro build other than the one the profile names, and a credential in its environment.
    • It stops with "not configured for this change" when the change does not touch the profile's function.
  • Otherwise it runs changed-targets on the merged commit (this is the "not configured" check), fetches the pinned image if it is missing, and installs the frozen lockfile inside the container (network on for that step only; skipped when the test runner is already installed). Then it runs probe-fault, test-brief, verify-test with the profile's test, and export-tests. It stops early when the old tests already catch the fault, or when the profile names no test to replay. The whole run has one time limit; running out of time gives "inconclusive", never a result.
  • It writes result.json, the job summary and the run folders for upload. It opens no pull request.
• The container's platform follows the machine it runs on (linux/amd64 on GitHub's runners), with the same pinned image digest. A machine without the image fetches it by that digest first. On a Linux host the container runs as the host user, never as root.

opro draft-test: your model writes the new test (new)

• opro draft-test --from-run <probe-fault run> --brief <test-brief run> sends the brief's generation-request.json, and nothing else, to your model with your key.
  • The key comes only from that command's environment. Name the provider with --provider when more than one key is set.
  • With Anthropic the default model is claude-opus-5-5; --model picks another.
• It writes candidate.patch and assertion-record.json from the one test the model returns, and runs opro verify-test on them unchanged.
• Attempts: at most three per fault, each kept in its own run folder. A repair sees its earlier test and the verdict. It may change setup lines only, never the declared assertion.
• The key: nothing OrangePro writes may hold it, and it is removed from the environment before git or docker starts. A *_BASE_URL endpoint is printed and recorded only by its variable's name.
• Records: a test written this way is recorded as written by the provider and marked assisted.

Signed test reports in probe-fault, verify-test and export-tests

• The Mocha stages of probe-fault, verify-test and export-tests now read their results from one signed report.
  • The reporter signs that report with a key that is new for each stage. It reads the key from stdin before any test file loads.
  • A missing, extra or unsigned report, or a report file in the output folder, makes the stage report_tampered, and its results are not used. The result is inconclusive, never a pass or a catch. The one exception is probe-fault's optional whole-suite run: when it is affected, the claim stays limited to the selected tests.
• verify-test refuses a test that writes to anything it did not make, built-in prototypes included.
• A test written on purpose to get past these checks is not ruled out, so a person still reviews the patch.

Static-only mode (new)

• opro start --static-only, or ORANGEPRO_STATIC_ONLY=1, reads a repository and drafts tests without running any of its code.
  • Skipped: proof (both lanes), coverage runs (--generate-coverage is ignored) and the Go compile check. Coverage files already in the repository are still read, as data.
  • Go drafts are kept and marked "Not compile-checked".
• As a backstop, the proof runner, prove-loop setup commands, coverage runs and containers refuse to start in this mode.
• Use it for runs on code you do not trust, such as a free report on a public repository.
• For a run without --static-only, the variable or a 0 limit, these changes alter nothing: same graph, ranking, reports and reuse key.

Test-support code leaves the worklist by what it imports

• A JavaScript or TypeScript file that is not a test file, and imports a test framework, is now treated as test support. The frameworks are @playwright/test, playwright/test, @jest/globals, vitest, cypress, and k6 (k6 and any k6/… module, because k6 scripts are load tests). Page objects, test.extend fixtures and helpers built on them are the usual cases.
• Plain playwright does not count: product code can use the browser library. Subpaths such as vitest/config do not count either.
• Imports are read from the code itself. A framework name in a comment or a string never counts.
• What changes: these functions leave the primary worklist with the reason test_support_helper, as test support found by folder name (fixtures/, mocks/, e2e-testing/ …) already did.
• What does not change: they stay in the graph, and they stay counted in the behavior total. They are taken out after scoring, so no other function's score or order moves. The score formula is unchanged (orangepro.ors.stable_population.v3).
• When test support is more than half of the counted functions, both reports show one line saying so, with the numbers. This is common in an end-to-end test repository whose application lives elsewhere.
• When nothing is left to rank, the one-page summary says why: the functions are test support, or none is ranked. It says every function is proven only when something was proven.

tests.csv and the trace matrix agree on file-level links

• When a test file imports a function's file (or is named after it) but no test was traced calling the function, rtm.md already called it a Candidate, while tests.csv said "No test found". JavaScript and TypeScript rows now say Candidate link only, not confirmed, and name the test file to read first. The detailed report's row says the same.
• The verb stays Write, and the score, rank and evidence counts do not change. Go, Python and Java rows are unchanged for now.

The launcher pins its engine exactly

• @orangepro/mcp-server now depends on @orangepro/orangepro-mcp at its own exact version, not a ^ range, so it always runs the engine it names. The release script keeps the pin exact.
• Every run records the launcher version next to the engine version (see below).

One set of evidence names everywhere

• The trace matrix (rtm.md, and opro rtm --format csv and --format json), COVERAGE_REPORT.md, the opro start and opro rtm summaries, opro doctor, the MCP tool descriptions and the agent docs now use the README's five evidence names in their headings, labels and tables: Proven, Runs under tests, Linked, Unconfirmed, No test found. They replace "Dynamically Proven", "Runtime-covered", "Associated signal" ("Statically Linked" in the CLI), "Candidate signal (unconfirmed)" and "No integration signal". A few sentences still use the old words: "runtime-covered" in the opro start coverage line and in COVERAGE_REPORT.md, and "dynamic Proven" in COVERAGE_REPORT.md. "Reproven (this run)" and "Generated-unverifiable" stay. Both reports and tests.csv already used the five names.
• For tools that parse rtm.md: the RTM status strings changed (the Status column and the summary table). opro rtm --status takes the new names, the short keys and the old row names (Runtime-covered, Associated signal, Candidate signal (unconfirmed), No integration signal); associated and linked select the same rows. It does not take Dynamically Proven or Statically Linked: use Proven or Linked. A name it does not recognize is ignored, and if none is recognized every row is shown. The machine key evidence_tier (proven, runtime, associated, candidate, none) is unchanged.
• No tier assignment, row or count changes.

The reports say what each number counts

• The one-page summary's "Delete data" card says it counts every ranked or proven function whose code reaches a delete, not only the top of the worklist. The worklist's "Deletes data" filter shows its share of that total ("68 of 126").
• "Left out" is split by the reason the analyzer recorded (generated; not near an API, route, command or job; utility, model and tooling code; types, constants and other declarations; trivial accessors; CI and test helpers), and a new line shows the subtraction: found − left out = counted, plus any requirements read from Markdown docs. Before, the functions left out as not near an entry point were labelled "types and constants".
• The worklist heading names the done rows that the "Everything" filter counts. The summary's Prove / Check / Write counts say "of the 20"; the sidebar says its counts are for the rows in the worklist.
• Dates are the run's local time, with the zone named ("3 Nov 2026, 21:31 local time (UTC−08:00)"). The zone is saved with the report data, so a re-render on another machine shows the same time and the same bytes.
• A previous report less than an hour old is "the run 10 minutes ago", not "the report of ".
• The Inputs card counts the test files that call a counted function and those that call none ("1,022 test files: 69 call a counted function, 953 call none that OrangePro traced").

Proof runs say what actually happened

• A test whose run on the original code hit the time limit could not run: it is "needs setup" (category timeout), and the reason names the limit. It was filed as "does not protect" before. This holds in JavaScript/TypeScript, Go, Java and Python.
• A mutated run that hit the time limit is never Proven. It reads "ran past the time limit" in both reports and in the note opro doctor --proof prints for it, never "does not protect". (The text output of opro doctor --proof still heads that entry "mutant survived"; its --json says mutant_status: "mutant_timed_out".) Before, the JS/TS proof script could call such a run proven when the test had reported an assertion failure before the kill: nothing counted it as Proven, but the ledger record and the attempt record both carried mutant_status: "proven".
• A Go package that does not compile in the proof copy is called a compile failure, with the remedy (go mod download, or build the package once). "Sandbox setup gap" is said only when the sandbox itself could not start.
• With no model key, a run that tried the repository's own tests says proof ran and only drafting new tests was skipped. "Auto-prove skipped" is kept for runs where nothing ran.
• A run folder no longer fails its integrity check because macOS Finder or Windows Explorer added .DS_Store, Thumbs.db or desktop.ini. Any other file the manifest does not list is still refused.

Fixes

• --proof-limit 0 now means no proof attempt. Before, the CLI read 0 as "not given" and used the default limits: up to 5 attempts, or up to 20 for a Python repository's existing tests.
• --generate-limit 0 now means no drafts for the report's test slots. Before, the CLI read 0 as "not given" and drafted for the default top 20 functions, up to two each. With a model key, auto-prove can still draft a test in order to prove it, unless --proof-limit 0 is also given.
• The Python draft check runs isolated (python3 -I, from the draft's temporary folder). Before, a repository file named ast.py or py_compile.py in the working directory could run during the check.
• When the worklist is empty, the detailed report's worklist heading says so ("Nothing left to rank. 3 functions are done.", or "No ranked functions."), instead of "The 0 highest-priority of … ranked functions".
• The command both reports print for reproducing a run is now npx -y @orangepro/mcp-server@latest start ., the one the README gives.

Documentation

• The README says what a deterministic draft (provider: deterministic, no key) returns in each language. A placeholder checks nothing, so it can never prove anything. It is still marked runnable, with a run hint, whenever its checks pass: normally in Python, Go and Java, and in JavaScript and TypeScript when OrangePro can reuse a related test file's imports or derive one from a source file. The README on GitHub said none was runnable (from 4 Nov; never in a published package). Drafting a test with real assertions needs a model provider.
• "What OrangePro doesn't count" gains a line: in JavaScript and TypeScript, calls on an object a test gets from a fixture (for example Playwright's test.extend), and calls to an inherited method, are not linked.
• The README documents orangepro_prove's arguments, and explains why the trace matrix says "Reproven (this run)" for a first proof.
• The agent docs name only MCP tools the server registers.
• New: docs/reproducibility.md, on how to pin both packages, run, and compare two runs.

Output shape changes, for tools that read OrangePro output

• tests.csv: the columns are unchanged. Which functions get rows changes: test-support functions recognized by their imports no longer get rows. In a repository that is mostly end-to-end tests, the file can now hold only the header. For JS/TS rows with a file-level link only, "Evidence today" is Candidate link only, not confirmed (was No test found), and "What to do" is Write: No test calls it. <test file> imports its file; read it first. (or is named after its file), instead of Write: No test points here.
• report-data.json: REPORT_DATA_VERSION goes from 1 to 3. It gains timeZone (offsetMinutes, label: the zone the run happened in), counts.testFilesLinked and counts.testFilesUnlinked, worklist rows may carry proofTimedOut, and the "What was counted" card gains an Arithmetic row. It also gains an optional testSupport object (count, counted, text), present only when test support is most of the counted functions, and worklist rows gain an optional fileLink (the test file linked to the function's file). A run that finds an older file makes a full run instead of reusing it.
• graph.json: the analyzer identity is now orangepro.analyzer.v12, because more nodes carry ranking_exclusion_reason_code: "test_support_helper". artifact_identity gains launcher_version: the @orangepro/mcp-server version that started the run, or null when the engine ran directly (opro). It is in no fingerprint, so comparisons between runs are unaffected. The identity schema stays orangepro.artifact_identity.v1.
• rtm.md (and the RTM csv and json): the status names change, see "One set of evidence names everywhere". Rows, order and tiers are unchanged. Test-support functions keep their rows and their status.
• opro start --json: analyze.test_links (test_files, linked, none). With no key and existing tests tried, auto_prove.reason is the new drafting message.
• Proof records: a mutated run stopped by the time limit has mutant_status: "mutant_timed_out"; a baseline stopped by it is classified needs_setup with category timeout. Each language's proof script reports the limit it used as timeoutMs.

Requirements

Unchanged from 0.2.59: Node ^22.23.4, ^24.22.1 or >=26.12.1, on macOS or Linux. probe-fault, verify-test, draft-test (through verify-test), export-tests and actions-run run the repository's tests in a Docker container.