Skip to content

Releases: QuintinBotes/orbit

Orbit 0.3.1

Choose a tag to compare

@github-actions github-actions released this 16 Oct 01:35
v0.3.1
58539eb
  • Each reviewer is told the tools it really has (ADR 0016). On the second 0.3.1 retest every demo goal ended EXHAUSTED: a packet-only approval started the tree's fresh review, and the Codex reviewer, told by the shared role prompt "You have no shell; use Read, Grep and Glob" (tools only the Claude CLI has), looked for them through MCP tool discovery, opened nothing, and blocked with a finding no repair could answer. That prompt is also why Codex reviews never made a tool call. The role prompt now names no tools; the review task names the provider's own (read-only shell commands for Codex, Read, Grep and Glob for Claude); and a Codex review's depth counts only completed commands, so tool discovery is not an inspection and such a review blocks the run with "the review inspected nothing" instead of looping through repair.
  • The ui fail-before probe installs dependencies in its control and probe checkouts as the command probe does for its probe tree (ADR 0016). On the second 0.3.1 retest the demo's control ran npx playwright in a fresh base checkout with no node_modules, the sandbox refused the download (E403), and the probe recorded that the journeys do not pass on the base revision. A run whose output or application log shows a dependency could not be fetched or installed is now an environment cause, never a journey failure or a failing base, and is not cached; every install attempt's log is read for the same failure, for the command probe too, so an install behind a shell pipeline that exits 0 with an incomplete install stops the probe.
  • The fail-before probe decides statements about fail-before (ADR 0016). On the 0.3.1 retest the planner wrote AC-3 "The new tests import round_price and so fail on the base revision", marked supported by the candidate's passing test log, and AS-3 "The unit-tests check runs pytest over tests/ with src on the path", left unverified at success, although its brief forbids both. Such a criterion is now supported only when its fail-before outcome is failed-before; such an assumption only when each check it names (or every probed check, when it names none) failed before, and one naming a check that is not configured stays unverified. Unverified, it blocks under required and is a residual risk under advisory or off. Wording only selects these statements, and only to make Orbit stricter: negated statements, quoted or dotted check names, the word orders the reviews found and one strict passive shape ("the pytest suite is run by the unit-tests check", which the third 0.3.1 retest's planner wrote) are handled; a phrasing outside that grammar keeps the planner's own status, as before (ADR 0016, Limits). The evidence report records the assumptions it judged, so a changed assumption makes it stale, and the final report shows a probe-decided status only for the same statement.
  • Browser journeys get fail-before evidence of their own (ADR 0016). The probe ran only command checks of category test, so the demo's ui criteria read "ui is not probed". A journey check whose directory holds a journey the candidate adds or changes is now probed through the same ui runner and app fixture as verification, after a control run of the journeys on the base revision alone (once per run, since the baseline runs command checks only). A failure counts only after a passing control; a skipped journey is not a failure; a browser that never started, a process that never reached the repository's code, or an application Orbit did not start from the probe's checkout (no ui.environment.start_command) establishes nothing; and one deadline covers the control and every attempt.
  • A review that inspected nothing no longer clears the tree (ADR 0016). On the 0.3.1 retest every Codex review approved from the packet alone, with no tool call and a few hundred reasoning tokens, and the report could only flag it ("NO TOOL CALLS"). An APPROVE whose recorded depth shows no tool call now starts the tree's one fresh review at high effort, told to open the changed files and tests before its verdict; an approval from that review clears the tree, and if it inspects nothing either, or review_rounds is spent, the run blocks with the reason ("the review inspected nothing") and orbit status says a new run is needed. A review whose depth was not recorded is unaffected. The fake Claude and Codex CLIs in the test suite now look at the repository once per session, as a real reviewer does, unless a step says inspect: false.
  • The implementer works faster on what the 0.3.1 retest showed (ADR 0016). The demo's ui goal took 963 s, 11 minutes of it in one implementer session that ran at xhigh on predictions alone (a complex classification and a planned coupled change), hit its 30-turn cap and spent turns on refused commands. A prediction now raises an implementer's effort at most to high; repeated equivalent failures or critical security impact still raise it further. The implementer brief names the commands that were refused: inline code (node -e, python -c), which the session refuses even inside the sandbox (the retest's npm test 2>&1 | grep was refused for the node -e in the same call, not for the pipe), and reads outside the worktree. And usage.output-budget-exceeded now judges the final structured output, estimated from its JSON, against the budget it instructs: the session's 45,643 output tokens, every file it wrote included, were reported over a 24,000 budget the final output never came near.
  • The way to run without delivery, when a delivering run has no GH_TOKEN or no gh, now names every setting to change. On the 0.3.1 retest the demo's policy (autonomous-delivery, every delivery action on) was refused with "or set mode: autonomous to run without delivery", and following that alone was refused again: a delivery action left on is invalid in a mode that does not deliver. orbit run, orbit doctor and the controller's preflight now say "or, to run without delivery, set mode: autonomous and actions.commit, actions.push_task_branch, actions.open_pull_request and actions.repair_ci to false in .orbit/config.yaml", naming only the actions that are on, and the demo notes say the same.
  • orbit doctor no longer fails a check whose executable the dependency install creates. On the 0.3.1 retest the Python journey's check ran .venv/bin/python, which dependencies.install_command (python3 -m venv .venv ...) creates in each run's checkout, and every run's checks passed, but doctor reported "1 of 1 check(s) cannot run" because the file was not in the source checkout. A relative executable inside the repository, with an install command set, is now a warning that says the install may create it and that a run blocks at its baseline if it does not; doctor does not run the install. A bare name, an absolute path or a path outside the repository still fails, as does any check with no install command.
  • Orbit no longer sends AppleScript notifications on macOS (ADR 0008 addendum). The desktop channel is skipped there, with a detail that points to the webhook channel; the notifications appeared as Script Editor's. Linux notify-send and the notifications.desktop setting are unchanged.
  • A review finding a repair addressed is resolved by the repair's test only when the finding's own test tells the repaired tree from the one before it (ADR 0015). On the 0.3.0 retest a high test-quality finding (TEST-1, "the fail-before evidence is only a load failure") was marked resolved with "a test exercising the claim passes on tree ... (new_test lint PASSED ...)": the resolution took the first required check that passed on the repaired tree, which was lint. Crediting instead a test check whose log named the repair's test as passing was broken by three adversarial reviews (a check credited with a test its command excludes, a skipped test counted as run, a title matched inside another test's title, a sibling test's pass credited to a skipped one, glyph-only skip marks, a compiler warning that names the file), and Orbit does not read test runner output, so that rule is gone. Now each required command check of category test that passed on the repaired tree, whose directory holds a test the repair named for the finding, and whose latest run on the pre-repair candidate passed without a rerun, runs with the fail-before probe's machinery (ADR 0014) on three trees in one probe checkout: two built from the pre-repair candidate's tree, production code unchanged (a control tree with the repair's test-owned files in their repaired state except every test named for a finding, and the finding's tree, the control tree with the finding's tests), and the repaired tree itself; a failing run is repeated once, within verification.fail_before_timeout_seconds for each tree, whatever verification.fail_before says. The finding is resolved only when such a check passed on the control tree and on the repaired tree in the probe checkout, failed on the finding's tree on every attempt, with no environment signal and its logs read whole, and passed on the repaired candidate: exit statuses alone decide. A fourth adversarial review had a first version of this rule, one probe tree with all of the repair's test files, credit a skipped test named for one finding with another finding's failing test, with an unnamed edit to an existing test, with a test only a second check runs, and with a pre-repair tree that failed the check by itself; each now leaves the finding open, with "unit fails on the pre-repair tree without the finding's tests too, so its failure cannot be tied to them" or "unit is not probed: it did not pass cleanly on the pre-repair tree". A skipped test, a test the check's command does not run and a test that does not check the fix pass on both trees, so the finding stays open and blocking with "the repair's test passes on the pre-repair tree too, so it does not tell the trees apart"; a timeout, an ERROR, an environment signal, a log th...
Read more

Orbit 0.3.0

Choose a tag to compare

@github-actions github-actions released this 13 Oct 07:50
v0.3.0
a7c31a8
  • Tests: the baseline cancellation test cancels once its check is running, not after a fixed 800 ms, which on a loaded runner could land before the first check started; a new test covers a cancellation before any check starts. The container isolation tests now check how docker run ended before they read its output, so an engine failure is reported as docker's own error instead of an empty isolation result. The container test's CI failure has no established cause yet (#61). No change to Orbit itself.
  • Tests: the mock demo test reports the demo script's exit code and output when the script prints no JSON result, instead of failing on the parse with no clue (it did once on CI, PR #70).
  • Tests: the .NET runner test whose stand-in MSBuild exits 0 on SIGTERM now installs that handler before it writes the denial record. The runner stops a check as soon as it sees the record, so on a loaded CI runner the signal could arrive first and the check died by signal instead of exiting 0. No change to Orbit itself.
  • Claude workers without a Bash grant are no longer offered Bash. Planner, reviewer, inquisitor and curator prompts and the policy summary explain that these roles have no shell and use Read, Grep and Glob. Writer and experiment-worker grants and sandbox rules are unchanged (#67).
  • Worker reports record Claude's real per-session turn counts and their maximum, including pre-ledger sessions, restart replay and CLI cap overshoot. Missing counts are reported as unavailable. The existing policy worker_turns_per_session already reaches --max-turns; the curator now also obeys a lower policy cap. Historical demo reports no longer imply that unmeasured turns were zero (#67).
  • orbit init explains when no scope was derived, why it kept the template's example paths and that they must be edited; when no checks were derived it names the marker files it looked for. Both notices are warnings in --json, whose existing fields keep their shapes. A newly created config gets advice to review and commit .orbit/config.yaml to share the policy, while runtime state stays excluded and is never committed (config.commit_advice in --json); /orbit:init relays the advice (#62, #63, #66).
  • In a newly added test file, an import line whose only comment is one line-local directive naming only rules that report on that one line (import position or order, an unused import, missing type stubs; for example import style_check # noqa: E402) is no longer a lint-disabled weakening signal (#64). Every other suppression still is: a bare directive, one naming any other rule (such as jest/no-mocks-import, which can let a test import an assertion helper that does nothing), a next-line, block or file-wide directive (eslint-disable-next-line, @ts-expect-error, /* eslint-disable */, # flake8: noqa, now recognized), a second comment or statement on the line, Pylint and Go directives, # type: ignore[...] (Pyright ignores its rule list and silences the whole line), and any suppression in a new production file. Suppressions added to modified test files and suppressions on assertions in new files still report weakening (#64).
  • New tests are checked against the base revision (#65, ADR 0014). A Python run that asked for "unit tests that fail before the change and pass after" ended SUCCEEDED on its check log alone, with nothing showing the new tests failing without the change. Once a candidate is otherwise acceptable (every required check passed, or failed exactly as its approved baseline exception records) and changes a test, Orbit now runs each test check a criterion cites, and that passed on the base revision, on a probe tree: the base tree with the candidate's test-owned files (test files, test directories, Go testdata/, JVM test source sets with their resources, a .NET test project's folder and project file), after the locked install, a failing run once more, within one bound (verification.fail_before_timeout_seconds, default 900). Each check is failed-before, passed-before or not established, judged by exit status only, so a test that fails to load counts as failing and the report says "failed on the base revision with the new tests", never "assertion failed"; every line of every attempt's log, a passing one's too, is read for an environment signal (a long log in overlapping windows, and a signal anywhere decides), and only a log read whole in one pass of the classifier, with nothing dropped, can show failed-before; the probe's locked install keeps to the same bound (an attempt the bound cuts short is the budget's; an install that failed is never accepted; an install whose log shows an environment signal, or was cut short, stops the probe even when it exits 0). A criterion citing several probed checks weighs every open check's cause, in any order. A candidate that changes a dependency manifest or lockfile outside those files is not probed, also when a person authorized the change once. verification.fail_before sets what the outcome does: advisory (the default) reports it in each criterion's new fail_before entry and, when new tests passed on the base or could not be probed, in one unverified line; required supports a failed-before criterion (the review packet carries the probe log and the reviewer is asked whether the failure is an assertion or only a load failure), sends a passed-before one to repair and, once the attempts run out, to a person who may accept it without fail-before for that tree, and blocks one that is not established with the same question, except for an environment, ERROR or install cause, which is fixed and then rerun by orbit resume; off keeps the old rule and says the probe was not run. orbit run --fail-before <mode> raises the policy's mode for one run and is refused when it would lower it. Probe attempts are recorded in their own table (fail_before_probes), never in check_runs or failures, and a repair of production code only reuses them; each probe is a verification.fail-before decision that orbit timeline shows ("unit failed on the base revision with the new tests (exit 1, 2 attempts): AC-1"). Test evidence is also per check in every mode: a test change supports a criterion only through a check it cites whose directory holds the change, so a test added under another criterion's check no longer counts.
  • Rebased-baseline recovery tests give their no-op lab check enough time on a loaded runner, while still requiring exactly one successful check for the old and rebased base trees.
  • The demo 2 acceptance test (a difficult task that is diagnosed, escalated and repaired) is skipped without Playwright's Chromium, as scenarios 17 and 18 are. Its criteria are UI criteria, so without a browser every journey failed to launch and the run ended EXHAUSTED on any machine that had not installed one; installed claude or codex CLIs, the home directory and the environment do not change the outcome (#52).
  • Doctor now checks an alternative policy with --policy, presents failures before warnings as concise actionable lines by default (a failure's details still show, since its fix points at them), and reserves full plugin and sandbox detail for --verbose. It reports an insufficient wall-time budget before a run starts, and explains whether a pinned unavailable Codex model should be changed or may need a CLI upgrade. The wall-time floor is documented. orbit timeline --json now emits one JSON object per event with its durable event name; this replaces the single object with run, entries and cost, so a script that read those fields must read the event lines instead (#54, #55, #56, #57, #58).
  • A base-revision check that ends in ERROR after it starts, such as a process that disappears, now blocks as incomplete with the runner's cause and log. It cannot be accepted as a baseline exception; a real FAILED check keeps its pre-existing-failure question (#48).
  • A worker cancelled with SIGINT is recorded as cancelled even when the provider's exit is handled before the shim's own signal handler runs. cancelShim writes cancel.json before it signals the group, and the shim now reads that durable intent when it writes exit.json, so the event order no longer decides between cancelled and failed. A unit test forces the losing order; the Codex and Claude adapter tests that failed intermittently on CI pass 20 runs in a row.
  • A check that exits after MSBuild records a sandbox-refused worker-node pipe
    is no longer recorded as passed or as an ordinary exit-1 failure. The runner
    records its immediate exit scan for stopped-wrapper exit 0s and retains the
    completed-output scan for exit 1s whose report becomes readable afterward
    (#49).
  • On macOS, a worker process that creates another process group or session is
    now tracked from the provider tree and stopped before the shim writes its
    result. The sweep remembers pid/start-time identities, so it cannot signal a
    recycled pid, and result collection cannot race a detached child continuing
    to write the worktree (#34).
  • The installed plugin now includes its README.md, CHANGELOG.md and docs/ tree. The default build copies those files into the plugin/ payload, and the packaging checks require every source documentation path (#41).
  • The README, configuration and troubleshooting guides now identify agents.allowed_plugins and providers.codex.data_policy_eligible as policy edits a person, not an agent, makes. orbit doctor repeats that note when either is the unmet fix, including that an auto-mode classifier may flag worker launches (#42).
  • A contract-required command check that cannot start on the base revision is now retained as an ERROR baseline failure and blocks at PLANNING, or at VERIFYING for a later contract amendment, before an implementer session is started (#36).
  • A clean rebase while awaiting CI now measures all contract-required command checks on the new base before the rebased candidate can be...
Read more

Orbit 0.2.2

Choose a tag to compare

@github-actions github-actions released this 08 Oct 14:07
v0.2.2
1325ba5
  • Tests in C#, F#, Visual Basic, Java, Kotlin, Scala, Groovy, Rust, Ruby, PHP, Swift, C and C++, Elixir and Dart are recognised (#30, ADR 0011). Test files were recognised only for .js/.ts/.py/.go/.vue/.svelte, so a candidate that added xunit tests "added or changed no test": every criterion whose check already passed on the base revision was unverified with "no new evidence", and the run ended BLOCKED. One set of rules now serves the new-evidence rule, the weakening signals, the unexplained-architecture trigger and the review packet, and follows each language's test runner: a .NET source counts when the project that owns it on each revision is a test project by its project file (IsTestProject, or a Microsoft.NET.Test.Sdk, xunit, xunit.v3, MSTest or TUnit reference, or a legacy test framework Reference; nothing under a condition), so neither a test-named file of a production project, nor a test project added around production files, nor a library that references only NUnit or MSTest.TestFramework (which dotnet test does not run) counts; JVM sources count in a test source set (src/test/ and its Gradle, Android and Kotlin Multiplatform variants), not a helper source set or a source set's resources/; Rust counts a crate's tests/ and a #[test] function, not ignored, that the change adds to a source cargo compiles (a crate root or a declared module of the crate's src/), so a test beside a [workspace]-only manifest, in benches/ or in an undeclared file is not new evidence; the test attribute is judged by its whole attribute group in the candidate, so changing only the attribute of a test whose #[ignore] is unchanged (made async, or given another test_case row) is not new evidence either, and neither is a test in a comment or string, under a cfg other than cfg(test), or in a module such a cfg leaves out. The weakening signals count a file that was a test on either revision, so changing a test project's file does not hide the tests deleted beside it. The layout is read from the directories above the changed files only, so a repository whose full listing is larger than 8 MiB no longer loses its test projects.
  • A check the contract requires that the base revision had not run is run there before any change is judged (#32, ADR 0012). With checks.format.mandatory: false, which orbit doctor advised for a dotnet format check that cannot run in the macOS check sandbox, PREFLIGHT skipped the check, the planner cited it in an optional criterion (contract.check-added: format), and on the candidate it failed on the sandbox's refused named pipe. The base revision had no result for it, so the candidate gate (ADR 0010) could not tell the environment from the code: the run went to diagnosis and a repair on an identical tree and ended EXHAUSTED, unexplained. Now PLANNING runs every check the contract requires that the baseline has no result for on the base revision, before any attempt, and VERIFYING does the same for a check a contract amendment adds later (a baseline amendment, recorded in baseline.json and as a baseline.amended decision; its checkout installs the locked dependencies again rather than reusing PREFLIGHT's record, as a resumed PREFLIGHT now does too). Each is classified as PREFLIGHT classifies a mandatory check: an environment failure or a misconfigured check blocks the run there, with PREFLIGHT's reason and a sentence that says the contract requires the check, and orbit resume after an environment fix runs it again; a missing target is settled against the contract, while the contract requires the check (an approved amendment that stops requiring it leaves nothing to block on); a pre-existing failure gets its baseline-exception question, withdrawn when the contract names the check as a proof; a pass makes a failure on a candidate the change's. A check whose command cannot be started at all (an executable that is missing) has no output to classify: it is recorded without a result and run on the base revision again on every later pass, and blocks the run only on the candidate, as a mandatory check that cannot start does at PREFLIGHT. After a rebase in AWAITING_CI the recorded baseline is of the old base revision, so nothing is amended there. The baseline.amended decision is recorded once the amendment is judged, and an amendment cut short before that is amended and judged again on the next pass, so a pre-existing failure it found is still asked about. The planner may still name an optional check, and is told that naming one runs it on the base revision first. Doctor's fix for a dotnet format check that cannot run no longer offers mandatory: false (remove the check and run it in CI), and an optional such check warns that a run whose contract requires it would block at its baseline.
  • A repair attempt that ends on a tree an earlier attempt produced, which verification judges FAIL exactly as before, ends the run EXHAUSTED as non-progress at once (#32, ADR 0012), also when a later attempt returned to an earlier attempt's tree (a conservative stop: a further diagnosis would know more attempts, but the run has already gone round once on that tree). The same tree is the same candidate, whose recorded check results are reused, so the repair loop kept asking for a diagnosis and dispatching implementers that could not change the outcome, until the non-progress threshold or the attempt budget stopped it. A new tree, a changed judgement, a resume or orbit repair since the attempt started, or a repair session that stopped before it finished (max turns, a timeout, a failure, malformed output) keeps the loop, and an open material question blocks the run for its answer instead; the budget rules are otherwise unchanged. orbit repair on a run paused in DIAGNOSING now records the request too (run.repair-requested), which it did only when it moved the run there.
  • Accuracy and wording found by the 0.2.1 retest (#33, ADR 0013). Doctor: checks.sandbox no longer says that a run would block at its baseline for a refused check that is not mandatory, which PREFLIGHT does not run, but that a run whose contract requires it would (it runs there first, #32), whatever the sandbox refuses it for, and it no longer offers "set mandatory: false" to a check that has it (the fix for a dotnet format that cannot run is to remove it, #32); checks.dotnet-packages reports a cache that holds packages as filled, a pass with no fill command, and warns only for an empty cache or a floating version; the fix and the toolchain line no longer say the dependency install restores from, or fills, the cache when Orbit skips its own install ("no lockfile to install from", and npm ci restores no NuGet package), but say what does; claude.plugins says workers "would load" only a managed plugin and "may load" one of a scope it cannot place (such as synced, which sessions were seen not to load), and that a report lists the plugins its sessions reported loading and says nothing when they loaded none.
  • The CLI's closing line, orbit status and the report no longer advise orbit resume where only a new run helps: an environment block of a candidate's check with no baseline exception to approve, and a frozen-policy block with a question open (orbit status says "a new run is needed" and why, in place of "will return to VERIFYING"; for a frozen setting that a fix outside the policy clears, such as providers.<x>.model after orbit models refresh, it also names orbit resume --force, as the outcome reason does). The notification says the same. A run whose attempt counter is full is not one of these: the counter is charged when an attempt starts, so a run that blocks in the middle of its last attempt (an authorization to answer, an expired login) resumes into that attempt and finishes, and keeps "answer, then resume".
  • gate.baseline, baseline.recorded and baseline.json agree: a baseline in which a check never produced a result of its own (the environment stopped it, or its command was rejected) is incomplete in all three, so the gate is unverified where it was recorded as pass and the event said complete: true just before the run blocked. gate.baseline is recorded as not applicable when nothing ran on the base revision (no mandatory check, no locked install, no audit), where it said pass with "0 check(s)", and a baseline amendment the run goes on from records it again with the checks it ran. gate.ui is recorded as not applicable, citing no evidence, when nothing needs UI evidence, and its reason is not listed as "Not verified" (the evidence report, final.json and final.md, the pull request body and orbit verify leave it out: nothing was checked, and nothing was left unchecked).
  • A run that ended before any worker session (blocked at PREFLIGHT) starts no curator, and says why in learning.json; orbit report's cost includes the curator's, as status and the timeline do, because the final report is written again once it is charged.
  • orbit init says the template it writes sets providers.codex.data_policy_eligible: false, so Codex does not review until it is set to true, and no longer tells a configuration that already defines checks to define them, also when it fails validation for another reason (it then says to fix the listed problems; checks_defined in --json).
  • orbit status, the report and a succeeded run's notification name the candidate ref that exists (refs/orbit/<run>/candidates/<n>, candidate_ref in their JSON) in place of a branch orbit/<run> for a run that never delivered, where no such branch was made; branch is set only once delivery created it: by a local delivery whose gate passed (a refused one now leaves no branch behind, where the branch was made before the gate ran), by a successful push (a delivery that pushed and then failed at the pull request or the gate no longer reports "no branch"), or by a recorded delivery. The route reason of an u...
Read more

Orbit 0.2.1

Choose a tag to compare

@github-actions github-actions released this 07 Oct 11:58
v0.2.1
70554f3
  • A worker's, approved operation's or release command's private temp directory is checked before it is reused: a link or a file left in its place is removed, and a directory owned by someone else is refused, so the sandbox never grants writes through it.
  • Approved operations, the UI application and release commands get the toolchain profile a check gets (#26). Each built its sandbox from the check profile alone, so .NET there had no NuGet cache, no NIS domain name rule and no -m:1 handling. Measured under srt on macOS (SDK 9.0.305): an approved command and a deploy, each with a fresh private home, died at the SDK's first run on every dotnet command ('NuGet-Migrations', the mutex under /tmp/.dotnet); ui.environment.start_command: [dotnet, run, --project, Web] for a web project referencing two libraries was never ready ("(no output)" after MSBuild's node retries); and built with -m:1 first, its host hung at startup and its HttpClient died on GetDomainName: -1. Now each finds its toolchains from its command and the checkout as a check does: the repository's dependency caches read-only, build state private to the command, NuGet's audit decided by the hosts its sandbox allows, and for .NET a private home prepared as a check's, the NIS rule and IPv4 sockets on macOS, and the runner's early stop when MSBuild records a refused worker node, with a note naming the fix for that command's field. A deploy stopped that way fails (srt exits 0 when stopped, which no longer reads as deployed), a verify command whose build was refused leaves the deploy UNKNOWN, and an application stopped that way ends the UI run at once instead of after ready_timeout_seconds. A release command and an approved operation, which may fetch on the hosts their sandbox allows, write dependency caches of their own (removed after the command), with the repository's read-only beneath them where the tool reads a second cache: NUGET_FALLBACK_PACKAGES, a GOPROXY whose first entry is the repository's module cache, and Gradle's and Maven's read-only layers; Cargo and pip start empty there, as under the private home they had before, so a Rust or Python deploy needs its registry hosts in network_hosts. Each finds the host's rustup installation as a check does, in RUSTUP_HOME of the controller's environment, else in ~/.rustup of the account's home (also when the caller names no home, as the sandbox profile already assumed), so rustup's cargo proxy finds its toolchain under the command's private home: on GitHub's runners a deploy cargo install --path . --offline failed with rustup could not choose a version of cargo to run. With the check's read-only caches alone they could fetch nothing: under srt on macOS a deploy go run . that fetched one module from a proxy in the repository failed with go: writing go.mod cache: mkdir <orbit home>/toolchains/<key>/gomod/cache: operation not permitted, and a NuGet restore from a feed in the repository and cargo install --path . were refused the same way, which would have failed every deploy or verify command needing something the run's install did not fetch (a deploy tool in another toolchain, go run tool@version, a merge commit's new package). An approved command that is itself a package install with a toolchain it uses, such as dotnet add package, is the install for that toolchain and may write the repository's cache of that toolchain only, on the policy's hosts plus the approved one, and its output says whose cache was writable. An approved command Orbit stopped (a refused node, the run's cancellation, the time limit) has no exit code in its receipt, where a cancelled one read exit 0, and a command that exited on its own before Orbit's look no longer reads as stopped. A deploy's and a verify command's temp directory, and the UI run's, are emptied before each use, so a node report an earlier invocation left cannot decide a later one. The application keeps the account's Python user site-packages (no private PYTHONUSERBASE). orbit doctor (checks.sandbox) judges ui.environment.start_command and, in mode release, each release environment's deploy_command and verify_command with the .NET rules (-m:1, dotnet run in two steps, dotnet format), as mandatory, with argv-ready fixes (["sh", "-c", "dotnet build Web -m:1 && dotnet run --project Web --no-build"]). Every .NET process under srt on macOS also gets DOTNET_USE_POLLING_FILE_WATCHER=1 unless it sets the variable: the sandbox denies the FSEvents service a native file watcher asks, and ASP.NET Core's host hung on it (ADR 0009 addendum, items 14 to 16).
  • orbit run no longer starts a run whose worker sessions would all be refused for a plugin the policy does not allow (#22). orbit doctor failed claude.plugins for the managed plugins that the defaults (agents.allowed_plugins: [], agents.allow_managed_plugins: false) refuse, yet orbit run --foreground went on: about ten minutes of base-revision checks, baseline-exception questions and a planner worker, then two refused planner attempts and a refused curator (about 0.8 USD, nothing produced). Run start now makes doctor's judgement (one shared function, judgeWorkerPlugins): orbit run --foreground refuses in admission, before a run row, a frozen policy, a check or a model call exists, with every plugin id and doctor's exact fix line (exit 4, "No run was created"); a run handed to the service (--detach) or resumed is judged by its controller at the start of PREFLIGHT, before the base-revision checks, and ends BLOCKED with nothing spent (no check, no question, no worker, no curator). Why two places and not one: only the process that drives a run can judge its Claude Code environment, and a run that exists has already frozen the policy the fix would change, so refusing before it exists is the clean outcome whenever this process drives it; a BLOCKED run that cost nothing is the fallback when only the service can judge, and its message says the way out (allow the plugin and start a new run, or remove the plugin and orbit resume --force).
  • A worker session that Orbit refuses after it started (a plugin the policy does not allow, an MCP server, another permission mode) is not transient and is no longer retried as a worker attempt (#22): the planner, the reviewer and diagnosis units, and the implementer (also a parallel work unit) end the run BLOCKED after that one session. The outcome line carries the whole refusal, naming every plugin and the config line that allows it (it was cut at 200 characters, mid-sentence), and says that orbit doctor shows the cause. A run that ended this way with no candidate does not start the curator, which would be refused in the same environment (worker_refusal in the run outcome, "refused" in learning.json).
  • orbit init in a linked git worktree says that the exclude file it wrote is shared (#3 follow-up). git reads only the common git directory's info/exclude, so the rules always went to the main checkout's .git, outside the worktree, which surprised a person who expected init in a worktree to change nothing outside it. Orbit keeps that shared write (one write covers every worktree, nothing is committed) and now names the file and states that every worktree of the clone shares it, both when it adds the rules and when they are already there. orbit init --json gains exclude_file (path and shared_across_worktrees, true in a linked worktree and false in a normal checkout; detected with git by comparing --git-dir with --git-common-dir); exclude keeps its path and added. A normal checkout prints what it always did.
  • PREFLIGHT classifies every check that fails on the base revision before it calls one pre-existing (ADR 0010), and only a usage error from the check's own direct invocation of a tool counts: the command's program (after a leading env assignment, env, an npx-style runner or python -m) is the tool, the command is not a shell chain or pipeline, and what the error names is what the command names. An argument error (MSBuild's MSB1001 outside dotnet test and MSB1008, go's undefined flag or unknown command, cargo's unexpected argument) is a misconfigured check: the run blocks at PREFLIGHT with the check, the tool's error line and checks.<id>.command, as a frozen-policy block whose advice names a new run, not orbit resume --force, which runs the same command again and so blocks again unless the tool changed outside the policy; before, it was offered as a baseline exception (#23). A missing target (MSBuild's MSB1003, MSB1009 and MSB1011, dotnet test's MSB1001 (a test platform's option such as --report-trx before the repository runs that platform), npm's Missing script for the script the command runs when npm ran no script before it (no > acme@1.0.0 test banner), pytest's file or directory not found and its unrecognized arguments with exit 4 (an option of a plugin or conftest.py not there yet, such as --cov without pytest-cov or -n without pytest-xdist, which a goal may add), dotnet's or cargo's unknown command, a script of the repository the shell cannot find) goes on to CONTRACTING: when the contract names the check as the proof of a criterion its question is withdrawn and it is expected to flip, otherwise the run blocks there as misconfigured, with advice of its own by cause (a goal meant to create the target says so in a new run, a target a tool provides that is not installed or restored yet, such as a cargo plugin, a dotnet local tool or a pytest plugin, needs a new run once it is, a wrong command is corrected in checks.<id>.command; never orbit resume --force, which reads the same recorded baseline and blocks again), before the contract is written or any other check's question is settled. orbit resume refuses that block (exit 5) with the same advice and the foreground summary points to it, instead of sending a person to fix `.orbit/config.yaml...
Read more

Orbit 0.2.0

Choose a tag to compare

@github-actions github-actions released this 06 Oct 14:52
v0.2.0
027e6ff
  • New orbit timeline <run-id>: the readable history of a run, one line per significant step in order with the local time. It shows state transitions with their reasons, routing and escalation decisions with their evidence, each attempt and candidate with its verification verdict, check results, the review outcome and reviewer, questions asked and answered, delivery actions, and the cost so far as measured and as charged. --follow streams a running run, --last <n> cuts it, --all adds housekeeping events and --json is for machines. orbit logs stays the raw output and its help now points to the timeline; /orbit:status <run-id> shows the status and then the last steps of the timeline.
  • orbit init proposes checks from what the repository declares, for the tools found on PATH: Node (package.json scripts lint, typecheck or a tsc config, test, build; npm, pnpm or yarn from the lockfile), .NET (dotnet build and dotnet test on the solution or the projects), Python (pytest, ruff or flake8, mypy when configured), Go (build, vet, test over ./...) and Rust (cargo build, test, clippy when installed). Each has a category, a timeout and a comment asking for review; a mixed repository gets all of them, a monorepo gets root-level commands, an existing config is never touched, and init prints what it proposed and why (checks.proposed and checks.not_proposed in --json).
  • orbit init no longer proposes folders that hold CI pipeline or build-system definitions (pipeline YAML with a top-level trigger, stages, jobs or extends template, Jenkinsfiles, GitLab CI, CircleCI) as scope.allowed_paths; it protects them, plus central build files such as Directory.Build.props, Directory.Packages.props, global.json, nuget.config and a root Makefile, in scope.protected_paths, and tells the person to narrow the scope to the goal (#4).
  • A linked git worktree is its own repository root (git rev-parse --show-toplevel): orbit init writes the worktree's .orbit/config.yaml with the worktree's branch as repository.base_branch, doctor, run and every other command use the worktree's branch, cleanliness and state, .git/info/exclude stays shared, and worker checkouts made from a linked worktree find its own git directory (#3).
  • Skills: /orbit:status, /orbit:doctor, /orbit:init and /orbit:inquisition are model-invocable, so an agent asked to use Orbit sees them; run, resume, repair and verify stay user-only and say that a person starts them; scripts/check-plugin.mjs enforces the allowlist (ADR 0006 addendum) (#2).
  • Skills: status and doctor pre-approve their own read-only orbit command with allowed-tools, so claude -p "/orbit:status" prints the status without a permission denial (#2).
  • Docs: README and installation say that orbit reaches the Bash tool's PATH only after /reload-plugins or in a new session, and what to use until then (#5).
  • Worker sessions with organisation-managed plugins no longer block every run (#9): agents.allowed_plugins (exact name@marketplace ids) and agents.allow_managed_plugins admit a plugin (default strict); a refusal names each plugin and the config line that allows it, every non-built-in plugin is recorded in the worker's result and the final report, and orbit doctor lists the plugins a worker would load.
  • A check the environment stopped on the base revision (a sandbox denial on a filesystem call outside its checkout, or a crash before any output) now blocks the run at PREFLIGHT with the first error line and a fix, and is never offered as a baseline exception; the same refusal on a candidate blocks without a repair, and a compile error or failing test in the output keeps the normal path (#10).
  • Checks get the .NET SDK's first-run settings (NuGet migrations marked done in the private home, DOTNET_CLI_HOME, telemetry, logo, certificate, tools path and workload check off), so dotnet build runs under srt instead of dying with EPERM on /tmp/.dotnet (#10).
  • orbit doctor (checks.sandbox) starts each check's executable in its sandbox with a harmless argument and reports a sandbox denial before any run (#10).
  • Linux: dotnet build checks run under srt with the .NET 9 and 10 SDKs. Checks set EnableSourceControlManagerQueries=false, so the build no longer reads git metadata that srt makes unopenable on Linux (an absent .gitmodules); and a write the sandbox refuses outside the checkout is recognised in its Linux form (EROFS, "Read-only file system") as well as macOS's EPERM, so orbit doctor and the baseline report it there too.
  • doctor lists every unmet reviewer prerequisite at once (login, data policy attestation, no qualified model), each with its own fix, and orbit init seeds the model registry and reads the Codex catalog so a fresh setup does not hit the second failure (#7).
  • Review falls back to a disclosed Claude review by default when no independent reviewer is usable: new review.providers (preference order, supported ids only) and review.when_unavailable: claude | ask | block, with the legacy keys mapped onto it; reports, decisions and orbit doctor say which reviewer was used and, for a same-provider review, why it is not independent (#8, ADR 0007).
  • review.when_unavailable: ask raises a material question and runs the same-provider review only after a person's yes (orbit decide), recorded as a decision (#6, ADR 0007).
  • Release automation. Pushing a v* tag runs .github/workflows/release.yml: it checks that the tag matches package.json, plugin/package.json, plugin/.claude-plugin/plugin.json and the version orbit --version reports, that CHANGELOG.md has a section for it and that the commit is on main; runs the full gate (typecheck, unit, integration, fault and acceptance tests, check:dist, check-plugin); builds an archive of the plugin/ payload with checksums and a GitHub artifact attestation (actions/attest-build-provenance); creates the GitHub release with that CHANGELOG section as the notes; and opens a pull request on QuintinBotes/claude-plugins that moves the orbit entry's ref to the tag. The last step needs the secret CATALOG_PR_TOKEN and is skipped with a notice without it. New scripts scripts/release-notes.mjs, scripts/check-release-versions.mjs and scripts/bump-catalog-ref.mjs, each with unit tests; the process is in CONTRIBUTING.md and the verification steps in docs/installation.md.
  • Notifications (ADR 0008). When a run ends or raises a question a person must
    answer, Orbit notifies through the new notifications policy section: a
    desktop notification (macOS osascript, Linux notify-send; on by default),
    a webhook (Slack-compatible text; the URL is read from the environment
    variable notifications.webhook.url_env names, must be https to a host in
    network.allowed_hosts) and a comment on the run's pull request or linked
    issue. Payloads carry the run id, state, a short redacted reason, the next
    action and question ids only. Each notification is sent once; delivery
    failures are recorded as events and never change the run.
    ORBIT_NOTIFICATIONS=off turns every channel off. New command
    orbit notify test.
  • Remote answers (ADR 0008). With notifications.remote_answers.enabled, a
    /orbit answer <question-id> <choice> comment on the run's pull request (or
    the linked issue) answers an open question, but only when the GitHub API says
    its author has write, maintain or admin permission when the comment is read.
    The answer is recorded like orbit decide, with the comment URL, author and
    permission; anything else is ignored and recorded. The service polls blocked
    runs and resumes them once no material question is open; without a service,
    orbit resume reads the comments first.
  • Toolchain sandbox profiles (ADR 0009): checks and workers that use Go, Rust, Python, the JVM or .NET (detected from the check's command and the repository's marker files) get the repository's dependency caches (GOMODCACHE, CARGO_HOME, PIP_CACHE_DIR, Gradle's and Maven's repositories, NUGET_PACKAGES) from <orbit home>/toolchains/<repo key>/, read-only, and their build state (GOCACHE, GOPATH, CARGO_TARGET_DIR, PYTHONPYCACHEPREFIX, java.io.tmpdir) in a private directory per check attempt or per worker. Only Orbit's dependency-install step writes the caches; nothing is shared between repositories and nothing points at your own caches. A check's env still overrides every variable.
  • A configured dependencies.install_command also reaches the registries of the toolchains its command and the repository use (for example index.crates.io and static.crates.io for cargo fetch --locked), not only the npm registry.
  • orbit doctor (checks.sandbox) adds one line per toolchain the checks or the repository use: whether it starts in the check sandbox and where its dependency caches live.
  • Toolchain profiles pass the host's absolute JAVA_HOME to JVM checks (macOS's /usr/bin/java stub needs it to find a JDK) and rustup's home (RUSTUP_HOME, else ~/.rustup) to Rust checks, both read-only and never into a container; orbit doctor starts each toolchain by its name, so a rustup proxy such as cargo is tested as itself.
  • A check's scratch is removed even after a tool made it read-only on Node 24, which reports that case as ENOTEMPTY.

Orbit v0.1.0

Choose a tag to compare

@QuintinBotes QuintinBotes released this 06 Oct 01:58
v0.1.0
e24af10

Orbit is an autonomous, evidence-driven engineering loop for Claude Code: it turns a goal into a testable contract, runs isolated workers, verifies every candidate against evidence bound to its exact tree, has an independent reviewer (Codex) clear it, and only then delivers. This is the first release.

Install

/plugin marketplace add QuintinBotes/claude-plugins
/plugin install orbit@quintinbotes

Then /orbit:init and /orbit:doctor in your repository. The plugin brings its only runtime dependency, the sandbox runtime (srt); see docs/installation.md for Codex, Playwright and terminal use.

Proven live

All three spec demos ran unattended against real Claude and Codex with real delivery to a demo repository:

  • Simple: SUCCEEDED on the routine tier (Sonnet) in one attempt, about 2.5 minutes, draft pull request opened.
  • Difficult: SUCCEEDED with an evidence-backed escalation to Opus (subsystem coupling), browser journeys under the sandbox, draft pull request opened.
  • UI: SUCCEEDED: CSV export with download journeys on desktop and mobile plus the existing accessibility and visual journeys, under the sandbox; draft pull request opened. Current models solve this goal on the first attempt, so the repair path for a UI defect is proven by acceptance scenario 17 (real Chromium, fake providers) rather than live.

The runs and their reports are public: QuintinBotes/orbit-demo and docs/demos/2026-10-06.

An end-to-end test as a new user (install, CLI journey, skills and hooks, safe stops, usability) found 39 defects in the first round and 20 more in the second; all release blockers are fixed with tests that failed first. The suite has more than 7,000 tests (coverage about 99% of lines) and runs on macOS and Linux.

Safety

  • Workers and checks run inside srt: a write allowlist, credential read-denials and host-filtered egress. Evidence never claims more than it shows: a candidate that changes nothing, or a criterion without new evidence, is not supported.
  • Browser checks on macOS run Playwright's bundled Chromium with two narrowly scoped Mach rules added to srt's profile (ADR 0001); on Linux the app and the browser share one sandbox.
  • Two independent adversarial security reviews (Claude and Codex) were run on the isolation changes; every finding is fixed.

Known limitations

  • UI exploration cannot run under srt on Linux yet; it stops with a clear reason (it is off by default).
  • With a ChatGPT login, the Codex reviewer runs in Codex's own read-only sandbox, where reads are unrestricted (recorded in every report); an API key moves it inside srt.
  • Visual baselines are per platform; Orbit never records them.
  • npm audit reports an advisory in node-forge, a dependency of the sandbox runtime.
  • Delivery and release modes need a GitHub token scoped to the target repository.

See CHANGELOG.md for the full list of changes and docs/testing-journal.md for every test round.