Skip to content

Orbit 0.2.2

Choose a tag to compare

@github-actions github-actions released this 08 Oct 14:07
· 41 commits to main since this release
v0.2.2
1325ba5
  • Tests in C#, F#, Visual Basic, Java, Kotlin, Scala, Groovy, Rust, Ruby, PHP, Swift, C and C++, Elixir and Dart are recognised (#30, ADR 0011). Test files were recognised only for .js/.ts/.py/.go/.vue/.svelte, so a candidate that added xunit tests "added or changed no test": every criterion whose check already passed on the base revision was unverified with "no new evidence", and the run ended BLOCKED. One set of rules now serves the new-evidence rule, the weakening signals, the unexplained-architecture trigger and the review packet, and follows each language's test runner: a .NET source counts when the project that owns it on each revision is a test project by its project file (IsTestProject, or a Microsoft.NET.Test.Sdk, xunit, xunit.v3, MSTest or TUnit reference, or a legacy test framework Reference; nothing under a condition), so neither a test-named file of a production project, nor a test project added around production files, nor a library that references only NUnit or MSTest.TestFramework (which dotnet test does not run) counts; JVM sources count in a test source set (src/test/ and its Gradle, Android and Kotlin Multiplatform variants), not a helper source set or a source set's resources/; Rust counts a crate's tests/ and a #[test] function, not ignored, that the change adds to a source cargo compiles (a crate root or a declared module of the crate's src/), so a test beside a [workspace]-only manifest, in benches/ or in an undeclared file is not new evidence; the test attribute is judged by its whole attribute group in the candidate, so changing only the attribute of a test whose #[ignore] is unchanged (made async, or given another test_case row) is not new evidence either, and neither is a test in a comment or string, under a cfg other than cfg(test), or in a module such a cfg leaves out. The weakening signals count a file that was a test on either revision, so changing a test project's file does not hide the tests deleted beside it. The layout is read from the directories above the changed files only, so a repository whose full listing is larger than 8 MiB no longer loses its test projects.
  • A check the contract requires that the base revision had not run is run there before any change is judged (#32, ADR 0012). With checks.format.mandatory: false, which orbit doctor advised for a dotnet format check that cannot run in the macOS check sandbox, PREFLIGHT skipped the check, the planner cited it in an optional criterion (contract.check-added: format), and on the candidate it failed on the sandbox's refused named pipe. The base revision had no result for it, so the candidate gate (ADR 0010) could not tell the environment from the code: the run went to diagnosis and a repair on an identical tree and ended EXHAUSTED, unexplained. Now PLANNING runs every check the contract requires that the baseline has no result for on the base revision, before any attempt, and VERIFYING does the same for a check a contract amendment adds later (a baseline amendment, recorded in baseline.json and as a baseline.amended decision; its checkout installs the locked dependencies again rather than reusing PREFLIGHT's record, as a resumed PREFLIGHT now does too). Each is classified as PREFLIGHT classifies a mandatory check: an environment failure or a misconfigured check blocks the run there, with PREFLIGHT's reason and a sentence that says the contract requires the check, and orbit resume after an environment fix runs it again; a missing target is settled against the contract, while the contract requires the check (an approved amendment that stops requiring it leaves nothing to block on); a pre-existing failure gets its baseline-exception question, withdrawn when the contract names the check as a proof; a pass makes a failure on a candidate the change's. A check whose command cannot be started at all (an executable that is missing) has no output to classify: it is recorded without a result and run on the base revision again on every later pass, and blocks the run only on the candidate, as a mandatory check that cannot start does at PREFLIGHT. After a rebase in AWAITING_CI the recorded baseline is of the old base revision, so nothing is amended there. The baseline.amended decision is recorded once the amendment is judged, and an amendment cut short before that is amended and judged again on the next pass, so a pre-existing failure it found is still asked about. The planner may still name an optional check, and is told that naming one runs it on the base revision first. Doctor's fix for a dotnet format check that cannot run no longer offers mandatory: false (remove the check and run it in CI), and an optional such check warns that a run whose contract requires it would block at its baseline.
  • A repair attempt that ends on a tree an earlier attempt produced, which verification judges FAIL exactly as before, ends the run EXHAUSTED as non-progress at once (#32, ADR 0012), also when a later attempt returned to an earlier attempt's tree (a conservative stop: a further diagnosis would know more attempts, but the run has already gone round once on that tree). The same tree is the same candidate, whose recorded check results are reused, so the repair loop kept asking for a diagnosis and dispatching implementers that could not change the outcome, until the non-progress threshold or the attempt budget stopped it. A new tree, a changed judgement, a resume or orbit repair since the attempt started, or a repair session that stopped before it finished (max turns, a timeout, a failure, malformed output) keeps the loop, and an open material question blocks the run for its answer instead; the budget rules are otherwise unchanged. orbit repair on a run paused in DIAGNOSING now records the request too (run.repair-requested), which it did only when it moved the run there.
  • Accuracy and wording found by the 0.2.1 retest (#33, ADR 0013). Doctor: checks.sandbox no longer says that a run would block at its baseline for a refused check that is not mandatory, which PREFLIGHT does not run, but that a run whose contract requires it would (it runs there first, #32), whatever the sandbox refuses it for, and it no longer offers "set mandatory: false" to a check that has it (the fix for a dotnet format that cannot run is to remove it, #32); checks.dotnet-packages reports a cache that holds packages as filled, a pass with no fill command, and warns only for an empty cache or a floating version; the fix and the toolchain line no longer say the dependency install restores from, or fills, the cache when Orbit skips its own install ("no lockfile to install from", and npm ci restores no NuGet package), but say what does; claude.plugins says workers "would load" only a managed plugin and "may load" one of a scope it cannot place (such as synced, which sessions were seen not to load), and that a report lists the plugins its sessions reported loading and says nothing when they loaded none.
  • The CLI's closing line, orbit status and the report no longer advise orbit resume where only a new run helps: an environment block of a candidate's check with no baseline exception to approve, and a frozen-policy block with a question open (orbit status says "a new run is needed" and why, in place of "will return to VERIFYING"; for a frozen setting that a fix outside the policy clears, such as providers.<x>.model after orbit models refresh, it also names orbit resume --force, as the outcome reason does). The notification says the same. A run whose attempt counter is full is not one of these: the counter is charged when an attempt starts, so a run that blocks in the middle of its last attempt (an authorization to answer, an expired login) resumes into that attempt and finishes, and keeps "answer, then resume".
  • gate.baseline, baseline.recorded and baseline.json agree: a baseline in which a check never produced a result of its own (the environment stopped it, or its command was rejected) is incomplete in all three, so the gate is unverified where it was recorded as pass and the event said complete: true just before the run blocked. gate.baseline is recorded as not applicable when nothing ran on the base revision (no mandatory check, no locked install, no audit), where it said pass with "0 check(s)", and a baseline amendment the run goes on from records it again with the checks it ran. gate.ui is recorded as not applicable, citing no evidence, when nothing needs UI evidence, and its reason is not listed as "Not verified" (the evidence report, final.json and final.md, the pull request body and orbit verify leave it out: nothing was checked, and nothing was left unchecked).
  • A run that ended before any worker session (blocked at PREFLIGHT) starts no curator, and says why in learning.json; orbit report's cost includes the curator's, as status and the timeline do, because the final report is written again once it is charged.
  • orbit init says the template it writes sets providers.codex.data_policy_eligible: false, so Codex does not review until it is set to true, and no longer tells a configuration that already defines checks to define them, also when it fails validation for another reason (it then says to fix the listed problems; checks_defined in --json).
  • orbit status, the report and a succeeded run's notification name the candidate ref that exists (refs/orbit/<run>/candidates/<n>, candidate_ref in their JSON) in place of a branch orbit/<run> for a run that never delivered, where no such branch was made; branch is set only once delivery created it: by a local delivery whose gate passed (a refused one now leaves no branch behind, where the branch was made before the gate ran), by a successful push (a delivery that pushed and then failed at the pull request or the gate no longer reports "no branch"), or by a recorded delivery. The route reason of an unvalidated model says nothing has validated one yet instead of quoting the router's "no eligible claude-cli model" while choosing the model, and where a model is validated and the router still found none eligible (a tier it does not justify) it says the model was chosen although the router reported that, not that nothing is validated.
  • orbit doctor says when workers cannot run a repository's tests because they cannot listen (#31). In a .NET retest five of six implementers ran their own dotnet test -m:1, which built and then aborted with SocketException (13): Permission denied at Socket.Bind: VSTest listens on loopback for its test host, and neither worker tier lets a worker listen. The workers reported their tests as not run, and the checks, which may listen by their local_binding, ran the same tests. Letting workers listen was measured and refused: on macOS 27 with srt 0.0.78 and Claude Code 2.1.292, allowLocalBinding lets a sandboxed process listen on every address of the machine, and a listener on 0.0.0.0 or :: answered a connection made to the machine's network address. Seatbelt has no narrower rule: one for "localhost" admits 0.0.0.0 and the network address too, a numeric address is refused, and a connection's peer is never checked. Workers run model-driven commands, so on macOS a worker still may not listen, in either tier: the Claude worker settings now write sandbox.network.allowLocalBinding: false (held there by their schema), and the adapter hands srt none whatever profile it gets. On Linux every sandbox has a loopback of its own (srt's source always starts bwrap --unshare-net; not measured on Linux), so workers there can run these tests. orbit doctor (workers.loopback) warns on macOS for a .NET or Gradle repository that workers cannot run test hosts that need a loopback socket, in either tier, and names the test checks that run them. A check's local_binding is unchanged: on macOS its listener is not limited to loopback either, and it reaches services on any address of the machine over TCP and UDP, as since the first live run, now stated in docs/security.md with the measurements (ADR 0001, "Workers and loopback").
  • A worker cannot read the token of an IDE extension's MCP server (#31, review). VS Code's and JetBrains' Claude Code extensions write <config dir>/ide/<port>.lock with the token of their server on loopback, whose tools edit files in the editor and run code in a Jupyter kernel outside any sandbox, and Claude Code also looks in ~/.claude/ide when CLAUDE_CONFIG_DIR is set. While a draft of this change let workers use loopback, a worker read the token (its Bash in the os-sandbox tier, the Read tool in the claude-sandbox tier) and the server accepted it. A check cannot read the lock, since its profile denies every Claude config directory, but on macOS a check that may listen can reach loopback and runs the code a worker wrote, so a token the worker copied into the worktree would reach the server through it. The worker profile now denies <config dir>/ide for reading and writing and every other Claude login Orbit knows of (CLAUDE_CONFIG_DIR, ~/.claude) whole, and the worker settings deny Read on all of them in both tiers, so with CLAUDE_CONFIG_DIR set the Read tool no longer returns a token from ~/.claude/ide in the claude-sandbox tier; the guard hook refuses ~/.claude/ide too. A lock directory Orbit cannot know of (a config directory set only in the editor's environment) is not covered. The worker's own .claude.json stays readable to it (the CLI needs it), with any MCP server headers you put there (docs/security.md).
  • Background tasks are off for Claude workers (#31, review). claude -p does not exit after its result while a background task is alive, so a long command (run_in_background, or a foreground command the CLI moves to the background at its timeout) held the session until the role's time limit, which ended as a timeout and went to diagnosis though the work was done. CLAUDE_CODE_DISABLE_BACKGROUND_TASKS=1 removes run_in_background and ends a foreground command at its Bash timeout, and the implementer and diagnosis prompts say to start and stop a server within one command. A process a worker detaches itself (nohup, setsid) still outlives the session on macOS, with the worker's sandbox, so it can go on writing the worktree after the session (docs/troubleshooting.md).
  • orbit doctor says when workers cannot build or test a repository's toolchain (#31, review): workers.toolchains warns in the claude-sandbox tier for Go, Rust and the JVM, whose build state (GOCACHE, CARGO_TARGET_DIR, GRADLE_USER_HOME, Maven's local repository) Claude Code's sandboxed Bash cannot write there, so a worker's own go test or cargo test fails before any test runs while the checks pass. workers.loopback and workers.toolchains are reported only when the config could be read, like doctor's other config lines; workers.loopback reported a pass for the default config when .orbit/config.yaml was invalid.