Skip to content

Orbit 0.3.0

Choose a tag to compare

@github-actions github-actions released this 13 Oct 07:50
· 20 commits to main since this release
v0.3.0
a7c31a8
  • Tests: the baseline cancellation test cancels once its check is running, not after a fixed 800 ms, which on a loaded runner could land before the first check started; a new test covers a cancellation before any check starts. The container isolation tests now check how docker run ended before they read its output, so an engine failure is reported as docker's own error instead of an empty isolation result. The container test's CI failure has no established cause yet (#61). No change to Orbit itself.
  • Tests: the mock demo test reports the demo script's exit code and output when the script prints no JSON result, instead of failing on the parse with no clue (it did once on CI, PR #70).
  • Tests: the .NET runner test whose stand-in MSBuild exits 0 on SIGTERM now installs that handler before it writes the denial record. The runner stops a check as soon as it sees the record, so on a loaded CI runner the signal could arrive first and the check died by signal instead of exiting 0. No change to Orbit itself.
  • Claude workers without a Bash grant are no longer offered Bash. Planner, reviewer, inquisitor and curator prompts and the policy summary explain that these roles have no shell and use Read, Grep and Glob. Writer and experiment-worker grants and sandbox rules are unchanged (#67).
  • Worker reports record Claude's real per-session turn counts and their maximum, including pre-ledger sessions, restart replay and CLI cap overshoot. Missing counts are reported as unavailable. The existing policy worker_turns_per_session already reaches --max-turns; the curator now also obeys a lower policy cap. Historical demo reports no longer imply that unmeasured turns were zero (#67).
  • orbit init explains when no scope was derived, why it kept the template's example paths and that they must be edited; when no checks were derived it names the marker files it looked for. Both notices are warnings in --json, whose existing fields keep their shapes. A newly created config gets advice to review and commit .orbit/config.yaml to share the policy, while runtime state stays excluded and is never committed (config.commit_advice in --json); /orbit:init relays the advice (#62, #63, #66).
  • In a newly added test file, an import line whose only comment is one line-local directive naming only rules that report on that one line (import position or order, an unused import, missing type stubs; for example import style_check # noqa: E402) is no longer a lint-disabled weakening signal (#64). Every other suppression still is: a bare directive, one naming any other rule (such as jest/no-mocks-import, which can let a test import an assertion helper that does nothing), a next-line, block or file-wide directive (eslint-disable-next-line, @ts-expect-error, /* eslint-disable */, # flake8: noqa, now recognized), a second comment or statement on the line, Pylint and Go directives, # type: ignore[...] (Pyright ignores its rule list and silences the whole line), and any suppression in a new production file. Suppressions added to modified test files and suppressions on assertions in new files still report weakening (#64).
  • New tests are checked against the base revision (#65, ADR 0014). A Python run that asked for "unit tests that fail before the change and pass after" ended SUCCEEDED on its check log alone, with nothing showing the new tests failing without the change. Once a candidate is otherwise acceptable (every required check passed, or failed exactly as its approved baseline exception records) and changes a test, Orbit now runs each test check a criterion cites, and that passed on the base revision, on a probe tree: the base tree with the candidate's test-owned files (test files, test directories, Go testdata/, JVM test source sets with their resources, a .NET test project's folder and project file), after the locked install, a failing run once more, within one bound (verification.fail_before_timeout_seconds, default 900). Each check is failed-before, passed-before or not established, judged by exit status only, so a test that fails to load counts as failing and the report says "failed on the base revision with the new tests", never "assertion failed"; every line of every attempt's log, a passing one's too, is read for an environment signal (a long log in overlapping windows, and a signal anywhere decides), and only a log read whole in one pass of the classifier, with nothing dropped, can show failed-before; the probe's locked install keeps to the same bound (an attempt the bound cuts short is the budget's; an install that failed is never accepted; an install whose log shows an environment signal, or was cut short, stops the probe even when it exits 0). A criterion citing several probed checks weighs every open check's cause, in any order. A candidate that changes a dependency manifest or lockfile outside those files is not probed, also when a person authorized the change once. verification.fail_before sets what the outcome does: advisory (the default) reports it in each criterion's new fail_before entry and, when new tests passed on the base or could not be probed, in one unverified line; required supports a failed-before criterion (the review packet carries the probe log and the reviewer is asked whether the failure is an assertion or only a load failure), sends a passed-before one to repair and, once the attempts run out, to a person who may accept it without fail-before for that tree, and blocks one that is not established with the same question, except for an environment, ERROR or install cause, which is fixed and then rerun by orbit resume; off keeps the old rule and says the probe was not run. orbit run --fail-before <mode> raises the policy's mode for one run and is refused when it would lower it. Probe attempts are recorded in their own table (fail_before_probes), never in check_runs or failures, and a repair of production code only reuses them; each probe is a verification.fail-before decision that orbit timeline shows ("unit failed on the base revision with the new tests (exit 1, 2 attempts): AC-1"). Test evidence is also per check in every mode: a test change supports a criterion only through a check it cites whose directory holds the change, so a test added under another criterion's check no longer counts.
  • Rebased-baseline recovery tests give their no-op lab check enough time on a loaded runner, while still requiring exactly one successful check for the old and rebased base trees.
  • The demo 2 acceptance test (a difficult task that is diagnosed, escalated and repaired) is skipped without Playwright's Chromium, as scenarios 17 and 18 are. Its criteria are UI criteria, so without a browser every journey failed to launch and the run ended EXHAUSTED on any machine that had not installed one; installed claude or codex CLIs, the home directory and the environment do not change the outcome (#52).
  • Doctor now checks an alternative policy with --policy, presents failures before warnings as concise actionable lines by default (a failure's details still show, since its fix points at them), and reserves full plugin and sandbox detail for --verbose. It reports an insufficient wall-time budget before a run starts, and explains whether a pinned unavailable Codex model should be changed or may need a CLI upgrade. The wall-time floor is documented. orbit timeline --json now emits one JSON object per event with its durable event name; this replaces the single object with run, entries and cost, so a script that read those fields must read the event lines instead (#54, #55, #56, #57, #58).
  • A base-revision check that ends in ERROR after it starts, such as a process that disappears, now blocks as incomplete with the runner's cause and log. It cannot be accepted as a baseline exception; a real FAILED check keeps its pre-existing-failure question (#48).
  • A worker cancelled with SIGINT is recorded as cancelled even when the provider's exit is handled before the shim's own signal handler runs. cancelShim writes cancel.json before it signals the group, and the shim now reads that durable intent when it writes exit.json, so the event order no longer decides between cancelled and failed. A unit test forces the losing order; the Codex and Claude adapter tests that failed intermittently on CI pass 20 runs in a row.
  • A check that exits after MSBuild records a sandbox-refused worker-node pipe
    is no longer recorded as passed or as an ordinary exit-1 failure. The runner
    records its immediate exit scan for stopped-wrapper exit 0s and retains the
    completed-output scan for exit 1s whose report becomes readable afterward
    (#49).
  • On macOS, a worker process that creates another process group or session is
    now tracked from the provider tree and stopped before the shim writes its
    result. The sweep remembers pid/start-time identities, so it cannot signal a
    recycled pid, and result collection cannot race a detached child continuing
    to write the worktree (#34).
  • The installed plugin now includes its README.md, CHANGELOG.md and docs/ tree. The default build copies those files into the plugin/ payload, and the packaging checks require every source documentation path (#41).
  • The README, configuration and troubleshooting guides now identify agents.allowed_plugins and providers.codex.data_policy_eligible as policy edits a person, not an agent, makes. orbit doctor repeats that note when either is the unmet fix, including that an auto-mode classifier may flag worker launches (#42).
  • A contract-required command check that cannot start on the base revision is now retained as an ERROR baseline failure and blocks at PLANNING, or at VERIFYING for a later contract amendment, before an implementer session is started (#36).
  • A clean rebase while awaiting CI now measures all contract-required command checks on the new base before the rebased candidate can be verified. The fresh baseline replaces the old-base record and recovers safely if the controller stops after its durable write (#37).
  • Tests: the lab's scenario files are replaced, never rewritten in place. On the v0.2.2 release gate the parallel review test counted three reviewer sessions (review-security twice) for a run that succeeded. The security reviewer's fake started while the UI reviewer's scenario was being rendered into the same file (emptied, then written), read it empty, and exited 1 with no result; Orbit recorded that session FAILED ("exited 1 without turn.completed or turn.failed"), noted worker.regenerate, and started the unit's second attempt, as it does for any failed session. It was not a double dispatch: a unit's next session starts only after its earlier one is recorded as finished and unusable, one session per attempt, and a failed session records no review. Measured with a reader process beside the renders: a quarter of reads found the file torn before the change, none after. A review focus can now have steps of its own in a scenario (reviewer@security, reviewer@ui); a new test runs a failed security session through its regeneration (two reviews, from the second security session and the UI session, and three review rounds charged), and the parallel review test names each session and its error when a unit runs twice. No change to Orbit itself.