Skip to content

Releases: NewMe505/SCIO

SCIO v0.9.0

Choose a tag to compare

@github-actions github-actions released this 30 Oct 17:20
c07a074

Added

  • scio dashboard [--root DIR ...] [--port 8765] (issue #68, ADR 0017): a live, read-only web view of every SCIO thread across workspaces, on 127.0.0.1 only. A threads list with the stop waiting for you and crashed threads; the build view (the milestone graph with the current step lit, feature runs with tier, status, iterations, escalations, cost and links to their threads, integration lines, the PR, the timeline, cost by role and by feature); the feature and bootstrap views. It reads the checkpoint DB with mode=ro connections, the cost ledgers and progress.json, reuses the scio report builders, and pushes changes over a WebSocket. Plain HTML/JS with no build step; the optional dashboard extra (FastAPI, uvicorn) installs it.
  • docs/OBSERVABILITY.md: the ways to watch SCIO (GitHub, an OpenTelemetry trace viewer with SCIO_OTEL_EXPORTER=otlp, the dashboard, the CLI views).
  • scio report --thread new-... reports a scio new thread (phase, product, revision rounds, bootstrap commit, cost by role, timeline) instead of crashing (I-108).
  • Always/Never guardrails with enforced guards (issue #61): ## Guardrails (Always / Never) at the top of docs/LESSONS_LEARNED.md, each rule citing its ledger id and naming its guard; check_guardrails in scripts/check_docs.py fails when a resolved ledger entry from I-104 on is cited by no rule or a named guard (symbol, test, check, hook) does not exist.
  • A push-gate Claude Code hook (scripts/hooks/push_gate.py, wired in .claude/settings.json): a git push waits while the branch's latest CI run is red or still running (SCIO_ACK_CI=<run id> pushes a fix on purpose), and a bare git stash is refused (issue #61, I-104); a git commit waits until the changed Python files pass ruff check and ruff format --check (I-106).
  • CLAUDE.md and CONTRIBUTING protocols 4 and 5: Always/Never rules each backed by a guard, and a bottleneck audit after every pass (issue #61).

Fixed

  • On the Claude Code worker backend a feature's python is the product's environment, with its approved dependencies, not SCIO's own: the worker turn's PATH starts with the product interpreter's directory (I-114).
  • The advisor-cap halt names the cap that fired: the feature's total, or the one call per step (I-115).
  • skip at a halted feature defers it to the next milestone instead of dropping it, and the release refuses while a feature in PRODUCT.md is not done; removing a feature from PRODUCT.md drops it (I-116, the owner's decision).
  • scio report on a thread whose run is still going no longer dies on the cost ledger's half-written last line: CostLedger.load_entries reads only complete lines (a complete line that does not parse still raises), and a worker transcript caught mid-write is read with its broken character replaced (I-127).
  • scio build --gh no longer fails at its first issue on a repository without the product-mode label: the GitHub client creates a missing label before it names it (an existing one is left as it is), and a GitHub refusal stops scio build with the error and how to resume instead of a traceback (I-113).
  • A hand edit of PRODUCT.md is no longer lost: the milestone's sync commit records it, a fresh feature attempt commits one made while the build waited, SCIO's own checkouts and resets carry it, and the next milestone of the same thread re-reads it (I-109).
  • A quality bar can no longer promise tools the stack's profile does not use: each profile names its fixed tools, the interviewer and the planner see them, and the normaliser replaces a contradicting line with the profile's fact, quoting the original (I-107).
  • A thread's recorded spend spans every CLI invocation: a resume carries the checkpoint's spend into the new process, so scio new and resumed scio build threads no longer report only their latest step (I-110).

SCIO v0.8.0

Choose a tag to compare

@github-actions github-actions released this 27 Oct 19:30
30da3b8

Added

  • scio upgrade --product DIR (issue #58): brings a product bootstrapped by an older SCIO up to the current product bar, additively and from the current scaffold for its own PRODUCT.md: a missing Dockerfile and .dockerignore, the README's marked run block (under its ## Run heading), [tool.coverage.run] for a python product, and the CI's README run step where the CI is still the old scaffold's; anything the product made its own is kept and reported. One marker commit; a clean tree is required; a second run does nothing.

Changed

  • The worker's run_app tool runs the app where the sandbox runs: inside a Docker sandbox it starts detached, its health is probed with the profile's command from inside, and it is stopped by pid, so it no longer refuses a non-local sandbox; an app that exits reports its real exit code. The app-start code moved from scio.product.acceptance to scio.aci.apps, which both share (issue #58).
  • SCIO's own commits, merges and tags carry its identity (scio) on the command line, so a product clone or a machine without a git identity builds and upgrades (I-104).
  • CI runs the test suite in parallel (pytest-xdist, twice the runner's cores); pytest-xdist joins the dev group (issue #58).
  • With --gh a product's release goes through a release PR, as SCIO's own do: release cuts the changelog on release/v<version> (with the last progress record), pushes it and opens the PR; release_merge_wait stops until GitHub shows it merged; release_publish tags the merge commit, so the tag is on main's history, and publishes the GitHub Release. Offline the release is unchanged (I-103). New state fields ReleaseRecord.pr_number and pr_url.

Fixed

  • A node-ts product can build in the Docker sandbox: the sandbox image carries Node 22 next to Python, each profile names a capability probe (sandbox_probe; node-ts: Node with TypeScript type stripping) that the readiness check runs inside the image, and scio build reads PRODUCT.md's stack before choosing the backend, falling back to local with the reason when the image cannot run the stack (I-101).
  • In GitHub mode a squash- or rebase-merged milestone PR no longer breaks the next milestone: after a confirmed merge scio build resets local main to origin's when only SCIO's own bookkeeping commits are local and origin's progress.json still records them, and halts with sync_failed (naming the commits) when a human's commit exists only locally (I-102).
  • The Node docker-live test skips on the plain Python image when no sandbox image is named, and still fails when a named image cannot run Node (I-105).

SCIO v0.7.0

Choose a tag to compare

@github-actions github-actions released this 27 Oct 16:02
857790d

Added

  • check_released_section_frozen in scripts/check_docs.py: the newest released CHANGELOG section must match what its tag shipped, which catches a branch's [Unreleased] entries that a merge across a release cut moved under the new version (I-089).
  • M11 coverage floor (issue #54, I-091): SCIO_COVERAGE_FLOOR (default 80) is filled into the profiles' coverage commands; the node-ts profile gains a coverage check (Node's built-in --experimental-test-coverage with --test-coverage-lines). A quality check marked verify: true (coverage in both stacks, typecheck in node-ts) also runs in every product feature's verify once the suite and lint pass; a miss is the failure kind quality, sent to the debugger and never escalated, so a feature meets the bar integrate holds the milestone to.
  • M11 release checks (issue #54): integrate adds a secret scan of the product's tracked files (prefixed token formats, private-key headers, and credential-named assignments whose value looks random; a line marked scio: allow-secret is skipped) and a release-files check (README, CHANGELOG, LICENSE, Dockerfile). Both are SCIO's own code (scio/product/release_checks.py), so they always run and a finding fails the milestone. Both scaffolds ship a Dockerfile and .dockerignore: the image runs the same command the acceptance runner starts (python -m <pkg> or node src/main.ts), with the approved dependencies installed; CI's docker-live job builds and runs all four (python and node, cli and http).
  • M11 README check (issue #54): the scaffold's README carries its run commands between <!-- scio:run --> and <!-- scio:run-end -->; integrate runs each one ([readme]; only commands that start with one of the profile's command words, no free-form shell) and the product's CI runs the same block in a README run commands step.
  • M11 release gate (issue #54): once no milestone remains, scio build runs release_check (every integrate check on main, acceptance included, and the version the product's own --version prints), stops at release_decision for a human (--decision release or stop; the stop is its own node, I-065), then release cuts the product's changelog (chore(release): v<version> [scio:release:v<version>]), tags v<version>, keeps <name>-<version>.tar.gz (a git archive of the tag, Dockerfile included, plus the build check's output when there is any) under .scio/sandbox/release/, and with --gh pushes the tag and publishes a GitHub Release with the artifacts (GitHubClient.find_release, ensure_release). Every step is an effect with a world probe, so a resumed release neither tags nor publishes twice; a failing check ends the build with release_failed; a version already tagged ends it with nothing to do; merge and stop still ends at done. New state ReleaseRecord; scio build prints the release line.
  • scio new --spec PRODUCT.md --dir DIR (M11): bootstrap from a given PRODUCT.md as written, with no interview and no planner call (the product graph starts at bootstrap when the spec arrives approved).
  • M11 product eval suite (issue #54): python -m evals.runner --suite product. A task (grader: product) carries a fixed PRODUCT.md (stack and interface pinned) and a hidden/ directory; evals/product_solver.py bootstraps it with --spec, builds every milestone to the release gate (answering merge, release and a green gate's proceed as the owner would, stop for anything else, every answer counted), then runs the hidden tests with SCIO's own interpreter from a scratch directory against the released product: APP_CMD and APP_DIR for a CLI, APP_URL for a server started from its ## Runtime. p001 (CLI todo with tags and JSON export) and p002 (REST todo API with a YAML store) pass on the Fake; a stricter hidden test fails the same build. Trial spend includes every feature's child harness (I-095).
  • p003 (M11): a todo web app (one HTML page with the list and an add form, http.server, a JSON store) graded in a real headless browser: the hidden tests use Playwright with Chromium (playwright in the dev group; python -m playwright install chromium; CI's checks job installs it, elsewhere the browser tests skip). The runner gains --task ID (repeatable) to run one product eval.

Changed

  • The http walking skeletons read HOST (default 127.0.0.1); the Dockerfile sets 0.0.0.0 so the container answers on its published port.
  • The coverage floor measures the product's own code, run inside the test process: the python scaffold sets [tool.coverage.run] source and omits __main__.py, the node check excludes tests/**. Both walking skeletons test their app in-process (node's src/cli.ts exports createApp()) and mark the process lifecycle (serve) with the stack's coverage-ignore comment. Products bootstrapped before this change (bookmarks_api, notes_service) counted their tests toward coverage and test through a child process, so their next scio build may stop at the honest floor (I-091).

Fixed

  • On the Claude Code backend a structured call whose answer Claude Code's schema check refused within the turn cap is re-asked once with the required keys and length limits spelled out; and a provider failure inside one feature ends that attempt (recorded, with its spend) and the build with feature_failed instead of a traceback, so the next scio build starts the next attempt (I-100).
  • A budget stop in the documenter's PR-body call, or at routing, is SCIO's graceful budget exit instead of an exception that ended a whole product build; a scan test keeps every model call in a node inside a BudgetExceeded guard (I-098).
  • On the Claude Code backend token caps scale by SCIO_CLAUDE_CODE_TOKEN_FACTOR (default 10), measured from the live product evals at about 380,000 billable tokens per estimated dollar, instead of the dollar factor 4, which made the token cap stop a healthy feature at about half its dollar cap (I-099).
  • Verify no longer accepts a feature whose selected tests all skipped: run_tests reports no tests ran: every selected test was skipped (a profile's skipped_only patterns: pytest's summary, TAP's counters), verify fails the feature with a hint that acceptance tests run only at integrate, and the acceptance run and the eval graders fail an all-skipped run too (I-097: the first live p002 build halted on it).
  • scio new's crash-resume probe for the bootstrap commit searched git log --grep=[scio:bootstrap] as a regex, a character class that matches nearly any commit message; it uses the fixed-string find_commit now, and a scan test keeps every --grep= in scio/ fixed-string (I-090).
  • scio build commits the install phase's manifest and lockfile changes (npm install --save rewrites package.json and writes package-lock.json) on the milestone branch as chore: install approved dependencies [scio:install:<M>:<node>], before the first feature's reset could delete the lockfile; a merged milestone no longer leaves a dirty tree (I-085). load's sync of progress.json with PRODUCT.md is committed the same way ([scio:sync:<M>:plan_milestone]) instead of being reverted by that reset.
  • The node scaffold no longer writes a non-registry dependency (file:, github:, a URL or a path) into package.json as a package of that name; the install phase writes it. Such a spec is also its own name on the approved list, so it only matches itself in install_deps (I-088).

SCIO v0.6.0

Choose a tag to compare

@github-actions github-actions released this 27 Oct 00:02
8bd6e38

Added

  • check_release_cadence in scripts/check_docs.py: the docs check fails when two milestones that docs/PRODUCT_MODE.md calls done are named only under [Unreleased] (I-086: M8 to M10 waited two weeks behind v0.5.0); CONTRIBUTING gains the release-after-every-milestone step and a merged-on-GitHub check before any post-merge step (I-087).
  • M10 live milestone on the Claude Code backend: notes_service (TypeScript, node:http plus zod) bootstrapped by scio new (the planner chose node-ts from the stack menu) and built through M1 (three features, one advisor escalation, npm install through the install phase, eslint, tsc typecheck and build, npm audit and black-box acceptance all green, merged locally), $4.61 estimated, nothing billed; the done criterion of issue #51 holds on the Fake and live. Open: the install phase leaves package.json and package-lock.json changes uncommitted (I-085).
  • M10 Second stack (issue #51, ADR 0014): a node-ts toolchain profile (configs/profiles/node-ts.yaml, TAP parsers in scio/toolchain/node_ts.py): Node 22 runs TypeScript directly, node --test is the runner, tsc, eslint and npm audit the quality bar (skipped while absent), npm install the install phase; one data-driven ProfileToolchain loaded from yaml with a StackParsers set per stack; protocol members for the product path (acceptance_dir, skeleton_command, command_words, app_target, syntax_check_argv, health_probe_argv, detect); Runtime generalised (env_dir, env_marker, optional create step and interpreter); profile detection from marker files (Settings.toolchain=auto is the default, RunContext.adopt_toolchain on the clone, the product load and the bootstrap); a node-ts scaffold (TypeScript walking skeleton with node:test smoke and acceptance tests, eslint flat config, CI with setup-node); the normaliser, acceptance runner, run_app, tool descriptions and Claude Code allow-rules speak the profile's language; the toolbox context's toolchain reaches edit_file and run_tests, and edit_file runs the profile's syntax check (module.stripTypeScriptTypes for TypeScript, I-083); planner prompt v2 lists the available stacks; resolve_command finds npm's .cmd shims on Windows (I-081). The Fake todo_api spec builds its first milestone in Python and in TypeScript (test_same_spec_builds_in_both_stacks); CI installs Node 22 for the Node-dependent tests.
  • M9 (issue #49), first slices: delete_file tool on every sandbox backend (Sandbox.delete; the coder and debugger have it; a new file rejected by the edit lint gate is now removed instead of left empty); the python profile declares a quality bar (checks: coverage with an 80% floor, wheel build, dependency audit) that integrate runs after the suite and lint, reporting a check whose tool is absent as skipped and failing the milestone on a real failure; coverage artifacts added to the profile's cache excludes; the Claude Code worker backend now refuses a non-local sandbox.
  • M9 runtime (ADR 0013): a per-product environment (.scio/sandbox/venv, a plain venv with SCIO's own environment linked through a .pth file on a local sandbox, the profile's test_tools installed on any other, I-074) created by the install phase with the network open for that call only (Sandbox.exec(network=True)) and the approved dependencies from PRODUCT.md installed; install_deps worker tool gated by that list (an unapproved package is refused with the escalation hint; context-aware tools via ToolDef.needs_context / ToolBox.context); run_app starts python -m <module>, waits for a local health URL or the exit, and stops it (local sandbox only); the python profile's runtime: block.
  • M9 live milestone on the Claude Code backend: bookmarks_api (http.server plus pyyaml) bootstrapped by scio new and built through M1 (four features, one advisor escalation, black-box acceptance against the started server, coverage passed, merged locally), $5.38 estimated, nothing billed; the done criterion of issue #49 holds on the Fake and live.
  • M9 Docker default (ADR 0013 decision 10): scio build runs in Docker when a daemon answers and the docker/sandbox.Dockerfile image (Python plus git) is there, and in the local sandbox otherwise or on the Claude Code worker backend, with the first ledger line saying which and what a local sandbox does not enforce (SCIO_PRODUCT_SANDBOX_BACKEND=auto|local|docker). DockerSandbox is one persistent container per sandbox on an internal network (no route out), python rewritten to the product environment, a timeout inside the container keeping it; exec(network=True) attaches an egress-proxy sidecar (scio/aci/sandbox/egress_proxy.py, CONNECT-only, Limits.network_allowlist) for the duration of the call, so the install phase reaches the registry hosts and nothing else. run_acceptance starts an http product inside a non-local sandbox and probes it from inside. CI gains the docker-live job that exercises all of it on the runner's daemon.
  • M9 acceptance (ADR 0013): PRODUCT.md gains a ## Runtime section (kind: cli|http, module, start arguments with {port}, health path) that the product planner fills and the normaliser checks; the scaffold ships tests/acceptance/ (black-box: a subprocess for a CLI, the health endpoint for an http product, which the scaffold now serves with serve --port); integrate starts an http product on a free port, runs the acceptance tests with APP_URL, stops it, and reports [acceptance] beside the other checks. The scripted Fake product todo_api (an http server with pyyaml on its approved list) runs through the whole milestone loop offline: install phase, two features, server started for the black-box tests, merge. The scaffold's test directories are packages, so unit and acceptance tests may share basenames (I-073).
  • Claude Code backend (ADR 0012, issue #47): SCIO's model work can run through the unmodified claude binary on the owner's Claude subscription instead of the Messages API. SCIO_PROVIDER_OVERRIDE=claude_code sends every tool-less role through ClaudeCodeProvider (one restricted, tool-less claude -p per call, --json-schema for structured outputs); SCIO_WORKER_BACKEND=claude_code sends every tool-using turn through ClaudeCodeWorker (one stream-json session in the sandbox root with per-role --allowedTools, then a typed summary on --resume), with the transcript rendered in the shape scio report reads and the usage folded into the ledger and budget by Harness.account. Verify, commits and the gate are unchanged. scio/loop/backends.make_worker chooses the backend at every worker call site; SCIO_CLAUDE_CODE_BIN names the binary.
  • On the claude_code backend every tier cap in budgets.yaml is scaled by SCIO_CLAUDE_CODE_BUDGET_FACTOR (default 4.0, the measured ratio of Claude Code's estimate to the API price, I-069); the binary's own budget stop is SCIO's graceful budget exit; a claude -p run that fails without a message is retried (I-068); over-long prose fields in a typed summary are trimmed rather than failing the turn (I-070). scripts/nightly_live.py --provider claude_code runs the nightly suites on the subscription, one trial at a time.
  • scio build --product DIR [--gh owner/name] [--decision "<text>"] and scio status --product DIR (Product Mode M8, ADR 0011): the milestone loop. The orchestrator takes the first open feature of PRODUCT.md; an Opus milestone_planner groups up to four into a milestone with a tier each (normalised to a consecutive prefix); a GitHub milestone and one issue per feature are created idempotently by title (offline: files under .scio/product/issues/); each feature runs through the task graph as a subgraph on its own checkpoint thread (build-<product>-<pk>-a<attempt>) on branch milestone/<n> with the tier pinned and its own budget; the profile's full suite and lint run on the branch; an Opus milestone_gate verdict that any failing check overrides; one milestone PR (cost, escalations, plan versus done, Closes #n) stops for the human merge; progress.json and NOTES.md are updated on main and the loop continues. A crash between features or after the PR resumes with no duplicated issue, commit or PR; a feature that halts stops for one decision (retry, skip, stop). merge and stop records the merged milestone and ends the thread; build threads are named per starting milestone (build-<product>-m<n>) and scio build starts a fresh one once the previous ended. New state ProductBuildState, FeatureRun, MilestonePlan; GitHubClient.ensure_milestone, ensure_issue, find_issue, pr_merged; Harness.child; scio report renders a product view for build-* threads; the product graphs are registered diagrams in ARCHITECTURE.md.

Changed

  • After scio new every feature in progress.json is todo, the walking skeleton included: the scaffold proves the toolchain and P1 is verified by its own feature run (ADR 0011 decision 1, the M7 open question).

Fixed

  • The local sandbox no longer sets RLIMIT_AS on POSIX: it capped virtual address space, which V8 reserves by the tens of gigabytes, and Node's TypeScript stripper died with "Cannot allocate Wasm memory" on CI (I-084); the CPU-time and wall-clock limits stay, and Limits.memory_mb is enforced by Docker.
  • On the Claude Code backend a --json-schema answer is a StructuredOutput tool call that Claude Code bounces with a schema error when it misses the schema; with --max-turns 1 the model had no turn to retry and the first live http milestone crashed error_max_turns in the explorer's summary (I-077). Schema calls (worker summary, provider structured) get three turns and a capped summary is re-asked once with the required keys named.
  • scio new --thread X --decision ... without --dir continued in a default directory instead of the one the thread was started in, so the first...
Read more

SCIO v0.5.0

Choose a tag to compare

@github-actions github-actions released this 12 Oct 09:40
1fbd463

Added

  • scio new --prompt "<idea>" --dir <path> [--gh owner/name] (Product Mode M7, ADR 0010): a product graph that interviews the user (two to four Opus questions), writes a normalised PRODUCT.md, stops for approval, then bootstraps a repository with the profile's scaffold (package with --version, smoke test, CI, README, CHANGELOG, LICENSE, .gitignore, NOTES.md, .scio/product/progress.json), runs lint and tests as ground truth and commits with a [scio:bootstrap] marker. Every human stop is an interrupt resumed with scio new --thread ID --decision "<text>"; bootstrap steps are idempotent effects. New roles interviewer and product_planner; new state ProductState.

Changed

  • Toolchain profiles (ADR 0009, Product Mode M6): every stack-specific command and parser (pytest argv and exit codes, ruff lint, syntax check, test-path rules, cache excludes, dependency files, error signatures, the prompt phrases that name tests) moved behind scio.toolchain.Toolchain, with the python profile in configs/profiles/python.yaml reproducing the pre-M6 behaviour byte for byte. Settings.toolchain selects the profile; test_toolchain.py guards the boundary.
  • scio run --tier pins the route as well as the budget (I-053); without --tier the router decides and a default budget is resized for the tier it picks (RunContext.apply_tier_budgets); explicit eval budgets are untouched.
  • Assumption F-8 ("targets are Python") superseded by ADR 0009; PROMPT.md's "No UI" non-goal narrowed to SCIO itself (ADR 0007).

Fixed

  • SCIO_SANDBOX_BACKEND, SCIO_SANDBOX_DOCKER_IMAGE, SCIO_SANDBOX_E2B_TEMPLATE, SCIO_EFFECTS_PATH and SCIO_RUN_LIVE_TESTS were declared and never read (I-054, I-059); make_sandbox selects the backend, RunContext.create honours the effects path, the live-test gate reads Settings.

SCIO v0.4.0

Choose a tag to compare

@github-actions github-actions released this 12 Oct 07:44
b01b180

Changed

  • Product Mode roadmap (docs/PRODUCT_MODE.md, M6–M11) and ADR 0007: a product orchestrator over the task graph, toolchain profiles for any stack, milestone approvals; two defects logged for M6 (I-053, I-054).
  • A passing verify whose diff is high impact (more than 8 files, dependency or schema files, a removed public symbol) now gets one advisor review before commit (ADR 0008): review_decision, a triage route from a passing verify, and an advise -> git_checkpoint edge for continue. Until now every advisor trigger required a failing verify, and no live run had ever called the advisor (I-056).

Added

  • t010_wide_rename (T1, pinned): format_cents becomes format_money(cents, currency) across nine caller modules; the correct diff passes verify and trips the impact detector, so the advisor reviews it. Fake script edits each caller in one whole-file edit because edit_file lints every edit (I-057).

SCIO v0.3.0

Choose a tag to compare

@github-actions github-actions released this 12 Oct 06:21
183ffe6

Changed

  • E2B is no longer a planned live tier: no account (I-029, wontfix). Local backend for own fixtures, Docker for untrusted repositories (ADR 0003).

Added

  • t008_snapshot_coupling: a formatting change that breaks a CLI snapshot test and a README example test outside the coder's filtered run; the first regression task built to make verify's full-suite step fail and the debugger run live.
  • t009_verify_only_lint (T1, pinned): an undefined name in a module no test imports; the coder's run_tests is green and only verify's lint step fails, so the debugger must run and wire the constant rather than delete the module (hidden tests check both). First live debugger run: one turn, $0.06 of a $0.34 trial, fix plus sweep. Fake scripts gained Step.debug_edits for the debugger's turn.
  • tier_pin: on an eval task runs that tier without consulting the router, which rated t008 T0 and skipped the nodes the task exists to exercise (I-050). RunContext.tier_pin; route records classifier_model="pinned".
  • Two realistic regression tasks: t006 (a change spanning the store module, the CLI and their tests) and t007 (a fixture whose own CI is red on ruff with the repo config and on pytest); hidden tests run the fixture's own lint rules. One live trial each passed ($0.35 and $0.40).
  • Suite cost cap: evals/runner --max-usd; the report records spend, cap and skipped tasks.
  • Nightly live suite: scripts/nightly_live.py (profile auth, cost cap, rows in docs/EVAL_HISTORY.md) registered as a Windows scheduled task on the dev machine with --max-usd 12; the GitHub nightly workflow gained the cap, parallel trials and auth-token secret support.
  • scio report tool signals, counted from the worker transcripts with no model call: calls and error rate per tool, repeated identical calls within a turn, view_file re-reads, and whether the documenter edited docs when code changed. The documenter's transcripts are now kept as log artifacts like the coder's.

Fixed

  • Eval trials leaked their temp directory on Windows (636 directories): read-only git objects and an open effect-log handle, both hidden by rmtree(ignore_errors=True). remove_tree clears the bit, RunContext.close() releases the handle (I-049).
  • Eval fixtures are copied without __pycache__ and tool caches, so bytecode left by running a fixture's tests in place can no longer become tracked content that shows up in the agent's commits (I-048).

SCIO v0.2.0

Choose a tag to compare

@github-actions github-actions released this 11 Oct 03:22
ca1d4e4

Fixed

  • Token budgets count billable-equivalent tokens (cache reads at 0.1x, writes at 1.25x) so prompt caching lowers the count it enforces (I-041).
  • Prompt caching now covers the conversation: a third breakpoint on the last message block makes each worker turn re-read its transcript at cache price (I-040, docs/IDEAS_TRIAGE.md).
  • A tool crash (e.g. view_file on a directory) is returned to the model as an error result instead of ending the run (I-032).
  • Budget pre-check reserves half of max_tokens instead of all of it; explorer loops capped at 6 model calls / 8 tool calls; small task budgets raised to the tier the live router picks (I-033, I-035).
  • tests_added wording: the tests that prove the features, existing or new; bug-fix tasks no longer loop on "add tests" (I-034).
  • T1 verify adopts the worker's reported tests when the plan's guessed test names match nothing (I-037); worker turns capped at 12 model calls and transcripts trimmed from 30k characters (I-038); the planner no longer emits exploration steps and normalize_steps drops no-op steps and covers missing features (I-039).
  • verify runs each reported test selector correctly: path::name node ids and file paths as positional pytest targets, bare words as -k; found by the first live Anthropic run (I-031).
  • pdoc no longer warns on PEP 695 generics: scio/prompt/repair.py and scio/loop/agent_loop.py drop the __future__ annotations import (I-008).

Added

  • evals/runner --parallel N: a task's trials run concurrently (independent sandboxes, checkpoints and budgets).
  • Live Anthropic acceptance recorded: t001 pass^5 = 1.00 (5 trials) via an ant auth login profile.
  • Live sandbox checks: tests/live/test_docker_live.py runs wherever Docker is available (CI runners included) and tests/live/test_e2b_live.py runs when E2B_API_KEY is set (I-016).

SCIO v0.1.0

Choose a tag to compare

@github-actions github-actions released this 10 Oct 22:21
44f1a3d

Added

  • Observability: per-run report (scio report), persisted cost ledger per thread, RunReport with cost by role, escalations and timeline.
  • A/B role configurations (evals/configs/*.yaml, SCIO_ROLE_OVERRIDES_PATH); eval runner --config; capability suite with scenario tasks (fixture, scenario, expect_pass); nightly evals workflow posting per-config tables.
  • evals/tune.py escalation threshold sweep; EscalationConfig enforces max_stall >= repeat_threshold - 1; ADR 0006.
  • Release workflow and scripts/release_notes.py; SCIO_FAKE_SCRIPT=1 offline demo mode for the CLI; the CLI runs the unified graph (router picks the tier).
  • README acceptance-status table; first real GitHub end-to-end run recorded.

Fixed

  • prepare_repo writes repo-local git excludes so test caches are never committed (I-028).
  • T1 and T2 tiers in one graph builder (scio/graph/build.py): scope, parallel explore (LangGraph Send, read-only), localize, plan (plan-owned test filters), post_plan (issue checklist), implement (one step per cycle), debug, advise, review; generated t1/t2 diagrams.
  • Escalation policy (scio/loop/escalation.py, scio/loop/impact.py): six signals, never-escalate list, per-step and per-run caps, advisor_cap exit with human interrupt; TriageDecision in state; EscalationRecord with trigger, brief, verdict, tokens, $ and outcome; advisor guidance injected into the next worker turn; ADR generation in the target repository.
  • ToolBox(role=...) on spans; WorkerOutput.sweep surfaced in the ledger; PR body carries the escalation log and reviewer notes.
  • Five real regression tasks (t001–t005) with fixture repositories, hidden tests and a data-driven ScriptedResponder; the runner reports advisor call and resolution rates from the graph.
  • Postgres checkpointer built with the serializer allow-list; the gate runs the docs check directly.
  • ADR 0005: escalation policy implementation.
  • Context layer (scio/context): artifact store, per-role prompt projections, size-triggered transcript trimming.
  • Loop layer (scio/loop): AgentLoop (tools until done, tool/model caps, typed WorkerOutput summary) and exit conditions.
  • T0 graph (scio/graph/tiers/t0.py): intake, route, prepare_repo, t0_loop, verify, git_checkpoint, document (docs commit, push, draft PR), gate with ground-truth override, pr_ready, merge_wait interrupt, halt. scio run / scio resume CLI.
  • WorkerOutput.tests_added; verify requires changes, feature tests, a passing suite and clean lint.
  • Eval runner drives the real graph for grader: tests tasks and grades with hidden tests; t001 fixture repository; ScriptedResponder for deterministic Fake runs.
  • Generated t0 Mermaid section in docs/ARCHITECTURE.md; LangGraph serializer allow-list for state types.
  • ADR 0004: loop and verification design.
  • State backbone (scio/state): the M0 schema as Pydantic models with reducers, WorkerOutput.sweep, error signatures stable across line numbers and paths, progress fingerprints and stall counting.
  • Graph runtime (scio/graph): checkpointer selection by DSN (memory, SQLite, Postgres), idempotent side effects keyed by hash(thread_id, node, intent) with a durable effect log and world probes, Mermaid rendering with doc sections checked in CI, graph registry.
  • Agent-computer interface (scio/aci): Sandbox protocol with local, Docker and E2B backends; repo_search, view_file, edit_file (parse- and lint-gated), run_tests (failures only), git_checkpoint; output truncation with hints; capability tags and the compile-time trifecta check; ToolBox dispatcher with tracing and a call cap.
  • GitHubClient (scio/harness/github): idempotent branch, issue-comment and draft-PR writes through gh, default-branch guard, write log.
  • Durability test: a 3-node graph with file and git side effects killed mid-run and resumed with zero duplicated effects.
  • scripts/gen_mermaid.py; check_docs.py now verifies generated graph sections.
  • ADR 0003: idempotent side effects and sandbox tiers.
  • Harness layer (scio/harness): ModelProvider protocol; Anthropic, OpenAI and deterministic Fake adapters; single retry layer with backoff and retry-after; BudgetGuard checked before every call; per-call cost ledger; prompt-cache breakpoint placement; OpenTelemetry GenAI spans with opt-in content capture.
  • Prompt layer (scio/prompt): versioned per-role templates (12 roles at v1); Pydantic → provider-safe wire schema; structured-output validate-and-repair loop (max 2 repairs).
  • configs/models.yaml (the only place model IDs live) and configs/budgets.yaml (per-tier budgets).
  • Eval skeleton (evals/): task loader, n-trial runner, pass@1 / pass^k / tokens / $ / latency / advisor-rate metrics, Markdown and JSON reports, three placeholder regression tasks.
  • CI (.github/workflows/ci.yml): ruff, mypy, pytest on the Fake provider, docs checks, regression evals, API docs artifact. scripts/check_docs.py, scripts/gen_api_docs.py.
  • ADR 0002: harness conventions.
  • Continuous-improvement protocols: CLAUDE.md, docs/ISSUE_LEDGER.md (I-001 to I-010), docs/LESSONS_LEARNED.md (M0 and M1 retrospectives), Debugger template v2 with systemic sweeps, PR template checklist.
  • Build spec and generation prompt (docs/PROMPT.md).
  • Research basis with sources (docs/RESEARCH.md).
  • ADR 0001: single writer, Sonnet 5 worker, Opus 5.5 planner/advisor.
  • Repository docs: README, CONTRIBUTING, issue and PR templates.
  • Milestone 0 architecture (docs/ARCHITECTURE.md): graphs per tier, State schema, agent contracts, directory tree, milestone plan, assumptions and open questions.