Skip to content

v0.1

  • v0.1
  • 2c34de1
  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
  • Choose a tag to compare

  • v0.1
  • 2c34de1
  • Choose a tag to compare

  • Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
@michaelwagstaff michaelwagstaff tagged this 10 Jun 14:05
* Add Spec27 eval GitHub Action with its own node CI

Initial import of the Run Spec27 Eval action, extracted from the spec27
monorepo (integrations/github-action/) so it can live alongside its own
node-based toolchain and CI rather than being exercised by a permanently
skipped subprocess test in a python+uv test lane.

The action starts a Spec27 eval run via the project-scoped API, polls until
the run reaches a terminal status, writes the final result JSON, and exposes
the key fields as action outputs. Inputs are base_url, token, eval_id, and the
optional poll_interval_seconds/timeout_seconds.

CI runs npm ci, the TypeScript build, and the node:test suite, and fails if the
committed dist/ bundle is out of sync with src/. spec27 owns and tests the HTTP
contract this action consumes; this repository owns and tests its parsing of
that response shape.

* Harden action output writing and poll resilience

setOutput now uses a random per-call heredoc delimiter
(ghadelimiter_<uuid>) and rejects names/values that contain it, matching
@actions/core. This prevents a value containing a delimiter line from
truncating itself or injecting/overwriting other action outputs.

pollEvalRun now retries transient fetch failures (network errors, HTTP
5xx/429, and a brief 404 immediately after a run is created) with linear
backoff, bounded by maxConsecutiveFailures and the existing timeout.
Clear client errors (401/403/400) and malformed-body responses fail fast
via a new typed Spec27ApiError that carries the HTTP status. Retries are
surfaced in the action log through a new warning() helper.

Adds tests for the delimiter hardening and the retry/fail-fast paths, and
documents the polling resilience in the README.

* Move to node24, keep workspace clean, and improve poll visibility

Runtime moves from node20 (end-of-life April 2026) to node24, with CI
and engines updated to match.

The results JSON now defaults to a unique file under RUNNER_TEMP named
by eval-run ID, so the action never dirties the caller's checkout and
repeated uses in one job cannot collide. A new results_path input
overrides the location when a workspace path is wanted.

Polling is no longer silent: the action logs the created run ID, result
URL and polling cadence at start, every status transition while waiting
(which also surfaces unrecognised statuses), and writes a job-summary
table with the status, accuracies and result link on completion.

Each HTTP request now carries a 30-second AbortSignal timeout so a hung
connection cannot stall the poll loop, and an unparseable base_url now
produces an error naming the input instead of a bare "Invalid URL".
Assets 2
Loading