Skip to content

ironharness 0.6.0 — benchmark-honesty block (IH-14..17)

Choose a tag to compare

@cezman cezman released this 10 Sep 17:58
· 336 commits to main since this release
c39f50f

Second reliability block from the 2026-09-09 audit: the benchmark's verdicts and the convention 'transports always have timeouts and quotas' are now enforced and pinned by tests.

anticheat (IH-14)

  • A wait-serial answer that arrives stamped before the stimulus write is pre-printed output - the run fails. A firmware that exits before answering a single stimulus step while matching the patterns fails too.
  • Plant tasks carry mid-run ambient disturbances: a tuned open-loop constant power drifts out of band and fails; feedback controllers track through.
  • The anticheat suite is parametrized over every task - new tasks are covered automatically.

journal + solve (IH-15)

  • TaskResult carries a structured error_kind (none/infra/timeout/run); solve's early exit reads the kind, never the error text - firmware printing 'not found' no longer stops a campaign.
  • solve feedback shows the failure reason.

machine shim + honest blink (IH-16)

  • The unix target can run GPIO-style firmware through a harness-provided machine shim; every pin transition is logged with a timestamp.
  • New 'events:' scoring: transition count and every consecutive period must fit the declared bounds - printing the expected strings does not pass. Golden blink-unix included.
  • 'ironbench clean' removes old run artifacts, only from ironbench-marked directories.

io-core session (IH-12/IH-17)

  • Atomic transport registry and sandbox quotas (parallel opens/writes cannot slip past checks), robust close, IRONHARNESS_MAX_CONNECTIONS.
  • Deadline and rate limits applied to session transports by default: IRONHARNESS_TRANSPORT_DEADLINE (600 s default) and IRONHARNESS_TRANSPORT_RATE=max/window.
  • ALLOWED_HOSTS rejects IPv6 literals loudly instead of silently never matching.

Full suite: 327 passed / 23 skipped; two protocol subagent review rounds per PR.