Skip to content

Releases: pranaysparihar/InfernoSIM

v4.0.1

Choose a tag to compare

@github-actions github-actions released this 07 Sep 21:15
b016141

InfernoSIM v4.0.1

Safety and recovery

  • Independent monitor assertions detect absent verdicts, false healthy status,
    stale response revisions, and bad request-side state estimates against a
    declared simulated effect ledger.
  • Bound, single-use approval assertions check matching arguments/tenant and a
    logical expiry window. Request predicates support synthetic identity and
    authorization fixtures. Compensation assertions require later,
    identity-matched rollback effects.
  • Shared call, recorded-token, and explicitly priced cost budgets fail on
    unknown usage. Costs are configured estimates, not provider invoices.
  • agent run/stress --restart-after-call N kills and restarts the explicit
    application at a simulated response boundary. Ledger/deduplication state and
    a private application checkpoint directory survive the restart.
  • Assertion coverage distinguishes exercised, violated, and unexercised paths;
    require_exercised makes absent safety behavior a release failure.

Deterministic exploration and regression gates

  • Explicit call-ID admission schedules and bounded pairwise fault enumeration.
    Every tool entry in a complete supported provider JSON envelope can be
    selected. Combined faults no longer erase earlier transport failures.
  • agent compare runs two explicit application argv arrays against the same
    corpus/cases/schedules and detects failed cases, missing assertions, and
    safety-coverage regressions. Reports include JSON, JUnit, SARIF, and HTML.
  • agent reduce finds a one-minimal fault set preserving a named violated
    assertion on repeated executions, with a hard execution budget.
  • MCP stdio correlates out-of-order responses to the correct runtime request
    IDs, preserves notifications, remaps cancellation IDs, and rejects duplicate
    in-flight IDs and incomplete transcripts. Already admitted work is distinct
    from a new call after cancellation.

Fixes, compatibility, and packaging

  • Missing effect identity/deduplication fields fail closed instead of collapsing
    unrelated effects into a shared key. Request identity hashing preserves large
    integer precision. JSON-RPC/tool errors cannot satisfy retry verification.
  • Updated golang.org/x/net to v0.56.0 (and x/sys to v0.46.0), removing
    GO-2026-5942 from the dependency graph. Injected HTTP waits honor cancellation.
  • Agent harness generation retains combined fault and schedule selections.
  • The release build targets the complete CLI package. Packaging is explicitly
    eight platform archives plus checksums; GitHub's two automatic source
    archives retain the established total of 11. Reports and internal JSON
    metadata are not release assets.
  • Existing agent configuration defaults, backend replay features, and encrypted
    bundle-v2 format remain compatible. New configuration fields are opt-in.

Validation and scope

Local validation includes the full Go race suite, vet, the existing safe/unsafe
agent smoke, the new recovery/safety/compare/reduce CLI smoke, official MCP Go
SDK v1.7.0 record/replay/reconnect integration, a Docker image build, a real
Testcontainers run, Kafka/AsyncAPI against Redpanda, and the installed Ollama
model smoke. The full suite was also run with the release toolchain Go 1.26.6.
After the dependency update, govulncheck reported no vulnerabilities in the
main-module and pinned SDK integration scans. The existing agent benchmark
passed 100/100 safe executions and rejected 80/80 unsafe controls across 20
iterations; the incident-to-test benchmark passed 100 runs with one stable
configuration hash, one stable harness hash, and zero detected fixture leaks.
The new crash/recovery example also passed inside a Linux/arm64 container
with external networking disabled. A local GoReleaser snapshot produced all eight platform
archives and their checksums verified. Cross-builds are not native Windows or
Intel macOS runtime tests.

These checks establish the behavior of the tested fixtures, not universal
agent safety, full MCP conformance, or superiority over competitors. Admission
schedules do not control arbitrary OS races. Approvals/credentials are local
test predicates, not real identity-provider enforcement. Process restart is
not power-loss recovery. Live model execution is separate from deterministic
recorded-envelope gates. Windows child termination does not contain descendants.

Start with the runnable safety/recovery guide and
the upgrade notes. Existing v4.0.0 release history follows.

InfernoSIM v4.0.0

Choose a tag to compare

@github-actions github-actions released this 24 Aug 13:29

InfernoSIM v4.0.0

Status: generally available

v4 adds a local, incident-derived reliability gate for tool-using agents. It
replays recorded LLM and tool protocols, injects deterministic semantic and
transport faults, accounts for real side-effect consequences, and emits
reviewable CI evidence without requiring a hosted InfernoSIM service or live
model during the test.

Highlights

  • infernosim agent cases, agent run, and agent stress plan stable
    baseline/single-fault matrices and execute only the explicit child command
    supplied after --.
  • MCP JSON-RPC over HTTP is recognized inside ordinary HTTP/HTTPS replay. The
    new agent mcp record and agent mcp replay commands support bounded
    newline-delimited stdio transcripts with semantic request matching and
    runtime request-ID rewriting. Lost responses keep the replay server alive for
    retries, and EOF evaluates assertion/fault coverage with optional reports.
  • Protocol adapters recognize OpenAI Responses, OpenAI Chat Completions,
    Anthropic Messages, and Ollama chat tool-call envelopes. Generic HTTP
    selectors remain available for custom gateways.
  • Faults can target response kind, provider, tool, HTTP method, host/path RE2,
    and occurrence. Actions include semantic JSON/SSE/NDJSON mutation, status and
    header changes, bounded delay/timeout, reset, truncation, empty success, and
    “side effect committed but response lost.”
  • An explicit side-effect ledger tracks hashed identities, idempotency
    deduplication, commit state, response delivery, and ambiguous completion
    without retaining raw tool arguments or results.
  • Consequence assertions cover exactly-once, at-most-once, forbidden effects,
    call budgets, verified retry, recorded-universe-only execution, and
    deadlines. Verification can require a delivered 2xx response matching a
    JSONPath/value predicate.
  • Streaming HTTP capture can preserve bounded SSE, NDJSON, and JSON-sequence
    response frames and inter-frame delays; replay can mutate structured frames
    without flattening their framing.
  • agent otel import normalizes OTLP JSON or JSONL into an owner-only
    correlation log. Raw prompt, argument, and result attributes are excluded;
    optional one-way content hashes support correlation.
  • Agent results produce private JSON plus JUnit, SARIF, and standalone HTML.
    JUnit contains one testcase per planned case. A separate JSON reliability
    surface reports unweighted observed pass rates by validated fault category;
    it is evidence for that matrix, not a general model score.
  • testgen --profile agent generates a stable case manifest and agent-aware
    Testcontainers, Compose, or GitHub Actions wiring. Generated Actions run the
    assertion engine and fail if a selected fault is never exercised.
  • Bundle-v2 archives automatically include and encrypt optional mcp.log and
    agent-spans.jsonl files because they remain regular incident files; the
    encrypted format itself is unchanged.

Public validation fixture

examples/agent-reliability contains a sanitized refund incident and recorded
MCP/OpenAI-style exchanges. examples/agentlab contains a defensive loop and
an intentionally unsafe control. The matrix covers:

  • a refund commit whose response is lost;
  • an HTTP-200 policy response missing its decision field;
  • an LLM refund tool call missing its arguments; and
  • an MCP tools list missing the required refund tool.

The checked-in raw benchmark completed 20 identical matrix iterations. The
defensive loop passed 100/100 executions, the unsafe baseline passed 20/20,
and the unsafe loop was rejected in 80/80 fault executions. Case-plan hashes
were stable and raw content was not retained. Timing is environment-specific.
See benchmarks/results/agent-reliability.json.

The optional local Ollama smoke passed with llama3.1-local:latest by producing
a native policy_check tool call that the adapter recognized. Ollama sampling
is compatibility evidence, not the deterministic release oracle.

Validation gates

  • Root and Testcontainers-module tests pass with the Go 1.26.6 race detector;
    module consistency and go vet are clean. The Go vulnerability scan reports
    zero reachable vulnerabilities.
  • Ten bounded fuzz targets pass, including the agent fault engine, semantic
    JSONPath implementation, MCP request matcher, OpenAPI matcher, bundle-v2,
    gRPC, streaming-template, healer, and message surfaces.
  • The safe/unsafe agent smoke confirms five defensive passes, the unsafe
    baseline control, and rejection of all four unsafe fault cases. The MCP stdio
    CLI fixture evaluates fault coverage and consequence assertions at EOF.
  • The final production image builds, runs as the unprivileged infernosim
    user, enumerates the five checked-in cases, and passes the real
    Testcontainers lifecycle test. Existing Node, Go Compose, and
    Kafka/AsyncAPI/Redpanda smokes remain green.
  • Generated agent GitHub Actions and Compose harnesses pass actionlint and
    docker compose config. All repository workflows pass actionlint.
  • A local GoReleaser snapshot builds eight platform archives plus
    checksums.txt; every archive checksum verifies. The tagged release workflow
    repeats the mandatory release gates before publishing that same artifact set.

Safety and compatibility

  • Existing incidents and replay configurations remain valid; the agent
    section is opt-in.
  • The minimum Go toolchain is 1.26.6. This is the first 1.26 patch that fixes
    the reachable standard-library advisories found by the release vulnerability
    scan; CI is pinned to it and the container builder downloads that exact
    toolchain through Go's automatic toolchain selection.
  • Both simulator listeners bind to loopback in the runner. Calls outside the
    recorded universe fail closed and can be asserted explicitly.
  • Agent bodies are capped at 16 MiB, MCP messages at 16 MiB, MCP transcripts at
    100,000 records, OTel imports at 64 MiB, call/case counts at 10,000, and fault
    delays at 60 seconds. JSONPath length/depth and regex length are bounded.
  • Configuration is strictly decoded. Duplicate IDs, unknown fields, invalid
    selectors, unsupported mutations, conflicting terminal faults, and an
    ambiguous-response fault without a matching declared effect are rejected.
  • Proofs expose hashes, operation names, counts, assertion results, and
    divergence reasons—not captured bodies, prompt text, or raw tool values.
    Bounded child stdout/stderr is stored only in owner-only JSON and should
    still be treated as sensitive application output.
  • Case execution has a bounded timeout and process-output limit. Incident data
    never supplies executable commands.
  • The release artifact contract is unchanged: eight platform archives plus
    checksums.txt are uploaded by GoReleaser. Benchmark JSON, reports, fixtures,
    and internal GoReleaser metadata are not release assets.

Deliberate boundaries

  • v4 explores the baseline plus one configured fault at a time. It does not
    claim combinatorial search or formal verification.
  • The first recognized tool call in a provider response is independently
    selectable. Parallel multi-tool envelopes are replayable but not yet
    individually fault-addressable within one response.
  • MCP stdio support targets newline-delimited JSON-RPC. Other stdio framing and
    binary MCP transports are not claimed.
  • OpenTelemetry support is an offline OTLP JSON/JSONL importer, not a live OTLP
    collector.
  • Provider adapters recognize envelopes; they do not replace an LLM server or
    make live model sampling deterministic.
  • Existing gRPC compression, reflection/remote descriptor, large-body, and
    bidirectional-stream branching limitations remain.

Upgrade and documentation

  • New guide: docs/AGENT_RELIABILITY.md
  • Upgrade notes: docs/UPGRADING.md
  • Maintainer gates: docs/RELEASING.md
  • Example fixture: examples/agent-reliability
  • Safe/unsafe reference loop: examples/agentlab
  • Local gates: scripts/agent-smoke.sh and scripts/ollama-smoke.sh

InfernoSIM v3.4.0

Choose a tag to compare

@pranaysparihar pranaysparihar released this 12 Aug 17:44
6ba0262

InfernoSIM v3.4.0

Status: generally available

v3.4 is a single GA release train. No alpha, beta, v3.5, or v3.6 aliases are
used. Publication is allowed only after the mandatory release workflow passes
on the exact commit being tagged.

v3.4 completes InfernoSIM's local incident-to-test path. A sanitized production
incident can now become an editable container test, an explainably stabilized
matcher configuration, a cross-protocol contract gate, and a deterministic
proof artifact without an InfernoSIM-hosted service.

Install

brew install pranaysparihar/infernosim/infernosim

Highlights

  • infernosim serve runs captured HTTP, HTTPS, HTTP/2, and gRPC dependency
    behavior as a standalone simulator with a separate health/reset/status/proof
    control API.
  • infernosim testgen produces readable Testcontainers-Go, Docker Compose, or
    GitHub Actions harnesses. A maintained Testcontainers-Go adapter is shipped
    as an independent module under integrations/testcontainers-go.
  • infernosim heal infers narrow semantic matchers from repeated observations,
    validates candidates on a held-out observation, protects security/business
    fields, hashes evidence, and refuses ambiguous proposals.
  • Kafka-compatible capture and replay preserve topic, partition, offset, key,
    headers, payload, schema name, correlation ID, timestamp, and payload hash.
  • Kafka connections support TLS, mTLS, SASL/PLAIN, SCRAM-SHA-256, and
    SCRAM-SHA-512 without accepting passwords on the command line.
  • Deterministic Kafka delay, drop, duplicate, poison, and reorder plans make
    asynchronous failure tests repeatable.
  • AsyncAPI 3.x validates JSON payload schemas, message/channel references,
    required fields, types, enums, patterns, and additional-property drift.
  • Explicit workflows verify ordered HTTP, gRPC, and Kafka observations with
    optional correlation and per-step timing bounds.
  • HTTP and Kafka captures share configurable deterministic tokenization,
    redaction, and drop rules.
  • Simulator and Kafka proof JSON records semantic fingerprints; workflow,
    AsyncAPI, and Kafka commands emit JUnit, SARIF, and HTML reports.

Safety and compatibility

  • Existing incidents and replay configuration remain valid. The new
    workflows section is optional and strictly validated.
  • Healing writes replay.proposed.yaml; it never silently replaces
    replay.yaml. Explicit --apply promotion creates replay.yaml.bak first.
  • Authorization, credentials, tenant/account boundaries, permissions, money,
    status, and personal-data fields cannot be automatically relaxed.
  • UUIDs and timestamps are relaxed only at allowlisted volatile locations such
    as request, trace, correlation, nonce, or timestamp fields; identity IDs stay
    exact. Regeneration removes stale InfernoSIM-managed healing rules.
  • A proposal that makes distinct recorded responses match the same request
    fails without writing configuration.
  • Control APIs do not expose captured bodies, keys, or headers.
  • Generated incident files remain owner-only on the host and are copied
    read-only into the isolated, unprivileged container.
  • Kafka capture subscribes only to topics explicitly selected by the user.
  • Kafka capture requires a privacy policy unless raw sensitive-data storage is
    explicitly enabled. Replayable policy-based capture also requires
    capture_bodies: true.
  • Invalid report formats, topics, authentication combinations, non-finite
    timing/confidence values, duplicate message IDs, duplicate privacy rules,
    unsupported AsyncAPI schema features, and unsafe generated paths fail before
    network side effects.
  • Homebrew publication does not require a cross-repository personal access
    token. The tap's own scheduled/manual workflow reads the latest public
    release and publishes the source-built formula with its repository-scoped
    GITHUB_TOKEN; a maintainer can also update it from an authenticated local
    session.
    Releases upload only eight platform archives plus checksums.txt;
    benchmark/report JSON is excluded.

Validation evidence

  • The checked-in category baseline completed 100 independent heal/testgen runs
    with the expected accept/reject behavior, one configuration hash, one harness
    hash, zero ambiguities, and zero seeded-secret leaks. On the recorded local
    run, healing p95 was 0.896 ms and test generation p95 was 0.607 ms. Raw
    results are in benchmarks/results/infernosim.json; timing is environment
    specific and is not a competitor claim.
  • Root and nested-module race tests, module consistency, vet, and reachable
    vulnerability analysis pass. The vulnerability scan reports zero reachable
    vulnerabilities.
  • The Kafka CLI wire test passes capture → AsyncAPI validation → prefixed replay
    → consume and verifies JUnit, SARIF, HTML, and proof files.
  • The production Dockerfile builds successfully, runs unprivileged, exposes the
    proxy/control ports, and passes the real Testcontainers lifecycle/proxy/reset
    test. Node and Go Compose smoke profiles pass against that image.
  • Targeted matcher, gRPC, template, OpenAPI, bundle, healer, and message fuzz
    smoke tests pass. GoReleaser builds all eight platform archives and a
    checksum manifest; the source-built Homebrew formula passes brew test.
    The upload-asset assertion reports nine project assets, which GitHub displays
    as 11 after its two automatic source downloads.
  • The Docker smoke passes capture → validation → replay → consume against
    Redpanda v25.2.9 using a pinned multi-architecture image digest and separate
    internal/external listeners. This remains a mandatory release-workflow gate.

Deliberate boundaries

  • v3.4 supports Kafka-compatible brokers, not RabbitMQ, NATS, MQTT, SQS, or SNS.
  • AsyncAPI validation covers JSON payloads. Avro, Schema Registry, and remote
    schema resolution are not implemented.
  • Kafka capture uses a consumer subscription rather than a transparent Kafka
    protocol proxy.
  • Healing is deterministic rule inference rather than an LLM and declines
    values it cannot classify safely.
  • Existing gRPC compression, reflection/remote descriptor, large-body, and
    bidirectional-stream branching limitations remain.

v3.3.0

Choose a tag to compare

@github-actions github-actions released this 31 Jul 18:59
8b7bdc5

Changelog

  • 8b7bdc5 Fix release without Homebrew tap token (#5)

v3.2.0

Choose a tag to compare

@github-actions github-actions released this 31 Jul 18:52
b411f69

Changelog

v3.0.1

Choose a tag to compare

@pranaysparihar pranaysparihar released this 16 Mar 18:39

Changelog

  • 658a813 add homebrew tap configuration

v3.0.0

Choose a tag to compare

@pranaysparihar pranaysparihar released this 16 Mar 12:40

Changelog