Skip to content

proef v0.15.0

Choose a tag to compare

@emrecdr emrecdr released this 09 Sep 09:09
· 81 commits to main since this release
54e85ab

Published retroactively on 2026-09-09. This version was released on
2026-08-25 — the release commit is on main and the changelog section below
was cut then — but its v0.15.0 tag was never pushed, so the release
pipeline never ran for it. This release therefore carries no binaries,
checksum sidecars or build attestations
; every other release has all
three. Install 0.15.0 from source, or use a later version — 0.16.0
superseded this one six days later.

Breaking

  • A quarantined test-failure reaches JUnit as <skipped> with a message,
    not <failure>
    — Jenkins marked builds UNSTABLE while proef exited 0;
    every dashboard now says what the exit code says (ADR-0019). Library:
    ScenarioSpec gains skip, ScenarioOutcome/ScenarioRun gain
    reason, Event::ScenarioFinished gains additive reason,
    write_junit takes the non-gating list.
  • --shard assignments re-deal: the hash gained a mixing finalizer. Raw
    FNV-1a's low bit is the XOR-parity of the input bytes, so a scenario named
    after its feature file — the commonest Gherkin convention — collapsed to
    one shard at N=2 and left odd shards empty at N=4, silently (the empty
    shard exits 0). shard_bucket now finalizes with Murmur3's fmix64; every
    scenario re-buckets, so all jobs of one matrix must run the same proef
    version (already true in practice). Round-18 finding, reproduced and
    mechanism-verified before fixing; the balance test gained the
    name-mirrors-file corpus it was structurally blind to.
  • Tag atoms glob. * and ? in a --tags / [run] exclusive-tags atom
    are now anchored wildcards (@FRD-* selects the family; ? is one
    character) — previously they were literal characters that silently matched
    nothing, the trap this closes. Metacharacter-free atoms are bit-identical
    to before, property-pinned. Case stays sensitive.
  • JUnit test identity is classname + name. classname carries the
    feature file, name the scenario alone; the old single name embedded
    file:line, so an edit above a scenario re-identified every test below it
    in Jenkins history and GitLab's MR diff. Anything keyed on the old
    file:line name strings must re-key. The suite skipped count is now
    spelled skipped (was disabled, which no consumer reads).

Added

  • [tag-links] turns tag cells into tracker links (RF's
    --tagstatlink, reduced to one mechanism): tag glob → URL template with
    {tag} substituted, honored by the HTML report's by-tag table and the
    GitHub summary; the pattern language is the same anchored glob --tags
    uses. Library (Breaking): render_html takes the link map;
    tags::atom_matches_public exposes the one matcher.

  • --console dotted|quiet (RF wave 3): one glyph per scenario (.
    pass, F fail, s skip, w warn — lowercase is non-gating, the
    pytest/RF convention, flushed per glyph, wrapped at 80) or just the frame.
    Purely presentation: the record, every report, the post-pool failure
    details and the exit code are identical in every mode; run.log mirrors
    the console verbatim, dots included — events.jsonl is the full truth.
    Library (Breaking): ConsoleReporter::new takes a ConsoleMode.

  • A --rerun now produces the one JUnit and the one report that cover
    the whole suite
    (E2's rerun half; Robot Framework's rebot --merge
    shape, done as composition): the run head records rerun_of, the JUnit
    carries the base's not-re-run scenarios as ordinary testcases, and
    proef report overlays the base into a merged page (banner named, base
    timestamps stripped so timelines never mix, rotated-away base degrades
    loudly). Exit code and totals stay the rerun's own.

  • --meta key=value and [meta]/[env.<name>.meta] record explicit run
    metadata
    (ADR-0020, RF wave 2): commit, build URL, team — recorded in
    the run head, shown by the HTML report, GitHub summary, explain,
    diff (which now also warns on cross-env comparisons) and the
    --output json body (additive keys). The active --env profile name and
    the --shuffle marker ride the same head. proef never harvests: no git,
    no hostname, no CI env sniffing — the shell harvests, proef records.
    Everything passes the sink-boundary mask, keys and values both. Library
    (Breaking): RunRecord::open and exec::execute take the head inputs.

  • Per-tag verdicts in the HTML report and the GitHub summary (RF wave
    2): tags now reach the record — additive tags on scenario_finished
    (finished-only: the cancel-skip path emits no start), additive
    exclusive on scenario_started (closes R11-6, the scheduler's own
    bool) — and both reports roll them up per tag (suite-only, Warned counts
    with passed). Requirement-tagged suites (@FRD-3.1) get their
    traceability matrix for free. Tags are deduped at the one accumulation
    point (first occurrence wins); the quarantine list is now derived from
    the outcomes' own tags — one owner, same behavior, pinned by the exit
    suite. Library (Breaking): ScenarioSpec/ScenarioOutcome gain tags.

  • @skip and @skip:<reason> park a scenario visibly (ADR-0019, RF
    wave 2): counted in every total, reasoned in the console, JUnit, TAP, the
    record, the HTML report, explain and flows --output json; the harness
    maps it to libtest's ignored flag. All-selected-skipped exits 0; the
    empty-selection refusal stays exit 2. --tags "not @skip*" unselects both
    spellings; an authored skip is never re-queued by --rerun, and diff
    gives skip transitions their own bucket instead of reading them as fixed.

  • flows shows the feature description. The prose block under
    Feature: was parsed and then dropped — the one paragraph written for
    exactly the reader flows serves never reached them. Human output prints
    it under the feature header; --output json rows gain
    featureDescription: string|null (additive). Library: FeatureFile gains
    description.

  • --shuffle re-deals the execution order, seeded by the run id — one
    determinism knob for order and fakes alike, so --shuffle --run-id <id>
    reproduces an order-dependent failure exactly (Robot Framework's
    --randomize, minus the parallel seed it threads separately). Applied
    after --shard, so membership never moves; under --watch every unpinned
    rerun re-deals, deliberately. The permutation is version-stable and
    pinned. Recording a shuffled marker in the run head is deferred to the
    planned RunStarted additions (env/metadata), one wire change instead of
    two.

  • The failing step's reproduce: curl … reaches the record. The engine
    always computed the redacted curl and the live console always printed it —
    and the record dropped it, so explain and the HTML report knew less than
    the console did. StepFinished gains additive reproduce_hint (absent on
    passing steps and every pre-field stream); explain and the report print
    it; the sink-boundary mask covers it like detail.

  • README documents every flag the binary exposes, enforced. v0.14.0
    shipped --shard and --max-fail with no README mention; the docs gate
    gains the flags direction (same vacuity guard as the command half), and the
    measured gap — those two plus schema --add-to — is closed.

  • JUnit carries what GitLab and Jenkins actually read (R3-6, specced from
    GitLab's parser docs and Jenkins' SuiteResult.java): file on each
    testcase (GitLab source linking), time on suite and root. timestamp and
    hostname stay absent deliberately — ignored or substituted by both
    consumers, and a hostname would undo R12-1's provenance fix.

  • The docs corpus is a website: https://emrecdr.github.io/proef/. mdBook
    renders docs/ on every push to main that touches it; the nav is
    docs/SUMMARY.md, which the existing docs gates link-check like any other
    doc, and the pages workflow refuses a corpus doc that is not on the site.
    The crate homepage points there from the next release.

Fixed

  • A failure detail is bounded before it reaches any sink. hurl's rendered
    assert error quotes the actual response, so a failed assert on a large body
    rode full-size into the record, JUnit, the HTML report and the GitHub
    summary at once. The engine now middle-cuts past 40 lines / 8 KiB with a
    marker naming the elision; the artifact pointer survives outside the cut,
    and the full output is one re-run away (Robot Framework's 40-line rule,
    adopted at the boundary where all sinks are covered at once).

  • The machine-body contract closes its last two paths: an empty selection
    (--scenario/--tags matching nothing — loud exit 2 by design) and a
    corrupt global-state file both emitted zero stdout bytes under
    --output json.

  • Identical errors collapse like identical warnings — a broken macro
    usually fails to lower everywhere, so the error wall was the more common
    fifty-block wall; distinct errors still render separately, and SARIF keeps
    every site.

  • Injected [Options] lines respect every section-ending shape. The
    section-end move covered one shape of five: an unfenced JSON/XML body after
    an author [Options] swallowed the injected lines into invalid hurl (exit
    2 on input that worked before), and an entry with an author section but no
    response line leaked its pending lines into the next entry, where hurl
    parsed retry: as an HTTP header and the artifact validated green. The
    section now ends at the first line that could not sit inside it.

  • A # inside a bind: value no longer hides the reads after it. The
    template probe parsed the value in an unquoted position where # opens a
    comment; it now probes the quoted variable: position bake actually
    injects into, so "{{a}} # {{b}}" reports both.

  • A setup that fails to load still emits the machine body — the last
    terminating path returning zero stdout bytes under --output json.

  • SARIF keeps one result per site again. The warning collapse shipped at
    the front-end aggregation, which also feeds SARIF — a code-scanning
    consumer lost every anchor but the first. The collapse now happens at
    console rendering only; SARIF carries all sites, the console one line with
    the count.

  • Every terminating path emits exactly one machine body (R17-2.3/2.4).
    An empty shard printed its prose note as the --output json body — jq
    failed on the very path a sharded matrix guarantees one job takes — and a
    setup abort printed nothing at all while JUnit carried the failure. The
    note now goes to stderr under machine output (and lost a stray-space run);
    never-ran paths report ADR-0014's suite-only zeros with the exit code
    carrying the verdict.

  • A failed teardown reaches JUnit as its own suite (R17-2.5) — #78's
    rule made symmetric: a phase appears in the reports when it fails. A gated
    pipeline used to read a fully-passing report on an exit-3 run.

  • A repeated warning is one warning with a count. One authored mistake in
    a macro shared by fifty scenarios rendered fifty times; identical warnings
    now collapse to their first occurrence plus "(N sites across the suite)".

  • explain's truncated-record fallback counts the suite only — a record
    that died mid-setup folded the phase scenario into the totals three lines
    above the label saying phases are excluded (ADR-0014).

  • bind: values are read by hurl's parser, not a text scan (R17-2.2). A
    hurl function ({{newUuid}}, {{newDate}}) no longer counts as an unbound
    variable — proef refused input stock hurl runs — and a sibling literal bind
    whose name sorts earlier now counts as a supplier, since injected
    [Options] variable: lines are written and evaluated in name order. The
    seam answers the question once: FragmentSupport::template_reads.

  • A fragment's own [Options] variable: lines now evaluate before the
    injected ones.
    Injection used to land at the section head, so the
    fragment-supplies-it route the unbound check accepts was assigned too late
    to be read at run time — accepted at dry-run, wrong at execution.

  • A {{x}} inside a bind: value is validated at --dry-run, not at run
    time.
    hurl templates the injected [Options] variable: line when the entry
    runs, so a name nothing supplies used to pass dry-run and die mid-run;
    proef::lower::unbound_placeholder now names the placeholder and the bind key
    at lower time, where the capture set is known. What legitimately supplies it:
    an earlier step's capture, the fragment's own [Options] variable:, a secret
    in scope, or a sibling literal bind whose name sorts earlier (injected lines
    are written and evaluated in name order).

  • A literal bind: that shadows an earlier capture is named, not silent.
    hurl's variable: assigns into one shared set, so the bound value replaces
    the captured one from that entry on — sometimes intended, so it is a warning:
    proef::lower::bind_shadows_capture. A secret bind cannot shadow (it skips
    the [Options] path) and draws no warning.

  • A failed [run] setup reaches JUnit, the GitHub summary, and PR
    annotations.
    The abort used to return before the CI-report block, so a job
    gating on --junit saw no file at all — indistinguishable from proef never
    running. The reports now carry the setup scenario itself (suite named by the
    setup feature file); exit codes are untouched (ADR-0014), and nothing is
    fabricated for the pool that never ran.

Internal

  • The machine body has one exit. execute's six terminating paths each
    pasted the same empty-body emission; they now return through a single
    funnel with the one emit_machine_body call after it, so a new path
    cannot forget the contract — and the empty-selection body takes its exit
    code from the refusal itself instead of restating it. Post-merge cleanup
    pass over the deep-audit cycle; behavior pinned by the existing path tests.
  • The probe and bake share the whole variable: line. template_reads
    re-spelled the injected [Options] variable: line by hand around the
    shared escaper; proef_core::lower::variable_option_line now builds it for
    both, and quote_option returns to being private (library-surface swap;
    unreleased either way). The section-end flush in bake_entry_options also
    drops its fence-branch duplicate — one check covers all shapes — and the
    console collapse builds its annotated message without cloning the
    diagnostic on the common single-site path.
  • The canary stopped trusting the index's tail twice over: it skips
    prerelease versions (the sparse index is publish-ordered, so a 9.0.0-beta
    would have become "latest"), and refuses a backport older than the pin by
    semver ordering (an 8.0.2 published after 9.0.0 would have produced a
    green about a downgrade).
  • deny.toml's advisory ignores were dead and are gone. The quick-xml
    pair was ignored under "the patched release is unreachable" — the quick-junit
    0.7 bump made it reachable and the workspace has been on the patched line;
    the stale ignores would also have silenced any new advisory against it.
    quick-xml itself re-pinned to quick-junit 0.7's in-tree copy (=0.41.0,
    one lock generation); lsp-server rides to 0.10, toml to 1.x.

Documentation

  • The corpus tells the truth again, audited claim-by-claim: CONFIG
    documents the [env.<name>.run] jobs-only rule a reader used to discover as
    a parse error; GETTING-STARTED can produce its own output (it now states the
    fixture token its §5 requires, and its reproduce command names the
    --secret the replay needs); EVENTS carries the provenance, totals, and
    field facts consumers implement against; TESTING-STRATEGY describes the CI
    that exists; RELEASING's gate list predicts CI; TROUBLESHOOTING's exit-1 row
    covers the --check family. The #N in an outline instance is documented
    as positional, with the column-placeholder naming that keeps identity stable
    across --shard and JUnit history.