Load testing foundations: YAML contract, engine levers, CLI load mode, run-flows Action (Phase 0+1) - #45
Merged
Merged
Conversation
CreateFlowRunner now accepts variadic functional options. Existing call sites (server flowexec session, CLI runner, tests) compile untouched and keep the CPU-derived default concurrency. WithMaxConcurrency(n) overrides the node-level parallelism cap; n <= 0 is ignored so callers can pass a config value through unconditionally.
Adds an engine-agnostic scheduler that runs a caller-supplied iteration function across a fixed pool of virtual users. It knows nothing about flows: callers pass any callback. Bounds are Duration, MaxIterations and context cancellation; whichever comes first stops new iterations being issued, and in-flight iterations are always drained before Run returns. Iteration errors are counted in the summary and never abort the scenario. Sequence numbers are handed out exactly once, contiguously from zero. Invalid profiles (no VUs, no stop condition, nil callback) are rejected before any work starts.
Long load runs retain every response body in the flow variable map, so memory grows with iteration count. WithLeanMode(true) makes HTTP request nodes write a placeholder instead of the decoded body. The seam is FlowNodeRequest.LeanMode: that struct is built in exactly one place in the workspace, so the flag reaches every node without touching any node constructor or the flow builder. The body is swapped in buildResponseVar, before it can be copied into the flow output, which also folds the duplicated duration conversion out of RunSync/RunAsync. Assertions are evaluated by ResponseCreateHTTP against the raw response rather than the variable map, so they keep working on a dropped body; only downstream extraction from response.body is given up, which is the point of the mode. Off by default: no option means byte-identical output to before.
The default-concurrency test compared against the package variable, which would still pass if the default were changed to a constant. Assert the variable itself is MaxParallelism() so that regression is caught. Also states in the scenariorunner docs when ctx.Err() is returned and that panics inside an iteration are not recovered.
Characterization tests for the yamlflowsimplev2 Import -> Export pipeline, one fixture per family (request headers/assertions/templates, if/for/for_each control flow, js/wait, graphql, websocket, sub-flows, multi-flow run block, environments/credentials). Each case asserts round-trip stability (export(import(export(import(x)))) is a fixed point) and pins the stable output as a .golden snapshot. This snapshot captures today's behavior including known bugs: HTTP assertions are silently dropped on import (GraphQL assertions survive correctly), credentials and active/global environment selection are not re-exported. sub_flow_trigger is deliberately excluded from the sub-flows fixture: it has a pre-existing node-ID tracking bug (converter_node.go processSteps) where the flow node's ID is overwritten to the flow's start-node ID but the NodeSubFlowTrigger.FlowNodeID implementation record keeps the original ID, so the exporter's lookup always misses and silently drops the step. That breaks round-trip stability outright (re-import errors with "depends on unknown step"), independent of anything in this change. Not fixed here since it's outside this change's scope.
Standalone packages/server/pkg/loadmetrics package for load-test metrics: HDR-histogram aggregation keyed by (step, status-class), interval frames via Aggregator.Flush, lossless Merge across frames, and ClassifyStatus for mapping HTTP status/transport errors (incl. timeout detection) to buckets. Adds github.com/HdrHistogram/hdrhistogram-go as a new packages/server dependency (1us..10min range, 3 significant figures per the frozen contract). TDD: loadmetrics_test.go written first against the not-yet-existing API (confirmed RED via compile failure), then loadmetrics.go implemented to green. Covers percentile correctness against a known uniform distribution, merge equivalence across split aggregators, status classification incl. timeout detection, concurrent Record under -race, and RPS math.
Add a version field to the yamlflow document: absent (0) is treated as the current version for backward compatibility, and versions newer than this build supports are rejected with a clear error naming the offending value and the supported ceiling. Export always stamps the current version as the first key of the document so the schema version is visible without parsing the rest of the file. Golden corpus updated (-update): every fixture gains exactly one "version: 2" line, nothing else changes.
HTTP request assertions were parsed from YAML and correctly merged through templates (mergeHTTPRequestDataStruct), but processRequestStep never converted them into mhttp.HTTPAssert records, so WorkspaceBundle.HTTPAsserts stayed empty for every HTTP request on import. They always exported fine (the exporter reads straight from HTTPAsserts), which is why the bug was invisible round-trip: assertions just quietly vanished on the way in. Mirrors the GraphQL assertion conversion in processGraphQLStructStep, which never had this bug: HTTPAssociatedData gains an Asserts field, processRequestStep converts finalReq.Assertions into HTTPAssert records bound to the HttpID of the request, and mergeAssociatedData folds them into the bundle. Golden corpus updated (-update): only the fixture exercising HTTP request assertions changes, and only by having its assertions reappear (including the use_request template-merge case, which now correctly shows both the assertions from the template and the ones added by the step itself).
Adds the four load-testing metrics/artifact models to the spec, following
the existing per-domain .tsp conventions (plain namespace, camelCase fields,
Protobuf.WellKnown.Timestamp/Map, small nested helper models):
- LoadMetricEntry: one (step, status-class) bucket for a reporting interval
- count/errorCount/bytes, hdrHistogram bytes, p50/p90/p95/p99/maxUs
- LoadMetricFrame: intervalStart/intervalMs + LoadMetricEntry[]
- LoadRunReport: total (LoadStats) + perStep (LoadRunStepStats[], which
spreads LoadStats alongside its step/statusClass key - mirrors how
CommonTableFields<T> is spread into per-domain child models)
- LoadFailureArtifact: step/vu/iteration, request{method,url,headers,
bodySample}, optional response{status,headers,bodySample} (absent on
transport-level failures), resolvedVariables map, error, capturedAt
LoadStatusClass enum members (TwoXx/ThreeXx/FourXx/FiveXx/Error/Timeout)
mirror packages/server/pkg/loadmetrics.StatusClass's wire values; header
key/value shape mirrors HttpResponseHeader/GraphQLResponseHeader.
packages/spec/dist is gitignored and not committed anywhere in this repo
(confirmed via `git log -- packages/spec/dist`), so generated output is not
included here - regenerate with `pnpm nx run spec:build`. Verified the
generated .proto/Go/TS appear correctly and packages/server, packages/db,
packages/spec, packages/auth-lib all still `go build ./...` cleanly.
Also runs `go mod tidy` in packages/server now that the workspace builds,
correcting hdrhistogram-go's require-block placement (it was marked
`// indirect` in the previous commit because go mod tidy could not resolve
the whole module until spec:build produced packages/spec/dist).
Composite action that downloads the released devtoolscli binary for the runner OS/arch, runs a .yamlflow.yaml file, and publishes a job summary plus JSON/JUnit reports as outputs (json-report, junit-report, success). Replaces hand-rolled pnpm/nx CLI builds in consumer CI: "latest" resolves to the highest cli@* release tag via git ls-remote (the repo also cuts desktop@/web@ releases on the same tracker), the exact asset is verified with curl -fsI before downloading, and errors name the tried URL. Linux/macOS only; every step is shell: bash.
Exercises the composite action end to end on ubuntu-latest and macos-latest: downloads the latest published cli@ release, runs the smoke fixture, and asserts the json-report/junit-report files and job summary were produced. Triggered on pull_request for actions/** changes plus workflow_dispatch.
Replace the DIY GitHub Actions snippet (which built via pnpm nx run cli:build - a server-only target missing the cli tag, so the resulting binary could not actually run flow commands) with the actions/run-flows composite action. Keep a corrected manual alternative using install.sh for Windows/air-gapped cases, invoked as `devtools` (the name install.sh actually installs it as, not `devtoolscli`).
Wraps the resolvedVariables map type per the repo prettier config (checked by root:lint:format); no semantic change - verified spec:build output is byte-identical before and after.
The claim() comment said all stop conditions are checked before the sequence number is reserved. Only cancellation and the deadline are: the iteration bound is enforced after the atomic add, by discarding an over-limit claim, so next overruns MaxIterations by up to VUs. That distinction matters. A maintainer trusting the old comment could delete the post-add discard believing a pre-check covered the bound, which would silently break the exact iteration count. Moving the check before the add would be worse still: several workers could read the same pre-add value and all conclude they were under the limit. Comment only; no behaviour change.
Configurable flow-runner concurrency, scenariorunner VU scheduler, opt-in lean execution mode. Review clean after fix round 1.
RunMultipleFlows previously ran run: block entries in file declaration
order and only checked a dependency if that dependency happened to have
already run earlier in that same order, so an unknown or forward
declared dependency name was silently ignored, and a failed dependency
made the whole function return immediately with no record of what
happened to the flows that never got a chance to run.
Replace the ad-hoc map[string]interface{} re-parse of the run: block
with the typed yamlflowsimplev2 structs (gets the depends_on
scalar-or-list form for free), then topologically sort entries with
Kahn algorithm before executing anything, breaking ties by original
list order for determinism. Unknown dependencies and dependency cycles
are now hard errors naming the offending value:
unknown dependency "Missing" in run block (known flows: A, B)
dependency cycle in run block: A -> B -> A
Flows still execute sequentially. A flow gated by a failed or skipped
dependency is now recorded and reported as skipped (status plus reason,
through the logger and any configured reporters) instead of silently
never attempted; flows with no such gate still run even if an unrelated
earlier flow failed. The overall call still returns a non-nil error
whenever any flow failed or was skipped, preserving the existing
failure-gate contract.
Adds TestAggregatorKeysByStatusClass, tying ClassifyStatus's output directly into Record/Key bucketing (the shape task 5's ingest will actually use and asserting 5xx outcomes also count as errors. Also covers the zero-code/ nil-error edge case in TestClassifyStatus, and simplifies TestMergeRPSMath's histogram setup to a single RecordValues call. EOF )
Composite action packaging the released CLI for CI, test workflow, docs. Review clean, zero findings above Minor.
Code review flagged three observable behavior changes in RunMultipleFlows beyond the brief's three permitted changes. All three are confined to run: parsing/execution and remove a silent-failure mode, so they are sub-parts of change 3 (dependency-ordered run: execution) rather than a separate, undisclosed change. Pinning each with a dedicated test instead of leaving them as incidental side effects of the rewrite: 1. TestRunMultipleFlows_UnknownFlowPreventsAnyExecution - flow not found is now a pre-flight check across the whole run block before anything executes, not a mid-loop discovery. A run block listing a known flow followed by an unknown one used to run the known flow and only then fail on the unknown one; now nothing runs, verified by the mock server never receiving the known flow request. 2. TestRunMultipleFlows_SingleFailureMessageShape - the aggregate error format changed. It used to be just the failing flow raw error text, picked via unordered map iteration once more than one flow failed. It is now the failing flow name and status wrapped around that same error text, built from the deterministic run order. Pins the prefix and suffix this package controls; deliberately does not pin the OS-level dial error text in the middle, which is not something this package produces or should assert byte for byte. 3. TestRunMultipleFlows_MalformedRunEntrySurfacesError - a run entry that is not a flow mapping used to be silently dropped by the old hand-rolled parser, via a failed type assertion that just continued past it. Parsing through the typed yamlflowsimplev2 struct means the same document now fails to unmarshal instead of quietly running fewer flows than declared. No production code changes; these tests exercise behavior already shipped in the dependency-ordering commit.
Code review verified a pre-existing, undisclosed gap that interacts with this task's deliverable: MarshalSimplifiedYAML synthesizes the run: block purely from flows: declaration order (the "Generate default Run configuration" step in exporter.go) and never reads depends_on at all, so any dependency graph declared in run: is silently destroyed on export. An exported-then-rerun file always degrades to plain declaration order, regardless of what the original run: block said. Not fixed here - fixing the exporter is a fourth behavior change outside this task's three permitted changes, and belongs on the hygiene backlog instead. Documented so it is visible rather than silently relying on fixture luck: - New golden fixture run_deps_lost_on_export.yaml declares flows: in an order that is deliberately NOT a valid topological sort of its own depends_on chain (C, B, A for a C-depends-on-B-depends-on-A chain, the reverse of the correct A, B, C). The committed golden shows the synthesized run: block reproducing that same wrong flat order with no depends_on at all, so the destruction is visible in the diff rather than hidden behind an order that happens to look right. - Both that fixture and sub_flows.yaml (which documents the separate sub_flow_trigger export bug) now carry a short comment naming the gap and the exact lines responsible, so a future contributor does not have to re-derive it from scratch. No production code changes.
Finding 1 from review: TestMergeRPSMath only exercised the RPS formula against a hand-built Frame, never the real Flush() path, so a regression reintroducing a blind echo of the nominal interval would go undetected. Adds TestAggregatorFlushIntervalReflectsElapsedTime: drives real Flush() twice with different sleep windows (20ms, 80ms) against a 5-minute nominal interval, asserts Interval tracks actual elapsed time (not the nominal constant) and differs in magnitude between flushes, and asserts IntervalStart advances. Verified the test has teeth by temporarily mutating Flush to echo the nominal interval instead of computing the real elapsed time - confirmed it fails for exactly the mutated fields, then reverted (net diff to loadmetrics.go is zero; the interval-unused design itself was independently reviewed and ruled correct, matching the design doc's flushed-at-interval- and-at-run-end semantics).
Finding 2 from review: LoadRunReport only carried total/perStep, omitting two of the three things the design doc assigns to it (RunReport = merged frames + threshold verdicts + environment fingerprint for baseline comparability - docs/superpowers/specs/2026-08-08-load-testing-design.md section 3.3, explicitly "the hard-to-retrofit piece designed in Phase 0"). Controller ruling was to add the fields now (optional), not defer them, since a deferred doc comment would recreate the retrofit risk Phase 0 exists to remove. Adds: - LoadThresholdVerdict: expression (string, the threshold as configured), success (boolean), observedValue (optional string, stringified since thresholds may target durations, rates, or counts). The success field name mirrors HttpResponseAssert/GraphQLResponseAssert.success - the closest existing precedent for "verdict of evaluating an expression" in this spec. - LoadEnvironmentFingerprint: workerVersion/region/machineClass, all optional strings. Modeled as its own small nested model (not inline fields on LoadRunReport) mirroring flow.tsp's Position model - a small named value group with no decorators, no primary key, referenced directly as another model's field type. - LoadRunReport gains thresholds?: LoadThresholdVerdict[] and environment?: LoadEnvironmentFingerprint, both optional (absent for exploratory runs with no thresholds, or for local/desktop runs with no cloud environment). Doc comments on both note the shape is frozen in Phase 0 but nothing populates them until thresholds/the Stresseur worker fleet ship in Phase 2. Regenerated via spec:build; verified byte-identical output across two consecutive cache-bypassed runs (idempotent), and packages/server and packages/spec still `go build ./...` cleanly.
Golden round-trip corpus, version field, HTTP assertion import fix, run-block dependency ordering with strict failure modes. Review clean after fix round 1.
HDR aggregation package + TypeSpec models incl. threshold verdicts and environment fingerprint. Review clean after fix round 1.
Ignore .superpowers SDD scratch in prettier.
Adds the `load:` section from the load-testing design spec: named scenarios that reference a flow by name, so flows are never edited to be load-tested. Only the constant-vus executor is implemented; any other value errors naming the offending executor, the supported set, and that ramping-vus and constant-arrival-rate arrive in Phase 2. Every validation error names the scenario it came from. Scenarios ride on WorkspaceBundle purely so the YAML round trip preserves them - they are not database-backed, and the field documents that. Existing goldens are byte-unchanged; the new load_scenarios fixture pins the block's export, including duration normalization to Go's canonical form.
Wires the VU scheduler, the flow engine and the metrics envelope together: each virtual user gets its own HTTP client (so its own cookie jar and connection pool), its own instance of every flow node, its own persistence side-channels and its own metrics aggregator, which are merged at the end. Duration reaches the scheduler through RunProfile only - deriving it from a context deadline would make every successful timed run report DeadlineExceeded. Lean mode is always on, and the response side-channel is drained and discarded so nothing persists per iteration; the drain measures response sizes on the way past, which is where the report's byte counts come from. A completed run is a success even with failing requests. Only an unreachable target - every VU failing its very first iteration - is a setup failure. Node graphs are built per VU rather than per iteration: node implementations hold configuration only, so a rebuild buys no isolation while costing ~52% of a zero-latency iteration.
The console gets the aggregate table - p50/p95/p99, RPS and error rate per step plus a TOTAL row - and a context block above it that states which steps the numbers cover, since lean mode only reaches HTTP request nodes. The JSON report keeps writing the bare array of flow results it always has. Only a load run switches it to an object, so the additive load_report has somewhere to live and no existing consumer sees a change. Inside it the metrics ride as the spec's LoadRunReport, which is what makes the CLI's report the N=1 case of the message the fleet will stream later. Status classes cross that boundary through an explicit mapping, with loadmetrics.StatusClass as the source of truth; a round-trip test fails if either side gains a value the other lacks. Per-step rows are sorted so map iteration order never reaches the file.
Adds load mode to `flow run`. --scenario picks an entry of the file's load: block; --vus with --duration and/or --iterations describes a profile inline. The two forms are mutually exclusive, since a scenario already carries a complete profile and combining them would silently discard one. Load mode is decided from which flags the user passed rather than from their values, so `--vus 0` is a load run with a bad profile - and says so - rather than a silent fall-through to a functional run. A load run drives exactly one flow: the scenario's, the positional argument, or the file's only flow. Anything ambiguous names the candidates instead of guessing. Exit codes follow the load-testing convention: a run that completed is a success even with failing requests, since gating on error rates is what thresholds will do. Only a run that could not happen exits non-zero. Without any load flag the command behaves exactly as before, verified by diffing console and JSON output against a binary built from the base commit.
scenariorunner and loadmetrics are the two packages whose whole job is to be correct under concurrency, and the plain suite runs without -race. This gives them a dedicated target so the race detector is a gate rather than something someone remembers to run. Scoped to those two packages and -count=3 to stay fast enough to run routinely. Wiring it into CI is separate.
An aggregator's interval begins when it is constructed, and construction happens while VUs are being built. Left alone, the wall time the report divides by includes setup, so RPS reads low by however long building the node graphs took - which grows with the VU count, exactly when the number matters. Flushing the empty setup frame away restarts every interval together at the instant the scenario starts. The new test also pins the surrounding contract: RPS comes from real elapsed time, never from the aggregator's nominal flush interval.
The isolation assertion sampled a single node per VU through a randomly ordered map range, so a build-once-and-share regression only tripped it when the samples happened to differ. Under a shared-graph mutation it caught the regression 92 times out of 100 - and that rate falls as flows grow, since more nodes make distinct samples more likely, so it degraded exactly as fixtures got realistic. Now every node is compared across every pair of VUs over a sorted key list. The same mutation is caught 100 times out of 100, and the failure names the node and the pair.
A soak whose first iteration per VU hit a cold target and then ran cleanly for half an hour exited 1 with no table and no JSON. The exit code was right; throwing the measurements away was not, because a failed run's numbers are the ones someone actually needs. Run now assembles the report before deciding the error and returns both, on the setup-failure path and the cancellation path alike. Result.Ran() tells a caller whether a report exists at all - false only when the run never started, so a bad profile still prints nothing but its error. The CLI renders the table and writes the JSON whenever a run executed, then exits non-zero on the failure.
LoadScenarios rides on WorkspaceBundle so the YAML round trip keeps the load: block, but there is no schema behind it. Importing such a file into a workspace therefore loses it, and a later export will not bring it back - silently, which is the wrong way to lose data. Import now says so at warn level, naming the count and the scenarios. Storage stays out of scope; this only makes the gap visible until it exists.
load: YAML scenarios, loadrun VU execution wiring scenariorunner + loadmetrics, aggregate console/JSON reporting, load flags, race target. Review clean after fix round 1; all 10 binding contract additions verified.
Measures what one load worker can actually generate, driving the real
apps/cli/internal/loadrun engine (not a reimplementation of its scheduling)
against a local, in-process httptest target with a fixed 5ms handler latency
- no external network dependency of any kind.
apps/cli/test/loadbench/loadbench_test.go: the target server (a /single route
and a /chain route sharing one atomic token counter), the two flow fixtures,
and fixture-correctness tests that run in the default suite (no guard). The
chained-flow fixture chains for real - step N sends the exact token step N-1's
response issued - which TestChainedFlowReallyChains and
TestChainedFlowMultipleIterationsStillChain both pin. Chaining goes through a
response header rather than the body: load mode always runs in lean mode,
which replaces the whole decoded body with a fixed placeholder once
assertions run, so a body-based chain would silently chain the placeholder
instead of a real per-response value. setupFlow duplicates loadrun's own test
helper (unexported, so it can't be imported) rather than inventing a third way
to wire a flow - the load-mode docs already flag this pattern's duplication of
cmd wiring as an accepted tradeoff.
apps/cli/test/loadbench/integration_loadbench_test.go: the actual RPS/
percentile matrix across VUs x {1, 10, 50} x {single-get, chained-5-step},
gated by both a loadbench_integration build tag and RUN_LOADBENCH=true so it
never runs by default. VUs=50 against this fast a target pushes throughput
high enough (thousands/sec) that packages/server/pkg/httpclient.New()'s
reliance on http.DefaultTransport's default MaxIdleConnsPerHost=2 exhausts
this machine's ephemeral port range - confirmed independently of the flow
engine with a raw net/http repro. Cells run VUs-major, each in its own t.Run
(so its server and DB are torn down before the next cell starts), separated
by a 35s cooldown longer than this machine's TIME_WAIT - otherwise one cell's
port pressure bleeds into the next and inflates its error rate, which is
exactly what an earlier, uninstrumented version of this harness got wrong
(a VUs=50 cell reporting 98.5% errors that were mostly leftover pressure from
the VUs=10 cell three seconds earlier).
docs/superpowers/specs/phase0-bench.md: the results table, environment block,
methodology, and the full anomaly writeup above - flagged throughout as
dev-hardware-only numbers that gate spec section 3.5 capacity math and
section 6 pricing as an upper bound, pending a Fly shared-cpu-machine run.
Guarded loadbench matrix driving real loadrun; measured dev-hardware numbers in docs/superpowers/specs/phase0-bench.md. Review clean, zero findings above Minor.
Root-causes the two version-string defects together: version.go's version becomes a var so -ldflags -X can inject it, and taskfile.yaml's build:release now does so, so `devtools version` identifies the build instead of always printing v0.1.0. Also: design-doc dates the "current state" table to branch base 36d6306 and documents the run: skip-and-continue semantics; adds direct unit tests for the run: topological sort (declaration-order tie-break, self-dependency cycle, duplicate-flow-name error) plus integration coverage for the transitive skip cascade and unrelated-flow continuation; fixes a duplicate flow name in a run: block producing a blank "dependency cycle in run block: " error instead of naming the duplicate; guards the yamlflowsimplev2 golden test against silently passing on an emptied corpus; adds a zero-request FormatLoadTable case (the GraphQL-only-flow shape); documents that JUnit output carries no load data; fixes goimports ordering in importer_load_test.go; and gives finalize.sh the same "why no set -e" rationale write-summary.sh already has. No engine behavior change except the duplicate-flow-name error message (explicitly scoped: parseRunEntries only, topoSortRunEntries untouched). Goldens untouched.
$GITHUB_STEP_SUMMARY is a unique file per step, so the later assert step could never observe the summary the composite action appended inside its own step — the greps read a freshly created empty file and the check failed deterministically on both ubuntu-latest and macos-latest. Replace the three summary greps with jq assertions on the JSON report the summary is rendered from: both smoke flows present, status success, and a non-zero duration. Those are exactly the fields write-summary.sh reads, so summary data correctness is covered transitively. A comment records why the summary must not be grepped again.
topoSortRunEntries hard-errored on any depends_on name that was not a flow in the same run: block. Shipped example files list cross-flow *step* names there — a pattern users copied because the pre-topological-sort code silently ignored anything it did not recognise — so the hard error broke 4 of 5 CLI integration fixtures and every user file following them. Unknown depends_on names now drop out with a warning on stderr that names the offending value, the valid flow names, and the fact that step-level dependencies in run: are unsupported and ignored; execution proceeds exactly as if the name had not been written. Cycles and duplicate flow names stay hard errors — they are structurally invalid, not a compatibility surface — and a run: entry whose flow: value names a nonexistent flow still aborts pre-flight. Example fixtures are deliberately left untouched: keeping them working unmodified is the point. Behavior-change register 4.2 row 2 updated.
…med after build:release interpolated PLATFORM into the output filename only. GOOS and GOARCH were never set, so every matrix row in release-go.yaml built for the architecture of its own runner and simply renamed the result. Both cross-compiled rows share a runner with their x64 sibling, so cli@1.0.3 published a linux-arm64 asset and a win32-ia32 asset that were byte-identical x86-64 binaries and could not run on the platforms they advertised. Derive GOOS/GOARCH from PLATFORM (darwin/linux pass through, win32 to windows; x64 to amd64, ia32 to 386, arm64 to arm64) and export them next to the existing CGO_ENABLED=0 — the CLI is pure Go, so every target cross-compiles. Also derive BINARY_SUFFIX=.exe for windows targets, which release-go.yaml never sets, so Windows assets stop shipping extensionless. An explicitly passed BINARY_SUFFIX still wins, and with no PLATFORM the build falls back to the host toolchain exactly as before. Verified locally: linux-arm64 is ELF ARM aarch64, linux-x64 is ELF x86-64, win32-ia32.exe is PE32 Intel 80386, all with distinct checksums. Existing 1.0.3 assets are left untouched; 1.1.0 ships correct ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 0 + Phase 1 (independent items) of the load-testing initiative, per
docs/superpowers/specs/2026-08-08-load-testing-design.md. Six tasks, each individually reviewed with fix rounds, plus a whole-branch final review (zero Critical findings) and a closing fix wave.New capabilities
flow run f.yaml --vus N --duration 30s(ad-hoc) and--scenario <name>(from the new additiveload:YAML block,constant-vusexecutor). Aggregate console table (p50/p95/p99/RPS/Err% per step) and additiveload_reportobject in the JSON report. Lean execution keeps memory flat; per-VU isolation verified by all-pairs tests under-race.run-flowsGitHub Action (actions/run-flows/): composite action packaging released CLI binaries for CI, with JUnit/JSON reports, job summaries, and its own test workflow (fires on this PR).scenariorunnerVU scheduler,loadmetricsHDR aggregation,CreateFlowRunnerfunctional options, TypeSpec load-metrics envelope (frozen wire shapes incl. threshold verdicts + environment fingerprint).docs/superpowers/specs/phase0-bench.md(~850 RPS per 10 VUs clean on dev hardware; VUs=50 findings root-caused to connection-pool defaults — Phase 2 input).Behavior changes (per spec §4.2 register — ship as CLI minor)
version: 2; imports accept absent/2, reject >2.run:blocks execute in dependency order with strict failure modes: an unknownflow:name aborts pre-flight, an unknown name insidedepends_onis ignored with a warning on stderr exactly as before (shipped example files put cross-flow step names there, so only cycles and duplicate flow names are hard errors), malformed entries error, dependency-failure skips are explicit, and a failed dependency skips-and-continues (previously the whole run aborted).devtools version/load_report.worker_versionnow report the real build version (ldflags injection inbuild:release).Zero-default-change enforcement: 10-fixture golden round-trip corpus (byte-stable), default-path byte-identity proven against base-commit binaries, full gate suite green including new
server:test:race.Known/pre-existing issues documented, not fixed here: exporter drops
run:depends_onandsub_flow_triggernodes (goldens document both in-fixture);TestFlowRun_WebSocketflake (pre-existing, first-priority ticket); JUnit output carries no load data (documented in--help).No visual surface — CLI/engine/CI only; desktop load-results UI is a separate future plan.
🤖 Generated with Claude Code