Feat/parallel builds - #38
Merged
Merged
Conversation
First building block for parallel resource builds. runGraph runs a worker over
a set of nodes forming a dependency DAG:
- Ordering: a node is admitted only once ALL its dependencies have succeeded
(a strict happens-before barrier per edge); independent nodes run in parallel.
- Concurrency: never more than `concurrency` workers in flight; ready nodes are
admitted in input order (deterministic).
- Failure: keepGoing=false (default) fails fast -- first failure stops admitting,
in-flight nodes finish, unstarted nodes are skipped ('aborted'); keepGoing=true
keeps going -- only the transitive dependents of a failed node are skipped.
Resolves with a per-node result map (never rejects on a worker failure -- the
caller decides); rejects only on an impossible graph (cycle / unknown dep).
Pure and self-contained (no listr/docker), so ordering, concurrency and both
failure modes are covered by deterministic gated-worker unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
findRunOrder returns a flat topological list, which is only correct when executed serially; parallel scheduling needs the DAG edges. findRunGraph returns the same predecessor-closure items (topsorted, deps-first) plus a Map of each node's DIRECT dependency ids within the closure -- exactly what runGraph consumes as its happens-before barriers. Extract the shared build-graph / cycle-check / closure / topsort into a private resolveRunSubgraph helper (plus an itemsFor helper). findRunOrder now delegates to it and is otherwise unchanged -- its existing graph.spec tests pass untouched. Tests first: 8 findRunGraph tests (closure, topsort, direct-only edges, diamond, name resolution, ambiguity, runAll, cycles) alongside the unchanged findRunOrder suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The resolution layer for parallel builds, implementing serial-by-default,
opt-in parallelism (flag > config > 1):
- resolveBuildConcurrency({ jobs, configured }): flag wins over config, both
fall back to 1 (serial). 'auto' (from either) resolves to autoConcurrency()
= min(cpu count, AUTO_CONCURRENCY_CAP=4); numeric settings are floored and
clamped to >= 1.
- New config field defaults.build.concurrency: a positive integer or 'auto',
validated by ajv (schema.ts regenerates from schema.json).
Tests first: 7 resolveBuildConcurrency tests plus 4 config-validation tests
(accepts integer/'auto', rejects 0 and non-numeric).
The --jobs/--keep-going CLI flags and the await-commit fix land in the next
phase, where BuildResourcesOperation is rewired onto the scheduler.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire BuildResourcesOperation onto runGraph + findRunGraph so independent
resources build concurrently while dependency edges stay hard happens-before
barriers.
- Each resource is a gated listr task (concurrent: true): it shows
"Waiting for <deps>", blocks until the scheduler grants its turn (all deps
succeeded + a free slot), then runs the normal 4-step sub-tree, or skips if a
dependency failed. The sub-tree is pinned serial so a resource's own steps
keep their order.
- Concurrency = resolveBuildConcurrency({ jobs, configured }): serial by
default; --jobs/-j <n|auto> or defaults.build.concurrency opt in.
- --keep-going/-k: keep building independents after a failure, skip a failed
resource's transitive dependents, then reject with an aggregated error naming
failures + skipped dependents; fail-fast (default) stops admitting new work.
- this.built is now a Set (the dependency-forced-rebuild cascade read is correct
because deps complete before a dependent starts).
- commit() is now awaited (operation + SentinelFileBasedBuilder._commit),
fixing the fire-and-forget sentinel write.
- runGraph gains an onSettle hook so skipped resources' listr tasks can skip
instead of hanging.
- --jobs/--keep-going added to `resources build` and `up` via a shared
buildFlags module (with -j <n|auto> validation).
Tests: the existing 6 operation tests still pass (behavior preserved); new
tests cover dependency order under -j, keep-going (independents continue,
dependents skipped, aggregated rejection) and fail-fast; runGraph's onSettle is
unit-tested. Rendering and real concurrent docker builds are covered next
(docker integration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The concurrent parent list sets exitOnError:false so one resource's
failure does not kill the others. listr2 merges a subtask list's options
over its parent task's, so each resource's own serial sub-list inherited
it and carried on after a failing step:
- a failing cache check was swallowed, the resource built anyway but
never committed its sentinel, and the run reported success — leaving
the cache permanently cold
- a failing prepare step surfaced as "Cannot read properties of
undefined (reading 'build')", masking the real error
Re-assert exitOnError:true on the sub-list. On its own that would hang
the run instead: only the build step settled `buildDone`, so an aborted
chain left the scheduler awaiting a promise that never settles. Route
every fallible step through a guard that rejects `buildDone` before
re-throwing, which also replaces the build step's hand-rolled try/catch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The branch shipped --jobs, --keep-going and defaults.build.concurrency
with no documentation and no changelog entry.
- cli.md: the flags on `emb resources build` and `emb up` (the only two
commands that mount them), plus sections on parallel builds and on
what happens when a build fails
- configuration.md: defaults.build.concurrency, including that
defaults.build rejects unknown keys, so a typo fails validation
rather than being silently ignored
- microservices tutorial: --jobs/--keep-going, and correct the
"independent components can build simultaneously" claim, which
implied parallelism was automatic — it is opt-in, builds are serial
by default
- CHANGELOG: the feature, and the two changes that affect users who
never pass --jobs (aggregated BUILD_FAILED errors, concurrent render)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
secrets.md's "Multiple Environments with Flavors" example nested the
patch operations directly under the flavor name. ProjectFlavorConfig
requires an object, so copy-pasting it made every emb command fail with
"/flavors/staging: must be object" until fixed. Nest them under
`patches:`, as every other page already does.
cli.md claimed --flavor and --json were available for all commands. They
are not, and passing them where they are unsupported is a hard parse
error. Per-command availability was derived from `--help` across all 30
commands:
- only -C/--root, --verbose and --help are truly global
- --flavor is absent from clean, components shell, containers[ prune],
images delete/prune, logs[ archive], tasks[ run] and all kubernetes
commands
- --json is absent from ps, logs, components shell and all kubernetes
commands
Call out that `emb run` takes no --flavor and therefore ignores
EMB_FLAVOR silently — exporting it in CI does not flavor task runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Findings from the docs audit. The source was treated as the truth and
every claim re-derived from it (mostly by running the real CLI):
- remove documented commands that do not exist (containers delete,
components logs, kubernetes logs --tail) and document ones that were
missing entirely (start, stop, logs archive, images push)
- kubernetes ps ignores its argument and is namespace-scoped; kubernetes
restart takes cluster deployment names, not components, and restarts
every deployment in the namespace when given none
- emb up -f also bypasses build caches, so it is a full no-cache rebuild
rather than a cheap --force-recreate
- emb clean gains its destructive -f; emb shell its -s; emb restart its
--no-deps (whose short form is -f and does NOT mean force); emb down
its variadic form; ps/images/containers their -a
- exit codes rewritten from measured behavior: unknown command exits 1
(it falls through to tasks:run), 2 is oclif's parse failure
- flavor patch order was documented backwards. Verified empirically:
component patches apply first, project patches second, so a project
patch wins a conflict on the same key
- namespace precedence omitted the CLI flag, which actually wins
- kubernetes shell defaults to bash, not /bin/sh
- plugins list omitted op; document plugin config surfaces
- a task must define script or pre, which the schema enforces
- store paths have no default/ segment; Node floor stated as 22+
- makefile guide gains the make -jN mapping, and the trap that EMB is
serial by default
Regenerate the four rotten exec blocks from real output. They must be
captured under the validator's environment (CI=1, TERM=dumb, non-TTY),
which renders tables differently from an interactive shell. validate-docs
now passes 33/33 (was 28/33, exit 1), with no loss of coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- package.json engines: >=20.0.0 -> >=22.0.0, matching .nvmrc, CLAUDE.md
and what CI actually tests. The docs already claimed 22+ in places and
20+ in others; they now all say 22+
- resources/build.ts examples had a literal `build` after
`<%= command.id %>`, which already expands to `resources build`. The
rendered README told users to run `emb resources build build`, where
the trailing word is parsed as a resource selector and fails. Fix the
source and regenerate: the run also picks up --jobs/--keep-going and
the --flavor that start/restart gained earlier on this branch
- website/README.md was untouched Starlight scaffolding. Document the
executable-block contract (exec/cwd/skip, the paired output block,
byte-for-byte comparison, npm link), which existed nowhere
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Disabled in e0d9147 ("for now") in January; the docs have drifted freely since, which is how a branch adding --jobs/--keep-going landed with zero website changes. The suite is green again (33/33, exit 0), so re-enable the gate. The website job has no ref condition, so this runs on pull requests targeting master. Drop DEBUG_VALIDATE=1: the validator already prints a per-block result and the full expected/actual diff on failure, so it only added a timestamped line per command. Note the gate is coarser than it looks: validate-docs ignores exit codes, and its link check only asserts internal links start with /emb/ (it resolves neither the page nor the anchor). It catches output drift, which is what rotted here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.