feat: add flowmesh stack bundle init#38
Merged
Merged
Conversation
`bundle export` only works inside a repo checkout — it reads
secrets/tls/* and configs/worker_config.yaml from CWD and shells out to
`pip wheel ./sdk ./cli`. A deploy host that installed `flowmesh[cli]`
from PyPI has none of those sources, so there was no way to land on the
on-disk layout `stack up` expects without first producing a tarball
elsewhere and `tar -xzf`ing it.
`bundle init` scaffolds that layout directly: empty
secrets/tls/{server,redis}/ placeholders, an empty
configs/worker_config.yaml, and a .env from the shipped example. The
TLS/worker_config paths are now driven by module-level constants shared
with `_copy_server_assets`, and those constants match the defaults
already encoded in .env.example, env_schema.py, and compose.yml
(secrets/tls/..., configs/worker_config.yaml) — `stack up` resolves to
the same paths bundle init / bundle export write to, so the bundle is
operable without editing path values in .env first.
Drive-by fix: `_copy_server_assets` now creates the configs/ parent
before copying worker_config.yaml. Previously worked only because
the destination filename had no parent component; the layout move
needed the mkdir to keep `bundle export` from crashing on the copy.
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
The is_resource call lived inside an `if not …: pass` block that did nothing, so it only existed to emit DeprecationWarning when callers exercised asset_path. as_file already raises FileNotFoundError on a missing resource, which the surrounding try/except already maps to AssetNotFoundError. Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
bundle_export unconditionally called _copy_redis_tls_assets(ca_only=True), a leftover from an earlier migration. A root-node bundle skipped redis-server.pem / redis-server.key, so the exported tarball couldn't boot local Redis even though the shipped .env.example and compose.yml point at those files. Surface the node role as a positional argument to bundle_export (default root); ca_only is now role == worker, so a root bundle stages the full Redis TLS material and a worker bundle keeps the CA-only shape. The role string was already encoded in stack.py and env_schema.py as ad-hoc literals. Promote it to a shared StrEnum (flowmesh.models.nodes. NodeRole), widen EnvVar.choices to Iterable[str] so the enum class can be passed directly, and replace the literals at every callsite. test_schema_compat.py adds NodeRole to the SDK <-> server enum-pair compat check. bundle_init next-steps fixes: - Append --env-file <path> to the printed `flowmesh stack pull` / `up` lines when --env-file is non-default (the bare commands would have read ./.env). - Suppress the "drop TLS certs into ..." line when --no-tls is set. Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
The previous commit on this branch role-gated only the Redis TLS file copy in bundle export. install.sh and stack init kept writing the static .env.example verbatim, so a worker-mode bundle still landed on a NODE_ROLE=root config that expected local Redis and the cert/key files the worker bundle had just declined to ship. Switch stack init from copying the checked-in .env.example to rendering live from STACK_ENV_SCHEMA via render_env_example, which gained an optional `overrides: Mapping[str, str]` kwarg. The shipped example is still authoritative (scripts/dev/check_env_examples.py verifies the root render equals the tracked copy), but stack init can now produce worker-shaped output without a parallel template. Add WORKER_ROLE_OVERRIDES next to STACK_ENV_SCHEMA: NODE_ROLE flips to "worker"; REDIS_TLS_CERT_FILE / REDIS_TLS_KEY_FILE blank out because src/server only reads REDIS_TLS_CA_FILE and the cert/key files are consumed exclusively by the root-profile-gated Redis services. The connection-side knobs (REDIS_*_URL, ACL, credentials, CA) stay populated; the operator still has to repoint REDIS_CONTROL_URL / REDIS_TELEMETRY_URL at the root node before stack up, which is intrinsic since only the operator knows that address. Plumb `--role` through stack init, bundle init, and bundle export's install.sh so a worker bundle's bootstrap chain ends in `flowmesh stack init --env-file "\$ENV_FILE" --role worker`. A shared parse_node_role helper replaces the inline try/except now duplicated across three callsites. Coverage: test_worker_role_render_passes_schema_validation runs the rendered worker .env through validate_env_values to pin the contract that role overrides don't trip schema-level required/min_value checks — catches drift if a future required=True lands on a blanked key. Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
stack init writes FLOWMESH_VERSION=dev (the local-iteration placeholder) by default. For deploy-shaped scaffolding the right tag is the one matching the running flowmesh-cli-stack, so compose pulls server/worker images at the same version the CLI was installed at. --deploy reads the installed package version via importlib.metadata and pins FLOWMESH_VERSION to it, falling back to 'latest' with a warning when the metadata is missing. The fallback keeps the bootstrap usable on hosts where the package isn't pip-installed in a way that exposes metadata; the operator can edit .env after. bundle init implies --deploy; bundle export's install.sh now emits --deploy in its chained stack init call. Combined with install.sh pinning flowmesh[cli]==X via _published_cli_spec, the resulting deploy host installs the pinned CLI, scaffolds .env at the same version, and compose pulls aligned images. No literal "latest" lives in the schema or override path. Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
e23b8b1 to
ad8c142
Compare
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Give pip-only deploy hosts a one-command bootstrap, and make the bootstrap role-aware so a worker bundle actually produces a worker-shaped deployment.
flowmesh stack bundle exportonly works inside a repo checkout (it readssecrets/tls/*andconfigs/worker_config.yamlfrom CWD, and shells out topip wheel ./sdk ./clifor the wheel bundle). A host that ranpip install flowmesh[cli]from PyPI has none of those sources, so the only way to land on the layoutstack upexpects was to produce a tarball elsewhere andtar -xzfit. And worker-mode bundles were previously half-broken: the tarball staging skipped the Redis cert/key files, but the chainedflowmesh stack initstill wrote aNODE_ROLE=rootenv that expected them.Changes
flowmesh stack bundle initscaffolds the deploy layout in-place (secrets/tls/{server,redis}/,configs/worker_config.yaml,.env) and chainsstack init. Layout matches the shipped.env.example/compose.ymldefaults, sostack upresolves to the paths the bundle wrote to.--roleis end-to-end role-aware acrossstack init,bundle init, andbundle export <role>. FlipsNODE_ROLEand blanks the Redis cert/key paths on worker; the worker bundle'sinstall.shpropagates--role workerinto its chained init. Workerbundle initnext-steps now point operators atredis-ca.pemonly, matching the scaffold.stack init --deploypinsFLOWMESH_VERSIONtov<installed flowmesh-cli-stack version>(fallbacklatest+warning if metadata is missing). Thevprefix matches the GHCR tag convention enforced byrelease-images.yml.bundle initimplies it;bundle export'sinstall.shappends--deployso CLI install and image pull land at the same version.install.shanchors to the bundle directory viacd "$(dirname "$0")", so./flowmesh_server_bundle/install.shworks from any CWD without scattering.venv,.env, or--include-wheelslookups outside the bundle.lumid-hooks/lumid-data-sdkswitched from git+url to PyPI pins; deadis_resourcebranch inassets.pyremoved (it was firingDeprecationWarningon everyasset_pathcall).Design
stack initnow renders live fromSTACK_ENV_SCHEMAinstead of copying the static.env.example. The shipped example stays the human reference, CI-verified against the root render.render_env_exampletakes anoverridesmap;WORKER_ROLE_OVERRIDES(next to the schema) holds the keys whose worker default diverges from root, and--deployadds a single dynamic override forFLOWMESH_VERSION.Image tag alignment happens automatically:
bundle export'sinstall.shpinsflowmesh[cli]==Xand then runsstack init --deploy, which reads the just-installed package's version viaimportlib.metadata— same X — and pinsFLOWMESH_VERSION=vX. No literallatestlives in the schema or override path.bundle initis non-destructive: existingworker_config.yamland TLS dirs are preserved;--forceonly governs.env.test_worker_role_render_passes_schema_validationpins the contract that rendered worker.envs are valid by the schema's own validators, so future drift trips at PR time, not on the deploy host.Test Plan
Manual end-to-end smoke from a clean directory:
Test Result
Pre-submission Checklist