deployment specification

Deployment Specification

This page is the durable reference for deploying noetl-server into any environment. It covers the runtime contract the binary expects, the resources it consumes, the network surface it exposes, and — critically — every environment variable it reads, with the why behind each one.

This page is the single source of truth for the deployment shape. Any code change that adds, renames, removes, or shifts the meaning of an env var MUST update the Environment Variables section in the same change set. Same rule for ports, dependencies, and runtime requirements. See agents/rules/wiki-maintenance.md.

The matching deployment manifests live in noetl/ops (Helm chart + kind overlays). This wiki page describes what the manifests need to provide; the manifests are the implementation.

Component summary

Field	Value
Repo	noetl/server
Binary	noetl-control-plane
Container image	noetl-server (built from the repo's Dockerfile)
Image versioning	crates.io version pinned in Cargo.toml; semver releases tagged vX.Y.Z
Current version	see Cargo.toml package.version
Language / runtime	Rust 1.91+; Tokio multi-threaded
Process model	Single binary, single process per pod

Runtime contract

What the binary expects from its environment to start cleanly:

1. Postgres reachable at ${POSTGRES_HOST}:${POSTGRES_PORT} with the noetl schema migrated. See Database.
2. NATS reachable at ${NATS_URL} (optional but strongly recommended; without it the server runs in degraded mode and doesn't publish command notifications).
3. Encryption key present in NOETL_ENCRYPTION_KEY — required to decrypt credentials at rest. Absent → credential routes return 500.
4. Machine ID in NOETL_SERVER_MACHINE_ID per replica (Phase F R1.5; see Snowflake ID generation). Absent → derives from HOSTNAME; fine for single-replica dev, NOT fine for production replicas (hash collisions produce duplicate snowflakes).

Network surface

Ports

Port	Protocol	Purpose	Bind
8082	HTTP	Main API surface — /api/events, /api/commands/*, /api/executions/*, /api/catalog/*, /api/credentials/*, /api/keychain/*, /api/worker/pool/*, /api/runtime/* (incl. /api/runtime/shard-info for the Phase F R3b drift-guard), /api/vars/*, /api/internal/*	${NOETL_HOST}:${NOETL_PORT} (default 0.0.0.0:8082)
8082/metrics	HTTP	Prometheus scrape endpoint. Gated by NOETL_DISABLE_METRICS	same
8082/healthz	HTTP	Liveness probe (returns 200 once Axum is serving)	same
8082/readyz	HTTP	Readiness probe (200 only when Postgres + NATS are reachable)	same

The metrics + health endpoints share the same port as the API intentionally so the LB only routes one port per replica.

Dependencies (outbound)

Target	Protocol	Why
PostgreSQL	TCP 5432 (default)	Event log, command queue, catalog, credentials, runtime registration. See Data Access Boundary.
NATS JetStream	TCP 4222 (default)	Publish command notifications to noetl.commands.{system|shared}.<execution_id>. Workers consume from these subjects.

The server does not call out to gateway, worker, or any third-party API. All third-party traffic flows through workers.

Resources

Recommended starting point for production. Scale on RAM (Postgres connection pool) and CPU (axum request handling); requests/sec scale linearly with replica count once each replica's Postgres pool is saturated.

Resource	Request	Limit	Notes
CPU	250m	1000m	One replica handles ~1k req/s sustained at p95 < 100ms on the Phase B R4 load smoke.
Memory	256Mi	512Mi	Dominated by sqlx pool + Tokio runtime; no large in-memory state.

Phase F (sharding) scales this horizontally — see sharding-design.

Health probes

Probe	Path	Initial delay	Period	Failure threshold	Effect
Liveness	/healthz	10s	10s	3	Pod restart
Readiness	/readyz	5s	5s	3	Removed from Service endpoints
Startup	/healthz	—	5s	30	Gives the bin 150s to come up before liveness kicks in

/readyz checks Postgres + NATS reachability; the LB takes the replica out of rotation while it's not ready instead of restarting the pod (NATS hiccups don't need pod restarts).

mTLS caveat. When NOETL_TLS_CLIENT_CA is set (mTLS), an HTTP(S) httpGet probe cannot pass — Kubernetes probes don't present a client certificate, so the TLS handshake is rejected before the path is ever evaluated. Switch the liveness/readiness probes to tcpSocket (a port-open check), or terminate mTLS at a sidecar / serve health on a separate non-mTLS port. Server-only TLS (no client CA) works with an httpGet probe at scheme: HTTPS. See Transport security.

Snowflake ID generation

Phase F R1.5 of noetl/ai-meta#49 moved event_id / command_id / execution_id generation out of the DB-side noetl.snowflake_id() function into an app-side generator. See sharding-design and src/snowflake.rs.

Each replica MUST set NOETL_SERVER_MACHINE_ID to a distinct 10-bit value. Without that:

• Local dev / single-node deployments: the server derives a machine_id by hashing HOSTNAME. Fine.
• Multi-replica deployments: two pods with hostname-derived ids can collide on the same 10-bit value. Collisions produce duplicate snowflakes (same execution_id minted by two replicas in the same ms). Set the env var explicitly.

Typical patterns:

• StatefulSet: derive from the pod ordinal — valueFrom: { fieldRef: { fieldPath: metadata.labels['apps.kubernetes.io/pod-index'] } }, pipe through an initContainer that emits the integer. Sub-1024 replicas all distinct.
• Deployment: assign a fixed integer per replica via Helm values (replicaIndex), or generate from a CRD operator.
• Last resort: leave unset and let HOSTNAME hashing do its job; accept the collision risk if replica count stays small.

The id is not security-sensitive; it's a deployment coordination knob. Leaking it does not weaken anything.

Sharding

Phase F R2 of noetl/ai-meta#49 added the ShardConfig + shard_for(execution_id, N) helper in src/sharding.rs. R2 is infrastructure only — the helper is available to handlers but no live request path enforces shard membership yet.

Default behavior (current deployments):

NOETL_SHARD_COUNT=1, NOETL_SHARD_INDEX=0 (both unset). ShardConfig::owns(execution_id) short-circuits to true for every execution — no routing change vs. pre-R2 behavior.

Enabling sharding (Phase F R4+ cutover):

1. Decide N for the cluster (typically a small power of two: 2, 4, 8, 16).
2. Set NOETL_SHARD_COUNT=N cluster-wide (Helm value or ConfigMap reference).
3. Set NOETL_SHARD_INDEX={0..N-1} per-pod — distinct for each replica. See the patterns in Snowflake ID generation (StatefulSet ordinal / Deployment Helm value).
4. Update the gateway / ingress LB to route by hash(execution_id) % N (Phase F R3 — not yet implemented).
5. Partition the per-execution DB tables (Phase F R4).

Routing key derivation:

The shard for a given execution_id is hash(execution_id) % shard_count where hash is twox_hash::XxHash64 with fixed seed 0. The seed + hash crate version pin the assignment forever — changing either invalidates every existing shard mapping, so neither must change once a deployment has started sharding.

src/sharding.rs documents the hash-function choice (xxhash for good avalanche on sequential snowflake i64s; alternatives rejected: DefaultHasher is release-unstable; ahash default is seed-randomized; FNV-1a has weak distribution on sequential i64s).

Diagnostic endpoint (Phase F R3b-1)

Added in noetl-server v2.12.0 (noetl/server#46). Public, deterministic, no auth gate.

GET /api/runtime/shard-info?execution_id=<i64>&shard_count=<u32>

Returns the shard the server's shard_for() selects for the given pair, plus the server's own configured shard for diagnostic completeness:

{
  "execution_id": 320816801799737344,
  "shard_count": 4,
  "shard_index": 2,
  "source": "noetl-server",
  "hash_function": "twox_hash::XxHash64",
  "seed": 0,
  "server_config": {
    "shard_index": 0,
    "shard_count": 1
  }
}

Validation:

• execution_id parses as i64 (decimal); non-numeric → 400.
• shard_count is required, range 1..=1024; 0 or > 1024 → 400.
• No default for shard_count — explicit param to avoid silently mixing the math output with the server's own deployment topology.

Companion endpoints:

• noetl-gateway's twin endpoint (Phase F R3b-2) returns the same shape with source: "noetl-gateway".
• The integration test in noetl/ops (Phase F R3b-3) POSTs to both and asserts they agree across a battery of (execution_id, N) pairs — catches end-to-end drift in the deployed twox-hash versions, seed constants, or byte-encoding choices.

The endpoint is intentionally cheap: pure math, no DB access, no NATS publish. Safe to leave reachable from internal networks; the shard math is not sensitive.

Environment variables

All env vars the binary reads at startup, with the why behind each one.

The envy prefix convention varies by config section:

• NOETL_* — main app config (src/config/app.rs).
• POSTGRES_* — DB config (src/config/database.rs).
• NATS_* — NATS connection (used directly in src/main.rs).
• A few unprefixed vars (DATABASE_URL, HOSTNAME) for compatibility with standard tooling.

Main app (NOETL_*)

Variable	Default	Required	Why
NOETL_HOST	0.0.0.0	no	Bind address for the HTTP server. Override only when you don't want to accept traffic on all interfaces (rare in containers).
NOETL_PORT	8082	no	Port the HTTP server listens on. Match this in the Service spec + readiness probe.
NOETL_WORKERS	(CPU count)	no	Tokio worker thread count. Default of CPU count is correct for most workloads; reduce only for memory-constrained pods.
NOETL_DEBUG	false	no	Enable debug-level tracing + verbose error responses. NEVER true in production — exposes internal error details to clients.
NOETL_SERVER_NAME	noetl-control-plane	no	Identifier the server uses in self-reporting (health endpoints, log fields). Useful when running multiple servers behind one LB.
NOETL_ENCRYPTION_KEY	—	yes	Base64-encoded 32-byte AES-256-GCM key for encrypting credentials at rest. Read at startup; never logged. Rotation is a multi-step procedure (re-encrypt all existing credential rows under the new key before retiring the old). Absent → credential routes return 500.
NOETL_INTERNAL_API_TOKEN	—	when /api/internal/* used	Constant-time-compared bearer token gating /api/internal/* routes (outbox claim, event projection, etc.). Only the system worker pool's K8s ServiceAccount should hold this token; user playbooks must NOT see it. Absent → internal routes return 401.
NOETL_PUBLIC_SERVER_URL	— (localhost fallback)	yes for kind/GKE	URL workers should call back on when they receive a NATS command notification. Embedded verbatim in the server_url field of each NATS message. When unset, a http://localhost:<port> fallback is used; this won't work cross-pod in kind or GKE, so the deployment manifest must override.
NOETL_DISABLE_METRICS	false	no	If true, the /metrics route is not registered. Use only when running behind a separate metrics-export sidecar.
NOETL_AUTO_RECREATE_RUNTIME	true	no	If true, the server re-registers the noetl.runtime row for itself at startup if missing. Set false in environments where runtime rows are managed externally (operator).
NOETL_RUNTIME_SWEEP_INTERVAL	30	no	Seconds between runtime-pool offline-detection sweeps. Larger value = workers take longer to be marked offline after a crash. Default is appropriate for production.
NOETL_RUNTIME_OFFLINE_SECONDS	(see code)	no	Heartbeat staleness threshold before a runtime row is marked offline. Increase only when worker heartbeats are unreliable (poor network).
NOETL_SERVER_MACHINE_ID	(derived from HOSTNAME)	per-replica in prod	10-bit machine id (0–1023) for the application-side snowflake generator. Each pod in a deployment MUST set a distinct value to avoid id collisions. See Snowflake ID generation. Added in Phase F R1.5 (noetl/server#42).
NOETL_SHARD_INDEX	0	per-replica when sharded	Phase F R2 (noetl/server#44). Shard index 0..N-1 this replica owns. Single-replica / pre-sharding deployments leave it unset (defaults to 0 with NOETL_SHARD_COUNT=1 — no enforcement). When NOETL_SHARD_COUNT > 1 each replica MUST set a distinct value. Startup validates shard_index < shard_count and panics otherwise (fail fast on config bug rather than silently mis-route). See Sharding below + sharding-design for the cross-cluster routing context.
NOETL_SHARD_COUNT	1	cluster-wide when sharded	Phase F R2. Total shard count for the cluster. 1 (the default) disables sharding — every replica owns every execution_id and ShardConfig::owns short-circuits to true. Every replica MUST agree on this value or routing diverges. When operators raise it to N > 1 (Phase F R4 cutover), the NOETL_SHARD_INDEX per replica must be set in lockstep.
NOETL_SCHEMA	noetl	no	Postgres schema the server reads from / writes to. Override only when running multiple NoETL deployments in one DB.
NOETL_TLS_CERT	—	with NOETL_TLS_KEY	PEM server-certificate-chain path. Set together with NOETL_TLS_KEY to serve HTTPS instead of plain HTTP. Setting exactly one of cert/key is a fail-fast misconfiguration (startup errors). See Transport security.
NOETL_TLS_KEY	—	with NOETL_TLS_CERT	PEM private-key path for NOETL_TLS_CERT (PKCS#8 / PKCS#1 / SEC1). Never logged.
NOETL_TLS_CLIENT_CA	—	for mTLS	PEM CA-bundle path. When set (cert+key already on), the listener requires + verifies client certs (mTLS) against this CA. Absent → server-only TLS (any client). Enables the authenticated worker↔server credential channel (Secrets Wallet Phase 4).

Secret providers (keychain resolution)

The keychain resolver fetches a provider:-backed credential alias from an external secret manager on a credential-store miss (Secrets Wallet Phase 3, noetl/ai-meta#61). build_secret_provider(provider) selects the backend by the keychain entry's provider id; each reads its config from the environment.

GCP Secret Manager (provider: gcp) — REST :access + ambient GKE Workload-Identity token.

Variable	Default	Required	Why
GOOGLE_CLOUD_PROJECT / GCP_PROJECT	—	when a gcp entry omits its project	Default project for a gcp secret ref that isn't a fully-qualified projects/.../secrets/... path.
NOETL_GCP_SM_ENDPOINT	https://secretmanager.googleapis.com/v1	no	Secret Manager base URL. Override for a mock in tests.
NOETL_GCP_METADATA_TOKEN_URL	(GKE metadata server)	no	Workload-Identity token URL. Override for tests.

Kubernetes Secrets (provider: k8s / kubernetes) — reads a Secret object from the in-cluster API server with the pod's ServiceAccount token, trusting the cluster CA. No cloud credentials; the only dependency is the API server, so this is the one secret backend fully kind-validatable end-to-end. A keychain entry references a value as [<namespace>/]<secret>/<key> (a bare <secret> requires the Secret to hold exactly one data key).

Variable	Default	Required	Why
NOETL_K8S_API_URL	https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT, else https://kubernetes.default.svc	no	API server URL. In-cluster the KUBERNETES_SERVICE_* vars are injected automatically; override only for an out-of-cluster / mock endpoint.
NOETL_K8S_NAMESPACE	projected <sa>/namespace, else default	no	Namespace used when a k8s ref omits its own (<secret>/<key> form). In-cluster the projected file supplies the pod's own namespace.
NOETL_K8S_CA_FILE	/var/run/secrets/kubernetes.io/serviceaccount/ca.crt	no	Cluster CA bundle, added as a trust root. Absent (e.g. an http:// mock) → system roots.
NOETL_K8S_TOKEN_FILE	/var/run/secrets/kubernetes.io/serviceaccount/token	no	ServiceAccount bearer token path; re-read per fetch so projected-token rotation is honored.
NOETL_K8S_TOKEN	—	no	Inline bearer token override (tests / mock API servers). Takes precedence over NOETL_K8S_TOKEN_FILE.

RBAC. The server's ServiceAccount needs get (and list) on secrets in each namespace it resolves from — the default ServiceAccount has none. The deployment manifests in noetl/ops must bind a Role granting secrets: [get, list] to the server SA before a provider: k8s keychain entry can resolve. (Kind validation uses a manually-applied noetl-server-secrets-reader Role/RoleBinding.)

HashiCorp Vault (provider: vault) — reads a KV v2 secret via the Vault REST API (GET <addr>/v1/<mount>/data/<path>), authenticating with X-Vault-Token. Like Kubernetes Secrets, Vault can run in-cluster, so this backend is fully kind-validatable. A keychain entry references a value as [<mount>/]<path>#<key> (a bare [<mount>/]<path> requires the secret to hold exactly one key); the metadata.version is carried as the value version.

Variable	Default	Required	Why
VAULT_ADDR	http://127.0.0.1:8200	yes (real Vault)	Vault server address. In kind: http://vault.<ns>.svc:8200.
VAULT_TOKEN	—	one of token sources	Vault token for the X-Vault-Token header. A platform credential (the server's own auth to Vault), so env/Secret is acceptable per the secrets rule. Production should prefer the Vault Kubernetes auth method (SA JWT → Vault token) over a static token — a follow-up.
NOETL_VAULT_TOKEN_FILE	—	alt to VAULT_TOKEN	Token file path; re-read per fetch (rotating tokens).
VAULT_NAMESPACE	—	Enterprise only	Sent as X-Vault-Namespace.
NOETL_VAULT_KV_MOUNT	secret	no	Default KV v2 mount when a ref omits <mount>/.
NOETL_VAULT_CA_FILE	—	for https:// Vault	CA bundle, added as a trust root.

AWS Secrets Manager (provider: aws / aws_sm) — JSON-over-POST against https://secretsmanager.<region>.amazonaws.com/ action secretsmanager.GetSecretValue, authenticated with hand-rolled AWS Signature Version 4 signing (no aws-sdk dependency). A keychain entry references a value as [<region>:]<secret-id>[#<json-key>] — bare <secret-id> returns the entire SecretString; #<json-key> picks a key out of a JSON-encoded secret (the common AWS convention for multi-field credentials). Cloud-only backend (like GCP); kind validation is at the unit-test layer.

Variable	Default	Required	Why
AWS_ACCESS_KEY_ID	—	yes	Access key id for SigV4 signing. In production this comes from EKS IRSA's temporary credentials (which set all three together).
AWS_SECRET_ACCESS_KEY	—	yes	Secret access key; never logged.
AWS_SESSION_TOKEN	—	with temp creds	Session token sent as X-Amz-Security-Token. Set by IRSA / aws sts assume-role; omit for long-lived IAM-user credentials.
AWS_REGION (or AWS_DEFAULT_REGION)	—	yes	Default region for the regional endpoint; a <region>: prefix on a single ref overrides this.
NOETL_AWS_SM_ENDPOINT	—	tests	Override the endpoint host (mock / VPC endpoint). Production uses the regional default.

IRSA follow-up. This round consumes the static env triple — which IRSA already populates in the pod via its injected credentials. A direct web-identity-token → STS AssumeRoleWithWebIdentity exchange (no env-injection step) is a clearly-scoped follow-up.

Azure Key Vault (provider: azure / azure_kv) — REST GET https://<vault>.vault.azure.net/secrets/<name>[/<version>]?api-version=7.4, authenticating with an OAuth2 bearer token from the Azure Instance Metadata Service (IMDS, used by Managed Identity on AKS / VMs) at http://169.254.169.254/metadata/identity/oauth2/token. A keychain entry references a value as [<vault>/]<secret-name>[#<version>] (a bare <secret-name> uses the default vault; the .vault.azure.net suffix is appended automatically). Cloud-only backend.

Variable	Default	Required	Why
AZURE_KEYVAULT_VAULT	—	when refs omit <vault>/	Default vault short name (e.g. prod-eu). The .vault.azure.net suffix is added by the provider.
AZURE_KEYVAULT_TOKEN	—	tests / mocks	Pre-fetched bearer token used in lieu of IMDS. Production should leave this unset and rely on Managed Identity.
NOETL_AZURE_KEYVAULT_DNS_SUFFIX	vault.azure.net	sovereign clouds	DNS suffix for the vault host (vault.azure.cn China, vault.usgovcloudapi.net US gov).
NOETL_AZURE_KEYVAULT_API_VERSION	7.4	no	Key Vault REST API version.
NOETL_AZURE_IMDS_TOKEN_URL	http://169.254.169.254/metadata/identity/oauth2/token	tests	Override the IMDS endpoint for mocks.

AAD client-credentials follow-up. This round supports IMDS (Managed Identity) + a pre-fetched token. The AAD client-credentials flow (tenant id + client id + client secret) is a follow-up — needed only when a workload is deployed somewhere IMDS isn't available.

Region routing (Phase 6a)

Secrets Wallet Phase 6a (residency-aware distributed resolution) plumbs the keychain entry's home region through the resolver into the provider, so each fetch hits the right regional endpoint / vault / cluster instead of the server's default.

• KeychainDef.region — optional field on each keychain entry (region: us-east-1, region: europe-west4). When set, it propagates into the [SecretRef::region] the provider receives.
• AWS consumes it as the regional endpoint host (secretsmanager.<region>.amazonaws.com) — same shape as the existing <region>: ref-prefix override; the prefix wins when both are set.
• Azure / Vault use it for vault / cluster routing (a per-region vault or Vault cluster lives behind a region-shaped DNS name).
• GCP includes it in the resource id where the project is region-bound.

When KeychainDef.region is unset, the resolver falls back to NOETL_SERVER_REGION (server-side env, see below). When that is also unset (legacy mode), the provider falls back to its own default (e.g. AWS_REGION) and the metric records region="-".

Per agents/rules/observability.md Principle 1 every resolution increments noetl_secret_resolve_total{provider, region, status} — status is ok / provider_fetch_error / template_error. Region is a low-cardinality label (operators deploy into low-tens of regions); resolution latency lives on the matching secret.resolve tracing span (with execution_id per Principle 4), not on the metric.

Variable	Default	Required	Why
NOETL_SERVER_REGION	(empty)	yes for residency-aware deployments	Server's home region (e.g. us-east-1). Used as the fallback when a keychain entry doesn't declare its own region:. Phase 6c (residency enforcement) will additionally compare this against an entry's region to fail-closed on cross-region fetches when residency: strict.
NOETL_SECRET_PROVIDER_TTL_SECONDS	0 (no TTL)	no	Phase 6b — TTL for the [ProviderRegistry] cache of (provider_id, region) → Arc<dyn SecretProvider>. When set, an entry older than the TTL is rebuilt on next access. Useful as an operator escape hatch before Phase 6d's dynamic-secret refresh path lands (short-lived AWS STS / Azure IMDS creds expire faster than the registry's process-lifetime default). Unset or 0 ⇒ cache for process lifetime.

Cross-region broker (Phase 6e)

Phase 6c's residency gate is fail-closed: a server in us-east-1 denied a credential whose home is eu-central-1 returns HTTP 403 — the workflow step fails. That's correct for hard-isolation use cases, but the more common operational shape is "the credential should be resolved IN the EU and the cleartext should never leave EU memory, but the worker that needs it happens to run in US." Phase 6e wires this pattern up by chaining residency-denied resolutions through a broker server in the credential's home region that re-seals the result to the requesting worker via the Phase-5 sealing primitives. Only the sealed envelope crosses the wire; only the addressed worker can open it.

Variable	Default	Required	Why
NOETL_SECRET_BROKER_REGISTRY	""	when running with cross-region fallback	JSON object mapping region → broker_url, e.g. {"eu-central-1": "https://noetl-broker-eu.example.com"}. Empty / malformed → fail-closed pre-6e behaviour (a strict-mode residency denial bubbles up as 403).
NOETL_SECRET_BROKER_TIMEOUT_SECS	10	no	Per-request timeout for the cross-region broker call. Tune up for high-latency interconnects.

New endpoint — POST /api/internal/cross-region/resolve (peer-server side). Body: {alias, worker_public_key_b64, worker_id, execution_id, parent_execution_id, expected_entry_region, requesting_region}. Response: a SealedEnvelope addressed to the requesting worker's pubkey.

Defense in depth: before sealing, the broker compares its own server_region() against the request's expected_entry_region and returns 403 on mismatch. Stale or misconfigured peer registries can't silently coerce a server into serving credentials for the wrong region.

Resolver fallback — when GET /api/credentials/{id}/sealed?worker_id=... fails with AppError::ResidencyViolation and NOETL_SECRET_BROKER_REGISTRY has an entry for the credential's region, the requesting server forwards the call to the broker (with the worker's pubkey carried across) and returns the broker's sealed envelope directly. Cleartext stays in the home region.

Per-credential opt-out — KeychainDef.no_broker_fallback: true forces the strict-mode behaviour regardless of registry config. Used for credentials whose policy says "this data physically cannot leave its home region, full stop."

Two new metrics:

• noetl_secret_broker_call_total{broker_region, outcome} — counter. outcome ∈ {ok, unreachable, denied_by_broker, wrong_region, bad_pubkey, resolve_error, serialize_error, seal_error}. wrong_region is the alert-worthy combination — it means a peer's broker registry is out of date.
• noetl_secret_broker_call_duration_seconds{broker_region} — histogram of cross-region round-trip latency, bucketed [0.05, 0.1, 0.25, 0.5, 1, 2, 5] seconds. Observed regardless of outcome so a dashboard surfaces "broker is slow" + "broker is failing" independently.

Dynamic short-lived secrets (Phase 6d primitives)

Some providers return secrets the issuer expires on a clock — AWS STS bearer tokens (15 min – 12 h), AAD access tokens (1 h default), GCP iamcredentials.generateAccessToken (1 h default, 12 h max), OAuth2 access tokens with expires_in. The Phase-3c keychain cache previously used a fixed 600 s TTL; caching a token past its expires_at means the next worker fetch gets a 401 and the playbook step fails.

Phase 6d adds two primitives so the concrete cloud-specific dynamic providers can land cleanly in follow-up rounds (6d.1 / 6d.2 / 6d.3):

• SecretValue.expires_at: Option<DateTime<Utc>> — issuer-reported expiry; providers that don't know leave it None.
• secrets::dynamic::effective_cache_ttl(expires_at, default_ttl, now) returns CacheFor(secs) for normal-case secrets, or SkipCacheAlreadyExpired when the issuer's deadline is already past (or inside the safety margin). Honours `min(default_ttl, expires_at
  • now - safety_margin)`, floored at 5 s.

The cache-write site (CredentialService::resolve_via_provider) now asks the helper for the cache TTL instead of using KEYCHAIN_CACHE_TTL_SECS unconditionally. When the helper returns SkipCacheAlreadyExpired the resolver logs a WARN, bumps the noetl_secret_cache_skip_total{reason="already_expired"} counter, and returns the freshly-resolved value to the caller without writing the cache.

For a map-shaped keychain entry that bundles several secrets, the effective expires_at is the earliest of any contributing secret's expiry — caching the bundle past the soonest expiry would mean the next worker fetch gets a 401 from whichever member already expired.

Variable	Default	Required	Why
KEYCHAIN_CACHE_DYNAMIC_SAFETY_MARGIN_SECS	60	no	How long before the issuer-reported expires_at to stop caching. Wider buffer for high-latency networks; narrower buffer when the operator wants to squeeze more reuse out of short-lived tokens.

Two new metrics:

• noetl_secret_dynamic_ttl_seconds — histogram of issuer-reported TTL at resolution time. Buckets [60, 300, 900, 3600, 14400, 43200] (1 min / 5 min / 15 min / 1 h / 4 h / 12 h). Tells operators whether their fleet is hot-pathing through short-lived creds.
• noetl_secret_cache_skip_total{reason} — counter; reason is already_expired (other reasons may follow in 6d.x rounds).

Concrete dynamic-secret providers (STS / AAD / iamcredentials) ride follow-up rounds.

Residency policy (Phase 6c)

Phase 6c puts a fail-closed gate in front of the resolver. A credential tagged region: eu-central-1 exists for a reason — usually data-residency obligations (GDPR's "data must stay in the EU") or contractual constraints. Without enforcement, a server in us-east-1 resolving an EU-locked credential routes the fetch to the EU endpoint (Phase 6a), but the cleartext still lands in this server's memory. This module is the gate that prevents the crossing.

Three values for KeychainDef.residency:

Value	Behavior on server_region ≠ entry_region
none (default)	Resolution proceeds. Back-compat for entries that pre-date 6c.
advisory	Resolution proceeds AND noetl_secret_residency_check_total{policy="advisory", decision="violation_allowed"} bumps. Operator-facing surface for the migration window before flipping to strict.
strict	Fail-closed. Resolver short-circuits with AppError::ResidencyViolation (HTTP 403, clear "credential X is region-locked to Y; this server is in Z" message that NEVER includes the value). Cleartext never enters this server's memory. Metric bumps with decision="violation_blocked" — alert-worthy.

Operators can grant per-credential allowlist exceptions via KeychainDef.allowed_regions: [...] — when this server's region appears there, resolution is allowed under any policy (decision="allowed_in_allowlist"). The empty string never matches an empty server region (defensive: a misconfigured allowlist can't silently disable the gate).

The gate runs BEFORE the provider registry lookup, so a strict-mode violation does NOT create a ProviderRegistry entry, does NOT call fetch, and does NOT increment the Phase 6b duration histogram. Only the residency-check counter records the denial.

Cross-region routing is the natural follow-up (Phase 6e): "if a server in A is denied a credential whose home is B, route the request through a broker in B that re-seals to A's worker." Until that ships, strict mode is a fail-closed boundary, not a routed-around constraint.

New metric. noetl_secret_residency_check_total{policy, decision}:

policy	decision	When
none	allowed_no_policy	Default policy or empty entry_region; no enforcement happened.
advisory	allowed_same_region	Server region matches entry region.
advisory	allowed_in_allowlist	Server region is in allowed_regions.
advisory	violation_allowed	Mismatch, but advisory mode lets it through. Migration-window signal.
strict	allowed_same_region / allowed_in_allowlist	Same as advisory.
strict	violation_blocked	Fail-closed denial. Alert.

Provider registry caching (Phase 6b)

The resolver's cache-miss path (resolve_keychain_entry) used to call build_secret_provider(provider) on every fetch — re-reading env vars, rebuilding the reqwest::Client (TLS bundle reparse on the rustls path), reparsing IMDS / token state. Phase 6b adds a server-side ProviderRegistry keyed by (provider_id, region) so the per-region instance is built once and reused.

Two new metrics per agents/rules/observability.md Principle 1:

• noetl_secret_provider_build_total{provider, region, status} — counter. status is cache_hit / ok / error. Together with the Phase 6a noetl_secret_resolve_total this answers two operator questions: "is the cache effective?" (cache_hit / (ok + cache_hit) ratio) and "is a region's provider down?" (error per-region rate).
• noetl_secret_resolve_duration_seconds{provider, region} — histogram of resolve wall-clock latency, bucketed [5 ms, 10 ms, 25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 s, 2 s, 5 s] to span the range where cloud secret managers and Vault clusters actually live. Observed regardless of outcome so a dashboard surfaces "everything's slow" + "everything's failing" independently (timeouts dominate failure-mode wall-clock).

execution_id is NOT a label on either — it lives on the matching secret.resolve tracing span per Principle 4.

Transport security (TLS / mTLS)

Secrets Wallet Phase 4a (noetl/ai-meta#61, noetl/server#103) — the transport half of sealed secret delivery. The control-plane API serves plain HTTP by default (unchanged); the listener opts in to TLS purely through the three NOETL_TLS_* env vars above:

NOETL_TLS_CERT	NOETL_TLS_KEY	NOETL_TLS_CLIENT_CA	Mode
unset	unset	—	Plain HTTP (default)
set	set	unset	HTTPS, any client
set	set	set	mTLS — client cert required + verified against the CA
exactly one of cert/key		—	Startup error (fail fast)

TLS is built on the ring rustls provider the rest of the stack already uses (no aws-lc-rs clash); axum-server bind_rustls serves the encrypted listener with graceful shutdown. mTLS uses a WebPkiClientVerifier over the NOETL_TLS_CLIENT_CA bundle.

The cert/key/CA are mounted into the pod from a K8s Secret; the manifests that wire the volume + env live in noetl/ops. mTLS is what authenticates + encrypts the worker→server credential fetch (GET /api/credentials/<alias>) so the resolved secret no longer travels plaintext on the wire. The worker-side mTLS client (ControlPlaneClient) is Phase 4b; encrypting the payload to the worker's key is Phase 5.

Probe interaction: see the mTLS caveat — with a client CA set, switch K8s probes to tcpSocket.

Sealed credential delivery (Phase 5)

Secrets Wallet Phase 5 (noetl/ai-meta#61) adds defense-in-depth on top of the Phase-4 mTLS transport: mTLS encrypts the wire, sealing encrypts the payload to a key only the recipient worker holds. The cleartext exists only briefly inside the server process at seal time; an operator with kubectl exec on the server pod sees only ciphertext in the response body.

• Phase 5a (noetl/server#107, v2.32.0) — src/crypto/sealed.rs primitives.
• Phase 5b (noetl/server#108) — wire format + endpoint (this section).
• Phase 5c — worker side (ephemeral keypair + unseal + zeroize).

Worker registration: the worker opts in by including a base64-encoded 32-byte X25519 public key in the runtime JSONB blob of POST /api/worker/pool/register:

{
  "name": "worker-rust-pool",
  "kind": "worker_pool",
  "status": "ready",
  "runtime": {
    "worker_public_key": "<base64-32-byte-x25519-pub>"
  }
}

No schema migration is needed — the runtime column already accepts arbitrary metadata. Workers that don't send a key keep working unchanged (only the sealing endpoint reads the field).

Endpoint: GET /api/credentials/{identifier}/sealed?worker_id=<name>

Returns a SealedEnvelope JSON addressed to the worker named by worker_id (matched against the kind=worker_pool row in noetl.runtime):

{
  "alg": "x25519-hkdf-sha256-chacha20-poly1305",
  "v": 1,
  "eph_pub":    "<32 bytes b64>",
  "ciphertext": "<n+16 bytes b64>"
}

The plaintext that the AEAD ciphertext encrypts is the same JSON CredentialResponse shape GET /api/credentials/{identifier} returns with include_data=true. The worker recovers it by running X25519 ECDH against its long-lived secret + the envelope's eph_pub, deriving the AEAD key + a 12-byte nonce via HKDF-SHA256 with info "noetl-sealed-v1", and ChaCha20- Poly1305 decrypting with AAD <alg>|v=<v>.

400 BadRequest is returned when the worker_pool row exists but didn't register a worker_public_key (or the row is missing entirely) — the error message points at the runtime-row condition so the operator can fix registration.

Observability: noetl_credentials_sealed_total{status} counter where status ∈ {ok, no_pubkey, worker_not_found, seal_error, credential_error}; pairs with a credential.seal span that carries worker_id, identifier, and (when provided) execution_id.

Database (POSTGRES_* + DATABASE_URL)

Variable	Default	Required	Why
DATABASE_URL	—	when set, overrides individual POSTGRES_*	Full Postgres connection URL. Standard ecosystem env var; honored as an override for cases where the connection comes from a managed Postgres service that emits a URL.
POSTGRES_HOST	localhost	yes	Postgres host. In kind: postgres.noetl.svc; in GKE: the Cloud SQL Proxy sidecar address.
POSTGRES_PORT	5432	no	Postgres port.
POSTGRES_USER	noetl	yes	Postgres user; needs SELECT/INSERT/UPDATE/DELETE on noetl.* plus EXECUTE on noetl.snowflake_id() (the latter is the DB-side fallback per observability.md).
POSTGRES_PASSWORD	(empty)	yes for non-trust auth	Postgres password. Should come from a K8s Secret, not the Deployment env directly.
POSTGRES_DATABASE	noetl	no	Postgres database name.
POSTGRES_MAX_CONNECTIONS	10	no	sqlx pool max. Phase F R4 sharding will partition this across shards; until then, the math is replicas × 10 ≤ Postgres max_connections − headroom. See agents/rules/data-access-boundary.md.
POSTGRES_MIN_CONNECTIONS	1	no	sqlx pool min — keep at least one connection warm.
POSTGRES_ACQUIRE_TIMEOUT	30	no	Seconds before a pool.acquire() call fails. Increase only when Postgres is known to be slow under load.

NATS

Variable	Default	Required	Why
NATS_URL	—	recommended	NATS server URL (nats://<host>:4222). Without it, the server runs in degraded mode and doesn't publish command notifications. Workers see no work; useful for read-only deployments.
NATS_USER	—	when NATS auth	NATS user.
NATS_PASSWORD	—	when NATS auth	NATS password. Should come from a K8s Secret.

The NATS connection accepts a user:pass@ segment embedded in NATS_URL; the NATS_USER / NATS_PASSWORD form is the preferred shape per async_nats::ConnectOptions::with_user_and_password.

Misc / standard tooling

Variable	Default	Required	Why
HOSTNAME	(set by container runtime)	—	Fallback source for the snowflake machine_id when NOETL_SERVER_MACHINE_ID is unset. Read at startup, hashed via FNV-1a to 10 bits.
COMPUTERNAME	(Windows)	—	Same fallback as HOSTNAME on Windows hosts (rare for production but useful for local dev).
RUST_LOG	(build default)	no	Standard tracing-subscriber filter. Default sufficient; set to noetl_server=debug,axum=info for targeted debugging.

Secret-resolution audit service (Phase 7b primitives)

Phase 7b adds a durable audit trail of every credential resolution. Compliance regimes (SOC 2, ISO 27001, FedRAMP, PCI-DSS) require a queryable trail with retention measured in years; the tracing-span surface alone evaporates with log retention. This round ships the in-process service — AuditSink trait, AuditEvent shape, sync

• async write modes, the strict-mode env gate, and the new metric. Phase 7b.2 wires the actual noetl.secret_audit table + the GET /api/internal/secret-audit query endpoint + the handler integration site.

Wire shape (AuditEvent):

Field	Type	Notes
audit_id	i64	Application-side snowflake (per observability.md Principle 3).
occurred_at	TIMESTAMPTZ	Wall-clock at event construction.
credential	TEXT	Alias / identifier the resolver was asked for.
operation	TEXT	Bounded enum: get_sealed / cross_region_broker_serve / resolve_keychain / get_credential.
outcome	TEXT	Bounded enum: ok / residency_violation / broker_unreachable / credential_not_found / no_pubkey / wrong_region / provider_fetch_error / template_error.
worker_id	TEXT?	When the surface knew one (sealed path).
execution_id, parent_execution_id	i64?	Forwarded execution context.
server_region	TEXT?	Which server-region served the request.
broker_region	TEXT?	When the resolution chained through Phase-6e cross-region routing.
kek_version	TEXT?	The KEK version on the record at access time (Phase-7a).
notes	TEXT?	Short free-text for operators. Never the secret value.

Two write modes:

• Async fire-and-forget (default) — SecretAuditService::record_async spawns a tokio task; the resolver doesn't block. Failed writes log a span warning and increment noetl_secret_audit_writes_total{status="dropped_async"}. Used when the audit is for forensics, not gate.
• Strict — SecretAuditService::record_strict awaits the write. If it fails, the resolution propagates the error. Used when compliance requires the row exist BEFORE the value is released. Flip via NOETL_SECRET_AUDIT_REQUIRED=true.

Variable	Default	Required	Why
NOETL_SECRET_AUDIT_REQUIRED	false	no	When true (or 1 / yes), strict-mode is on: a failed audit write blocks the resolution. Used for compliance regimes that mandate the row exist before the value is released.

New metric. noetl_secret_audit_writes_total{operation, outcome, status} — counter. status ∈ {written, dropped_async, failed_strict}. failed_strict is the alert-worthy combination — it means the wallet refused to release a credential because the audit couldn't be recorded.

Wallet KEK rotation primitives (Phase 7a)

Phase 7 of the Secrets Wallet umbrella (noetl/ai-meta#61) hardens the wallet for the operational lifecycle — rotation, audit, and auto-renewal. Phase 7a ships the rotation primitives: per-record KEK version tracking surfaces (already present in the Phase-1 storage format via WrappedDek.key_version), the KeyManager trait reports its current key version, and EnvelopeCipher::rewrap_storage_string turns one stored envelope into a re-wrapped one without ever reconstructing the plaintext.

The actual rotation endpoint + table scans land in Phase 7a.2 — this round establishes the in-process API the rotation job will iterate.

API surface:

• KeyManager::current_key_version() -> &str — what the next wrap_dek call will tag. LocalDevKms reports the version string from its constructor ("v1" by default); real KMS implementations override with their provider-reported version.
• EnvelopeCipher::rewrap_storage_string(raw: &str) -> AppResult<RewrapOutcome>:
  • Parses raw as a stored envelope (StoredEnvelope JSON).
  • If wrapped.key_version == current_key_version() → returns RewrapOutcome::Skipped { key_version }.
  • Else: unwraps the DEK under the historical KEK version, re-wraps under the current version, returns RewrapOutcome::Rewrapped { old_key_version, new_key_version, new_storage_string }.
  • Plaintext is never reconstructed — pure DEK re-wrap; AES-GCM ciphertext bytes stay byte-identical.

New metric:

noetl_wallet_rotate_total{table, status} — counter, per agents/rules/observability.md Principle 1. table ∈ {credential, keychain}; status ∈ {skipped, rewrapped, failed_unwrap, failed_wrap, parse_error}. failed_unwrap is the alert-worthy combination — it means the rotation can't complete without operator intervention (the KMS deleted the historical key version).

Operator workflow (Phase 7a.2 endpoint surface, pending):

1. Operator bumps the KEK version in the KMS provider (GCP KMS: :rotateAtTime; AWS KMS: aws kms create-key-rotation-schedule).
2. New records (POST /api/credentials) automatically use the new version — no migration needed for them.
3. To re-wrap existing records, operator triggers the rotation job (POST /api/internal/wallet/rotate-kek, 7a.2). Each row's stored envelope flows through rewrap_storage_string; skips are cheap (no KMS call), rewraps emit the new storage string back to the row.
4. Diagnostic GET /api/internal/wallet/key-status (7a.2) reports per-table per-version counts so operators can confirm the rotation completed before deleting the old KMS version.

Secrets handling

Secrets must NOT be passed via plain Deployment.spec.template.spec.containers[].env[].value.

Secret	Storage	Mount as
NOETL_ENCRYPTION_KEY	K8s Secret (production) / sealed-secret (kind)	valueFrom.secretKeyRef
POSTGRES_PASSWORD	K8s Secret	valueFrom.secretKeyRef
NATS_PASSWORD	K8s Secret	valueFrom.secretKeyRef
NOETL_INTERNAL_API_TOKEN	K8s Secret	valueFrom.secretKeyRef

In Cloud Build / GKE the K8s Secrets are projected from GCP Secret Manager via the standard CSI driver.

Per agents/rules/execution-model.md "Secrets and credentials rule": business-logic secrets (third- party API tokens, tenant database DSNs) belong in the NoETL keychain, NOT in env vars. The env vars above are the platform-runtime secrets — the server's own keys for talking to its own infrastructure.

Observability

• Metrics: Prometheus surface at /metrics per agents/rules/observability.md. Cardinality discipline: execution_id is a span attribute, NOT a metric label (would blow up the registry).
• Tracing: tracing spans on every boundary; execution_id is a span field.
• Logs: structured JSON via tracing-subscriber's json layer. Default filter suppresses health endpoint noise per agents/rules/logging.md.
• Snowflake ID is logged once at startup with its derivation source (NOETL_SERVER_MACHINE_ID vs derived from HOSTNAME). Useful for confirming the per-replica configuration is correct.

Validation procedure

When changing any of the above, validate per agents/rules/deployment-validation.md:

1. cargo build --release --bins
2. cargo test --quiet
3. Build image locally + load into kind: kind load docker-image …
4. Apply the manifests against kubectl --context kind-noetl
5. Smoke-test the changed surface:
   • For env-var changes: confirm the var lands in kubectl exec … env | grep <VAR> AND that the binary behaves as documented when the value is present, absent, or invalid.
   • For port / probe changes: confirm kubectl get svc shows the right port + kubectl describe pod shows the probes passing.

Only after kind passes does the change roll forward to Cloud Build + GKE.

See also

• sharding-design — Phase F design that introduced NOETL_SERVER_MACHINE_ID.
• event-envelope — the wire format the server's HTTP API uses.
• runtime-shape — the 4-binary deployment blueprint (server + publisher + projector + system_pool).
• noetl/ops Helm chart + manifests — the deployment-time implementation.
• agents/rules/wiki-maintenance.md — the rule that requires this page to update in lockstep with code.