We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
deployment-spec: GCS auth-mode env (WI/ADC) for the result tier (#104) Document NOETL_OBJECT_STORE_GCS_AUTH (auto/none/static/adc) and the Workload-Identity / ADC token-minting path the GCS object backend now takes in prod; clarify NOETL_OBJECT_STORE_GCS_TOKEN as the static-only override and NOETL_OBJECT_STORE_GCS_ENDPOINT as the auto-mode selector. Tracks noetl/server#265 (review).
docs(deployment-spec): NOETL_RESULT_TIER_GC env var (#104 Phase F)
docs(deployment-spec): NOETL_RESULT_MINT_AUTHORITATIVE (#104 Phase D)
deployment-spec: add #104 Phase C result object-store + cell-registry env vars NOETL_OBJECT_STORE_BACKEND/GCS_{ENDPOINT,BUCKET,TOKEN} + NOETL_RESULT_CELL{,_ENV,_REGION}/NOETL_RESULT_SHARD_COUNT — the GCS object backend + cell registry for the resolve-by-URN read path (noetl/server#262).
deployment-spec: document NOETL_RESULT_URI_ACCEPT (#104 Phase A) The server's canonical-result-URI shadow-accept flag (noetl/ai-meta#104 Phase A, noetl/server#260) — default false / byte-identical no-op; flag-on parses + validates reference.uri via noetl_tools::locator and records noetl_result_uri_accept_total{outcome}, never failing the event. Refs noetl/ai-meta#104
deployment-spec: add NOETL_REPLICA_COHERENCE env var (#115 program-scale, server v3.38.0) Multi-replica coherence for the off-server drive's watermark + descriptor: local (default) vs nats_kv (JetStream KV buckets, head CAS + descriptor merge). Per wiki-maintenance Rule 2a (env-var catalogue is the source of truth).
deployment-spec: document NOETL_EVENT_INGEST_PUBLISH_ONLY (CQRS write-path cutover gate, v3.29.0+; cancel/finalize joined v3.29.3)
deployment-spec: retire NOETL_ORCHESTRATE_PLUGIN_SHADOW row + shadow metric The in-server orchestrate shadow + orchestrate-shadow build feature + NOETL_ORCHESTRATE_PLUGIN_SHADOW were the slice-4 cutover-confidence harness for noetl/ai-meta#108. Retired in noetl/ai-meta#110 now that the worker-driven drive is default-on and proven (drops the wasmtime server dep). Trimmed the stale shadow sentence from the DRIVE row too. Refs noetl/ai-meta#110
docs(deployment-spec): NOETL_ORCHESTRATE_PLUGIN_DRIVE now defaults true (#108 (c)) The worker-driven orchestrator drive is on by default after the scale soak proved zero noetl.event burst + full system-pool isolation. Records the revert (NOETL_ORCHESTRATE_PLUGIN_DRIVE=false) and the system-pool prerequisite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
deployment-spec: NOETL_ORCHESTRATE_PLUGIN_DRIVE — worker-driven drive (#108 slice 3) server: when on, issues system/orchestrate to the worker pool instead of driving in-process; the drive runs off-server on the pool. Default off.
deployment-spec: NOETL_ORCHESTRATE_PLUGIN_SHADOW — in-server plug-in shadow (#108 slice 4) server: when on (+ orchestrate-shadow feature), every drive runs the plug-in on the same WorkflowState and diffs commands; metric noetl_orchestrate_shadow_total.
deployment-spec: NOETL_SYSTEM_PLUGIN_DIR — boot-seed built-in system plug-ins (#108) server: seeds system/<stem>@1 from this dir into noetl.plugin_module on boot; image bakes orchestrate.wasm at the default /opt/noetl/plugins.
docs(server): NOETL_PROJECTOR_OWNS_SNAPSHOT + /projection/advance (#103 2b, server#204)
docs(server): CQRS write-path env vars — NOETL_EVENT_STREAM_* tailer (#103 2a, server#202) Replaces the NOETL_EVENT_OUTBOX_ENABLED trigger section (server#201, closed) with the event-log → JetStream tailer config + metrics.
docs(server): NOETL_EVENT_OUTBOX_ENABLED — CQRS write-path producer (#103 2a, server#201)
deployment-spec: add NOETL_PG_STATEMENT_CACHE_CAPACITY env var sqlx prepared-statement cache capacity (v3.5.1, noetl/server#191). Default 100 (sqlx default); set to 0 behind a transaction-mode pooler (prod pgbouncer pool_mode=transaction in front of Cloud SQL) to avoid 'prepared statement does not exist' errors.
docs: add /api/result/* to the network surface (v2.58.0; ai-meta#70)
deployment-specification: Phase 7c token auto-renewal primitives secrets::dynamic::should_refresh decision primitive (true when expires_at is set AND still valid AND inside the refresh window); KEYCHAIN_CACHE_REFRESH_WINDOW_SECS env (default 60); noetl_secret_refresh_total{outcome} counter (triggered/succeeded/ failed/stampede_collapsed; failed alert-worthy) + noetl_secret_refresh_duration_seconds histogram (50ms-5s buckets). Cache + resolver wire-up + stampede mutex deferred to Phase 7c.2. Tracks noetl/ai-meta#61 Phase 7c + noetl/server#124.
deployment-specification: Phase 7b secret-resolution audit service AuditEvent wire shape; AuditSink trait + NoopAuditSink default; SecretAuditService with record_async + record_strict modes; NOETL_SECRET_AUDIT_REQUIRED env (default false; true enables strict mode that blocks resolution on audit-write failure); noetl_secret_audit_writes_total{operation, outcome, status} counter (failed_strict alert-worthy). DB sink + GET /api/internal/secret-audit query endpoint + handler integration deferred to Phase 7b.2. Tracks noetl/ai-meta#61 Phase 7b + noetl/server#122.
deployment-specification: Phase 7a wallet KEK rotation primitives KeyManager::current_key_version() trait accessor; EnvelopeCipher::rewrap_storage_string primitive (parse → check version → if older, unwrap + re-wrap → return new storage string); noetl_wallet_rotate_total{table, status} counter. Plaintext never reconstructed during rotation — pure DEK re-wrap; AES-GCM ciphertext bytes stay byte-identical. Endpoints + table scans defer to Phase 7a.2. Tracks noetl/ai-meta#61 Phase 7a + noetl/server#120.
deployment-specification: Phase 6e cross-region broker POST /api/internal/cross-region/resolve peer endpoint; BrokerRegistry config from NOETL_SECRET_BROKER_REGISTRY env (JSON); NOETL_SECRET_BROKER_TIMEOUT_SECS env (default 10); resolver fallback in get_sealed when residency-denied + broker configured; KeychainDef.no_broker_fallback per-credential opt-out; AppError::CrossRegionUnreachable → HTTP 502; two new metrics: - noetl_secret_broker_call_total{broker_region, outcome} - noetl_secret_broker_call_duration_seconds{broker_region} histogram Tracks noetl/ai-meta#61 Phase 6e + noetl/server#118.
deployment-specification: Phase 6d dynamic-secret primitives SecretValue.expires_at + secrets::dynamic::effective_cache_ttl helper honors issuer-reported TTL; KEYCHAIN_CACHE_DYNAMIC_SAFETY_MARGIN_SECS env (default 60); two new metrics: - noetl_secret_dynamic_ttl_seconds histogram (1m/5m/15m/1h/4h/12h) - noetl_secret_cache_skip_total{reason="already_expired"} counter Concrete cloud-specific dynamic providers (STS / AAD / iamcredentials) ride follow-up rounds (6d.1 / 6d.2 / 6d.3). Tracks noetl/ai-meta#61 Phase 6d + noetl/server#116.
deployment-specification: Phase 6c residency policy gate KeychainDef.residency (none/advisory/strict) + allowed_regions allowlist; ResidencyViolation HTTP 403 on strict mode mismatch with clear message that NEVER leaks the value; noetl_secret_residency_check_total{policy, decision} counter with the alert-worthy strict+violation_blocked combination. Tracks noetl/ai-meta#61 Phase 6c + noetl/server#114.
deployment-specification: Phase 6b ProviderRegistry caching + metrics NOETL_SECRET_PROVIDER_TTL_SECONDS env (operator escape hatch for short-lived STS / IMDS creds), the new ProviderRegistry caching section under Secret providers, and the two new metrics: - noetl_secret_provider_build_total{provider, region, status} (cache_hit / ok / error) - noetl_secret_resolve_duration_seconds{provider, region} histogram Tracks noetl/ai-meta#61 Phase 6b + noetl/server#112.
deployment-specification: Phase 6a region routing + NOETL_SERVER_REGION Adds the region-routing section under Secret providers — KeychainDef.region field, how each provider consumes it (AWS regional endpoint, Azure/Vault cluster routing, GCP resource id), fallback to NOETL_SERVER_REGION env, metric noetl_secret_resolve_total{provider,region,status}. New env: NOETL_SERVER_REGION (yes for residency-aware deployments). Tracks noetl/ai-meta#61 Phase 6a + noetl/server#110.
deployment-spec: sealed credential delivery endpoint (Phase 5b) GET /api/credentials/{identifier}/sealed?worker_id=<name> returns a SealedEnvelope JSON addressed to the worker; worker registers via the runtime JSON blob ({"worker_public_key":"<b64>"}). Documents wire format, error shape, and the noetl_credentials_sealed_total counter + credential.seal span. noetl/server#108 (Secrets Wallet Phase 5b, noetl/ai-meta#61).
deployment-spec: AWS SM + Azure Key Vault provider env AWS Secrets Manager: AWS_ACCESS_KEY_ID/SECRET/SESSION_TOKEN + AWS_REGION + NOETL_AWS_SM_ENDPOINT; ref shape [<region>:]<secret-id>[#<json-key>]; hand-rolled SigV4 (no aws-sdk). Azure Key Vault: AZURE_KEYVAULT_VAULT / _TOKEN + NOETL_AZURE_KEYVAULT_DNS_SUFFIX / _API_VERSION / NOETL_AZURE_IMDS_TOKEN_URL; ref shape [<vault>/]<name>[#<version>]; IMDS Managed Identity. Cloud-only backends (kind-val at unit-test layer). noetl/server#104 (Secrets Wallet Phase 3.x, noetl/ai-meta#61).
deployment-spec: TLS/mTLS listener env vars + probe caveat (Phase 4a) NOETL_TLS_CERT / NOETL_TLS_KEY (opt-in HTTPS) + NOETL_TLS_CLIENT_CA (mTLS) in the Main-app env table; a Transport-security subsection (mode matrix + ring provider + worker-channel rationale); and an mTLS probe caveat under Health probes (httpGet can't present a client cert -> use tcpSocket). noetl/server#103 (Secrets Wallet Phase 4a, noetl/ai-meta#61).
docs(deployment-spec): add Vault provider (VAULT_* env) to Secret providers VaultSecretProvider (KV v2, provider: vault) env catalogue + the note that production should prefer Vault K8s auth over a static VAULT_TOKEN. server#101, tracks noetl/ai-meta#61.
docs(deployment-spec): add Secret providers section (GCP + k8s env + RBAC) Documents NOETL_K8S_* (k8s secrets provider, server#96) and backfills the previously-undocumented GCP Secret Manager env vars. Notes the server SA needs secrets:[get,list] RBAC. Tracks noetl/ai-meta#61.