Capacity-Weighted Shards, Credential-Safe Telemetry, Owner-Routed Caching

v1.46.0 is a hardening release. The embedded workflow engine moves to dwarf v0.10.5, which replaces the single templated DSN with per-shard declarations — each shard names its own database and its vCPU count, from which the engine sizes that shard's connection pool and its share of new-flow placement — and drops cross-replica messaging entirely, so a fleet now coordinates only through the database it shares. Flow state becomes a typed workflow.State rather than a bare map. Alongside that, a sweep through the request path closes a credentials-in-telemetry leak (headers and query strings no longer reach logs or span exporters, in any deployment), splits CORS origins by whether they are trusted with credentials, and refuses to start a PROD or LAB microservice that would carry secret configs over a plaintext transport. The distributed cache is rebuilt around rendezvous hashing so every operation is a unicast to the key's owner instead of a fan-out to every peer, endpoint inputs and structured configs gain declarative validation, and three separate paths stop pinning request payloads for the lifetime of a handler.

Highlights

• Per-shard database declarations. The foreman's SQLDataSourceName and NumShards are replaced by one Shards config: a list of {index, dsn, virtualCPUs, cordoned}. Each shard carries its own connection string — there is no %d template — and its vCPU count drives both its connection budget and how much new work it is given. (Breaking — see below.)
• Engine-derived worker and pool sizing. Workers and the renamed MaxOpenConns now default to "let the engine decide", derived from each shard's capacity and measured round-trip time. Both remain available as expert overrides.
• No cross-replica messaging. Replicas sharing a database discover pending work, flow outcomes and fleet membership by polling it, so the foreman's Signal endpoint and its peer multicast are gone with nothing replacing them. (Breaking — see below.)
• Typed flow state. FlowOutcome, FlowStep, flow.Snapshot() and the flow's baggage all traffic in workflow.State, which carries typed accessors (GetInt, Get, Parse) instead of requiring a type assertion per read. (Breaking — see below.)
• Credentials never reach telemetry. Spans record only structural request attributes; the header loop and query string are gone, in every deployment rather than only in production. Span.SetRequest is removed. (Breaking — see below.)
• CORS origins split by credential trust. AllowedOrigins becomes AllowedCredentialedOrigins and AllowedUncredentialedOrigins, making any-origin-with-credentials inexpressible rather than merely discouraged. (Breaking — see below.)
• Secret configs require a secure transport. In PROD and LAB a microservice that defines a secret config refuses to start over a plaintext NATS connection, rather than silently shipping secrets in the clear. (Operational prerequisite — see below.)
• Owner-routed distributed cache. dlru now assigns each key to exactly one replica by rendezvous (highest-random-weight) hashing, so a Load or Store is a single unicast rather than a broadcast plus an ack window.
• Validation at the trust boundary. Function, task and inbound-event inputs, and structured config values, are validated via dv8 tags after decoding; a malformed directive fails the microservice at startup rather than mid-request.
• Parallel takes a context. Each job receives a cancelable subcontext that is canceled when any sibling errors, so a job that observes its context can abandon work early. (Breaking — see below.)
• Request payloads are released, not held. The outbound body is dropped once sent, and generated marshalers release the inbound body the moment it is decoded. Measured over both transports with 64 concurrent 512KB requests parked in a handler, receiver-side retention falls from ~100% of the bodies in flight to ~1%.

Capacity-Weighted Shards

Sharding was previously one DSN with a %d placeholder plus a shard count, which assumed every shard was a numbered database on comparably-sized infrastructure. v1.46.0 replaces both configs with a single Shards list in which every shard is declared explicitly:

foreman.core:
  Shards: '[{"index":1,"dsn":"postgres://user:pass@db1:5432/flows","virtualCPUs":16},
            {"index":2,"dsn":"postgres://user:pass@db2:5432/flows","virtualCPUs":32}]'

index is encoded into every flow key created on the shard, so it must be unique, stable across restarts, and identical on every replica. dsn is used verbatim — no formatting, so a percent-encoded credential survives intact. virtualCPUs is the CPU count off the database instance's spec sheet, and it does two things: it sizes that shard's connection pool at the measured knee, and it weights how many new flows are placed there. A shard left at the assumed default of 2 vCPUs runs at a fraction of a large instance's capacity, so it is worth stating. cordoned excludes a shard from new-flow placement while everything already resident keeps executing, which is how a shard is retired without draining it by hand.

Because the engine now derives sizing from that declaration, Workers and MaxOpenConns (formerly SQLConnectionPool) default to deriving rather than to fixed numbers. Set them only when something the engine cannot see constrains the budget — a shared database, an external pooler, a deliberate global concurrency cap. Workers: 0 remains meaningful and distinct from "derive": it stands up a replica that creates, awaits and serves reads but never executes a task.

Changing the shard set is a coordinated restart of the whole fleet, not a live config edit: a flow created on a shard a peer does not know about is unroutable there.

The Flow State Model

Flow state is now a workflow.State value rather than a map[string]any. It holds each field as JSON and decodes only what is read, and it carries the accessors that every task previously hand-rolled:

// before
n, _ := outcome.State["count"].(float64)
total := int(n)

// after
total := outcome.State.GetInt("count")

Get unmarshals into a typed target, Parse unmarshals the whole state into a struct, and Len/IsZero replace len(m) and m != nil. The same type is what FlowStep.State, FlowStep.Changes, FlowStep.InterruptPayload, flow.Snapshot() and workflow.BaggageFrom(ctx) return.

Two state primitives change with it. flow.Delete is renamed flow.Del, and flow.Transform is removed — a rename across a contract boundary is now written as a snapshot, a clear, and explicit re-sets, which is the same operation without a bespoke primitive:

snap := flow.Snapshot()
var messages []llmapi.Item
snap.Get("messages", &messages)
flow.Clear()
flow.Set("conversation", messages)

Credentials Never Reach Telemetry

Spans and logs carried request headers and query strings, which routinely hold Authorization, Cookie and tokens — worst on error paths, where a forced trace exports exactly the spans that just had a credential attached. The resolution is to drop rather than redact, because a hardcoded sensitive-name list can never cover an application's own secrets:

• Spans record only structural attributes — method, scheme, host, port, path, body size. The header loop and the query string are gone, and the attributes are attached at span creation in every deployment, so a forced trace carries request context without ever carrying a credential. trc.Span.SetRequest is removed.
• The ingress logs the path, never the query string, and the SMTP ingress no longer attaches inbound mail headers to its span.
• Secret configs require a secure transport. Secret values travel from the configurator to each microservice over the bus, so on a plaintext connection they would cross the wire in the clear. In PROD and LAB a microservice that defines any secret config now refuses to start unless the transport is TLS — or is a short-circuit bundle with no wire at all.
• JWKS keys are evicted on rotation. A successful fetch replaces an issuer's key set rather than merging into it, so a key an operator has pulled stops being trusted at the next fetch instead of lingering for the life of the process.
• JWTs must be signed with an expected method, and the derived NATS subject of a request is capped at 1024 characters — rejected with 414 before it reaches the bus, with the ingress enforcing the same bound at the edge so an oversized URL never enters the mesh, its logs, or its spans.

Reverse-proxy handling is now explicit rather than implicit. Inbound X-Forwarded-* headers were trusted whenever present; they are now ignored and rewritten from the actual request unless TrustedProxyHops states how many proxies sit in front of the ingress. And AllowedOrigins splits into AllowedCredentialedOrigins — where the wildcard is rejected outright — and AllowedUncredentialedOrigins, so an origin's access to credentials is always a deliberate statement.

Owner-Routed Distributed Cache

The distributed cache assigns each key to exactly one replica, chosen by rendezvous (highest-random-weight) hashing over the current membership set, and routes every operation to that owner as a unicast. Previously an operation fanned out to every peer and waited out an ack window.

The design's load-bearing idea is that only the peer set has to be agreed on; the key-to-owner map never does — each peer computes ownership locally from the set it holds, so there is no coordinator, no election, and no version service. The generation is a hash of the sorted member IDs, so any two peers with the same view derive the same generation with zero coordination, and a membership change is observable as a generation change. During a change the two views briefly disagree, which the cache tolerates rather than prevents: a request is stamped with the caller's generation and the owner accepts it if it matches the current or, within an overlap window, the immediately previous one. Capacity still scales linearly with replica count, without paying a cluster-wide fan-out per operation.

Validation at the Trust Boundary

Endpoint inputs and structured config values are validated declaratively. The generated marshalers run dv8 over the In struct of every function, task and inbound event after decoding, so an invalid payload is rejected 400 before the handler runs. Tasks are included deliberately: a workflow's state is a cross-service contract, populated by callers, by LLMs and by tasks hosted elsewhere, so a violated contract should fail the flow at the offending step rather than propagate.

Structured configs get the same treatment through the new cfg.Validator option — a struct-valued config is validated on the way in and rejected before it is committed, with a failing fetched value falling back to the default. dv8.Compile runs at startup over every input type, so a malformed directive fails the microservice immediately rather than on the first request that exercises it.

Validation is opt-in: it only fires on api types that carry dv8 tags, so an existing project sees no behavior change until it adds them. The new add-type skill documents the anatomy of an api type — json, jsonschema_description and dv8 tags, plus the optional Validate method — and the directives are projected onto the OpenAPI document, so a constraint stated once reaches both the runtime and the generated schema.

Payload Retention

Three paths held request payloads far longer than they were needed, making a microservice's resident memory a function of how slow its own handlers, or its downstreams, happened to be.

• The outbound body is released once sent, not held until the response arrives. The duration of a remote call is not the caller's to control, so holding a body for it made in-flight memory a function of how slow the far end was: a hung downstream turned every queued request into a retained body for the whole time budget. Releasing at the ack window makes the retention a function of something the caller owns.
• The inbound body is released the moment it is decoded. The generated marshalers set r.Body = http.NoBody right after decoding into the In struct or the flow carrier, at which point the buffer is dead weight for the entire handler. The connector cannot do this itself — it has no way to know when a handler has finished with a body, or whether it intends to stream it — so only the generated marshaler is in a position to. Web handlers are raw and are left alone.
• The multicast response queue clears each consumed slot instead of only advancing its cursor, so responses a caller had already read no longer stay referenced until the whole iteration ends.

Fragment reassembly was overhauled alongside it: reassembly state is created synchronously before the ack, so a consecutive fragment cannot race ahead of the first, and each cache runs its sweeper lazily — started on the first admit, self-stopping when empty — because an always-on per-connector sweeper added enough scheduler contention across a large bundle to tip tight-timing tests into failure. The single-node LRU also reaps expired entries from the tail under the lock it already holds, closing the case where an unread expired entry lingered because the cache was under its weight limit and eviction never ran.

Other Changes

• Tickers no longer leak a goroutine per stop, and ticker goroutines are accounted in the shutdown drain so it waits for their exit.
• The ingress binds its listeners synchronously, so a port conflict fails startup instead of surfacing later.
• metrics.core accepts its secret key in an Authorization: Bearer header (the query argument is kept as a legacy fallback), compares it constant-time over SHA-256 digests, and replaces its goroutine-per-service scrape with a single multicast.
• The configurator no longer drops a config change when coalescing concurrent refreshes.
• Server histograms drop the route and canonical labels, both of which were derivable from the service, port and name labels that remain.
• Dependencies. dwarf v0.10.5, seamster v0.4.0, sequel v1.11.2, and a general refresh. The sequel connection-pool wait metrics became counters in the process, which renames them in PromQL.

Breaking Changes

• Parallel takes a context. Parallel(jobs ...func() error) becomes Parallel(ctx, jobs ...func(ctx context.Context) error), on the connector, the *Service base type and the service.Executor interface. Each job gets a cancelable subcontext canceled on the first sibling error; a job that must run to completion regardless names its parameter _ and captures the outer context.
• trc.Span.SetRequest is removed. The structural attributes it added are now attached to every span at creation. A caller that relied on its client-IP side effect calls span.SetClientIP(r.RemoteAddr).
• The ingress AllowedOrigins config is removed. It is replaced by AllowedCredentialedOrigins and AllowedUncredentialedOrigins; setting the old name to any non-empty value refuses startup rather than silently changing the CORS posture. Note that the old * reflected the caller's origin with credentials, whereas the new wildcard is uncredentialed by construction.
• The foreman's SQLDataSourceName and NumShards configs are removed, replaced by Shards. SQLConnectionPool is renamed MaxOpenConns and now defaults to deriving from the shard declaration, as does Workers.
• The foreman's Signal endpoint is removed, along with its SignalIn/SignalOut types, since replicas no longer message each other.
• Flow state is workflow.State, not map[string]any. FlowOutcome.State, FlowOutcome.InterruptPayload, FlowStep.State, FlowStep.Changes, FlowStep.InterruptPayload, flow.Snapshot(), flow.InterruptRequested(), flow.SubgraphRequested() and workflow.BaggageFrom() all change type. Indexing, len, range and comparison against nil no longer compile.
• flow.Delete is renamed flow.Del, flow.Transform and flow.SetState are removed, workflow.MergeState is removed, FlowSummary.Duration() and FlowStep.Duration() are removed, Graph.HasFanIn() is removed, and the flow and graph renderers' Render() returns a single string with no error.
• The Executor's input flow is a *workflow.RawFlow, so WithInputFlow takes one. Seeding a carrier's state is a raw-orchestration operation now.

Operational Prerequisites

Two changes are not code migrations and cannot be applied from a project checkout — plan them with the upgrade:

• PROD and LAB must connect to NATS over TLS. A microservice that defines a secret config refuses to start otherwise, with refusing to start with secret configs over an insecure transport. Several shipped core microservices declare secret configs of their own — metrics.core, bearertoken.core and foreman.core among them — so an app that runs any of them is affected even if it declares no secrets itself.
• Upgrading the workflow engine is a maintenance window, not a rolling deploy. The engine's schema migrations are forward-only and run at startup, so the first replica of the new version migrates the database every old replica is still using, and there is no downgrade path. Back up every shard, drain the fleet, start one replica and confirm it comes up clean, then start the rest. Flows survive it untouched: pending steps stay pending and interrupted flows stay parked.

Dashboards and alerting rules kept outside the project need two renames applied by hand: sequel_pool_wait_count becomes sequel_pool_waits_total, sequel_pool_wait_duration_seconds becomes sequel_pool_wait_duration_seconds_total, and any query grouping microbus_server_request_duration_seconds or microbus_server_response_body_bytes by route or canonical has to be re-aggregated on the labels that remain.

Migration

From inside a Microbus project, ask Claude Code to upgrade Microbus:

{{< prompt >}}
Get the latest version of Microbus.
{{< /prompt >}}

The upgrade-microbus skill handles the mechanical work: it rewrites Parallel call sites, splits the CORS config with you, removes SetRequest calls, renames flow.Delete, migrates the state model to workflow.State, rewrites the foreman's shard configuration with you, drops Signal call sites, and rewrites the renamed metric series in any dashboard checked into the repo. A project with no workflows skips the engine half automatically. The skill also raises the two operational prerequisites above — it cannot verify or change a deployment environment, so it reports them while you are mid-upgrade rather than leaving them to be discovered at startup.

Documentation

• Updated: foreman package reference — per-shard declarations, derived worker and pool sizing, and the removal of cross-replica signaling.
• Updated: dlru package reference — owner-routed operations under rendezvous hashing.
• Updated: httpingress package reference — the split CORS origin configs and TrustedProxyHops.
• Updated: metrics package reference — the Authorization: Bearer secret key and the multicast scrape.
• Updated: Distributed tracing and Logging — headers and query strings are never recorded.
• Updated: Metrics — the server histogram labels and the renamed connection-pool wait series.
• Updated: Configuration — structured config validation and the secure-transport requirement for secrets.
• Updated: JWKS pinning — key eviction on rotation and signing-method enforcement.