Expose orchestrator_lock_timeout in JsRuntimeOptions (Rust supports it, Node bindings do not)

[affandar]

Summary

RuntimeOptions.orchestrator_lock_timeout is configurable in the Rust crate but is not surfaced through JsRuntimeOptions. Node consumers are pinned to the 5s default with no override.

The asymmetry is the tell: JsRuntimeOptions already exposes workerLockTimeoutMs → worker_lock_timeout, but its orchestrator twin was never mapped. This looks like an oversight rather than a deliberate omission.

Current state

Rust crate (duroxide 0.1.29, src/runtime/mod.rs):

/// Lock timeout for orchestrator queue items.
/// When an orchestration message is dequeued, it's locked for this duration.
/// Default: 5 seconds
pub orchestrator_lock_timeout: Duration,

Default Duration::from_secs(5), consumed in src/runtime/dispatchers/orchestration.rs, and explicitly covered by tests/lock_timeout_tests.rs (which exercises Duration::from_secs(10), from_secs(1), from_secs(60), …). The crate's own doc example even shows orchestrator_lock_timeout: Duration::from_secs(10).

Node bindings (duroxide npm 0.1.27) — JsRuntimeOptions exposes:

orchestrationConcurrency, workerConcurrency, dispatcherPollIntervalMs, workerLockTimeoutMs, logFormat, logLevel, serviceName, serviceVersion, maxSessionsPerRuntime, sessionIdleTimeoutMs, workerNodeId, workerTagFilter, workerTagFilterTags

Not mapped: orchestrator_lock_timeout, orchestrator_lock_renewal_buffer, worker_lock_renewal_buffer, dispatcher_long_poll_timeout, max_attempts, session_lock_timeout, session_lock_renewal_buffer, session_cleanup_interval, activity_cancellation_grace_period, unregistered_backoff, supported_replay_versions.

The only DUROXIDE_* environment variable in the native binary is DUROXIDE_PG_POOL_MAX, so there is no env-var escape hatch either.

Motivation: a 5-day production livelock with no available mitigation

A PilotSwarm deployment (8 worker pods × orchestrationConcurrency: 4 = 32 orchestration dispatchers against a single Postgres-backed queue) entered a self-sustaining livelock:

1. Under contention, fetch_orchestration_item began taking ~6.4s — longer than the 5s orchestrator_lock_timeout.
2. The lock token was therefore already expired by the time the turn tried to ack: ack_orchestration_item: Invalid lock token.
3. Because the ack failed, retry bookkeeping never committed — including the poison-marking after max_attempts. The runtime logged Orchestration message exceeded max attempts, marking as poison on every single pass, but the marking could never land.
4. With no backoff committed either, the row's lock simply expired after 5s and the message became visible again. The effective retry interval was pinned at exactly the lock lease.
5. A permanently non-empty queue kept all 32 dispatchers in constant contention — which is precisely what made the fetch slow in step 1.

The loop is stable rather than self-correcting, because step 5 causes step 1.

Observed at the point of intervention:

Metric	Value
Max attempts on a single instance	78,987 (against max_attempts: 10)
Total accumulated attempts	781,446 across 12 instances
fetch_orchestration_item latency	~6.4s, very tightly clustered
Effective throughput	zero
Duration before manual intervention	5 days

Latency clustered extremely tightly (6.40, 6.39, 6.39, 6.26, 6.27, 6.45s), which is the signature of queueing/serialization rather than scan cost.

Note that lock renewal cannot help here. The documented strategy renews at 0.5 × timeout when orchestrator_lock_timeout < 15s (so 2.5s at the default), but the 6.4s is spent inside fetch_orchestration_item — before any item exists to renew.

Recovery required manually deleting the affected orchestration instances out-of-band. With orchestrator_lock_timeout exposed, raising it to 10s would have let the ack — and therefore the poison-marking — commit, allowing the runtime's own poison handling to drain the queue without operator intervention.

Ask

Map orchestrator_lock_timeout into JsRuntimeOptions as orchestratorLockTimeoutMs, exactly as workerLockTimeoutMs already maps to worker_lock_timeout.

Ideally also expose orchestratorLockRenewalBufferMs and maxAttempts, which are part of the same tuning surface.

Secondary observation: default divergence

Two values differ notably between the Rust defaults and what Node consumers run, and they compound the contention that triggers this:

Option	Rust default	Observed in Node consumers
dispatcher_min_poll_interval	100ms	10ms
orchestration_concurrency	2	4 (× N pods)

At 32 dispatchers polling every 10ms, that is up to ~3,200 fetch_orchestration_item calls/sec against one queue table.

It would be worth documenting the interaction explicitly: total dispatchers × poll frequency determines contention on the orchestrator queue, and orchestrator_lock_timeout must exceed worst-case fetch_orchestration_item latency under that load, or the system cannot make forward progress.

Related

The underlying reason a too-short lease is unrecoverable rather than merely slow — the poison-marking path depending on a lock token that has already expired — is filed separately against the Rust crate.

Environment

• duroxide (npm) 0.1.27
• duroxide (crate) 0.1.29
• Postgres-backed provider

[affandar]

Root-cause counterpart filed against the Rust crate: microsoft/duroxide#46 — why a too-short lease is unrecoverable rather than merely slow (the poison-marking path commits through an ack keyed on the already-expired lock token, so max_attempts can never be enforced).

[affandar]

Additional evidence posted on the crate-side issue that bears on the mitigation argument here: microsoft/duroxide#46 (comment) — microsoft/duroxide#46 (comment)

Short version: the measured retry interval across 12 trapped messages was flat at ~6s (mean 5.96s vs the 5s lease), independent of attempt_count (1,042 → 78,896) and of how long the message had been stuck (1.7h → 115h). So no part of the retry path commits, and raising orchestrator_lock_timeout is the only lever that breaks the cycle — which is exactly the lever Node consumers cannot reach today.

[affandar]

Refinement to my earlier comment — I said raising orchestrator_lock_timeout is "the only lever that breaks the cycle". That is now only partly true, and I want to correct it rather than leave an overclaim standing.

Reading the provider SQL (details: microsoft/duroxide#46 (comment)), the lease is computed from the caller-supplied p_now_ms, taken before a potentially multi-second blocking advisory-lock wait and never refreshed:

PERFORM pg_advisory_xact_lock(hashtext(v_instance_id));   -- may block for seconds
v_locked_until := p_now_ms + p_lock_timeout_ms;           -- clock never refreshed

So a larger timeout widens the margin but does not remove the defect — the advisory wait is unbounded, so a sufficiently deep convoy still produces a lease that expired before it was granted.

This does not weaken the case for exposing the knob; if anything it strengthens it, because Node consumers currently have no dial at all, not even the imperfect one. But the honest framing is: exposing orchestratorLockTimeoutMs buys operational headroom to survive an incident, while the actual fix is crate-side (refresh with clock_timestamp() after the wait, bound the retry loop, and stop blocking on the advisory lock).