Bootstrap and validate Redis through detached shadow mode

[lan17]

Priority

P2 — Medium — production hardening for safe Redis rollout on sensitive tracked caches.

Problem

DialCache 0.12.0 can validate sampled tracked Redis hits, but Redis must already be serving and populated. With ramp.remote: 0, shadowing cannot observe or seed Redis, so a cold deployment never becomes useful for coherence validation.

The existing one-shot comparison can also report a mismatch after the originally observed Redis payload has already changed.

Confirmed design

shadowRamp is an independent, deterministic cohort for detached Redis work. It does not make a Redis value eligible to serve the caller.

served tracked hit                     valid remote policy, ramped down
        |                                           |
return decoded C0 immediately                start and return caller S
        |                                           |
        +--------------- detached shadow -----------+
                            |
                    obtain tracked Redis C0
                    (reuse served C0 when present)
                       /                  \
                 clean miss               hit
                    |                      |
          tracked fill from S        deserialize C0
                                      compare C0 vs S
                                      /           \
                                   match       candidate mismatch
                                                  |
                                           tracked Redis C1
                                           /              \
                                  missing/changed       raw C1 == C0
                                         |                  |
                                   superseded           mismatch

Caller isolation

• A served Redis hit returns immediately; all SoT, comparison, and confirmation work is detached.
• When Redis serving is ramped down, the caller starts and awaits its normal bounded SoT invocation exactly once.
• Detached shadow work may read, deserialize, compare, serialize, and write Redis, but the caller never awaits any of it.
• A shadow-only Redis value is never returned or published into request-local or process-local cache.
• Ramped-down shadowing does not create process single-flight or change caller coalescing.

Cold-miss bootstrap

A clean shadow-only C0 miss may populate Redis from the same SoT value accepted by the caller:

• use the already-resolved remote TTL snapshot;
• use the existing serializer and tracked watermark-aware write;
• emit filled, fill_blocked, or fill_error;
• never fill after Redis read failure, caller fallback failure/timeout, or shadow abandonment;
• never repair, refresh, or overwrite a non-null C0, including an undecodable payload.

No separate population flag or ramp is added. Enabling shadowRamp authorizes this narrow tracked clean-miss fill.

Coherence confirmation

For every sampled tracked hit:

1. Retain the original serialized payload C0.
2. Obtain the source value S.
3. Deserialize an isolated C0 snapshot and compare it semantically with S using the default or custom comparator.
4. Emit match when equal.
5. Otherwise reread tracked Redis directly as C1, bypassing local caches.
6. Emit superseded when C1 is missing or differs byte-for-byte from C0.
7. Emit mismatch only when raw C1 == C0.
8. Emit confirmation_error when the confirmation read fails or reaches its read deadline.

The extra Redis read runs only for semantic mismatch candidates. Shadow work never repairs, invalidates, evicts, or refreshes an existing value.

Metrics

No new metric instrument or label key is introduced.

• Caller-serving Redis work remains layer="remote".
• Detached Redis reads, serializer work, payload sizes, and read/write errors use the existing layer label with layer="remote_shadow".
• A ramped-down caller still emits disabled{layer="remote", reason="ramped_down"}.
• The existing shadow outcome metric emits exactly one terminal outcome: match, mismatch, superseded, filled, fill_blocked, fill_error, redis_error, source_error, deserialization_error, comparison_error, confirmation_error, timeout, or dropped.
• A clean C0 miss emits ordinary miss{layer="remote_shadow"} telemetry and terminates with its fill/source/timeout outcome.

Deadlines and capacity

• Reuse deterministic shadowRamp, exact-key deduplication, shadowMaxInFlight, and no-queue behavior.
• Use one monotonic shadow deadline and the effective remoteReadTimeoutMs for each Redis read. Served-hit timing starts when detached validation begins; ramped-down timing starts immediately before the caller's SoT invocation.
• Scheduler and detached deadline handles remain unreferenced.
• Started uncancellable Redis reads and dispatched writes retain their shadow slot until settlement.
• Expiry after serialization prevents a Redis write from being dispatched.
• A caller-owned raw loader may continue after the shadow deadline without retaining the shadow slot, including when fallbackTimeoutMs is null.

Compatibility and residual boundaries

• For valid policies, shadow-specific source calls, cache-path Redis traffic, returned values, and metrics are unchanged when shadowRamp is omitted or 0. The independent runtime-ramp and observer hardening is tracked by Reject out-of-range runtime ramps instead of activating cache layers #102 and Consume rejected promises from logger and metrics adapters #103.
• No Redis client method, protocol field, Lua script, Redis key, or dependency changes.
• Public unions widen: MetricLayer gains remote_shadow; ShadowValidationOutcome gains the new bounded outcomes. Exhaustive TypeScript consumers must add them.
• The C0 miss and later fill are not atomic. A concurrent writer can be overwritten by the normal tracked last-writer-wins fill.
• The tracked watermark remains the publication fence; futureBufferMs must cover the full SoT/serialization/queue/network/write interval.
• A dispatched write can complete after DialCache reports timeout or fill_error; outcomes are operational evidence, not a transaction result. filled and fill_blocked require the semantic client result before the whole-job deadline.
• Arguments, captured source state, and returned values must follow DialCache's existing immutability contract through detached comparison/serialization.

Acceptance criteria

• ramp.remote: 0 plus nonzero shadowRamp returns only the caller's single SoT result and runs Redis work detached.
• A clean ramped-down Redis miss performs at most one tracked fill.
• A non-null C0 is never repaired or overwritten.
• Candidate mismatches use conditional C1 confirmation.
• Detached work is isolated under layer="remote_shadow".
• Caller-serving remote latency and error metric semantics remain unchanged.
• Timeouts, capacity, source/serializer/comparator/Redis failures, and late settlement are covered.
• Redis and Valkey are covered through node-redis and GLIDE.
• For valid policies, shadow-specific cache results, cache-path Redis traffic, and metrics are unchanged when shadowing is off.

Implementation

#111