Add sampled shadow validation for tracked Redis cache hits

[lan17]

Priority

P2 — Medium — opt-in correctness observability for sensitive tracked Redis caches.

Problem

A successful Redis hit is trusted after the configured serializer loads it. DialCache reports hits, misses, failures, and latency, but operators cannot sample successful hits and ask whether the value served from Redis still agrees with the source of truth (SoT).

For sensitive use cases, this leaves coherence confidence dependent on indirect signals or application-specific probes. The desired mode is observational: validate a controlled sample of successful tracked Redis hits without delaying the caller or changing cache state.

Current baseline at main@e06d833ba245706a499d66056fdad15dc1210b68:

• The semantic Redis client returns serializer payloads as string | Buffer:

  DialCache/src/redis-client.ts

  Lines 37 to 38 in e06d833

  	/** Serialized cache data, independent of any Redis client or wire framing. */
  	export type RedisCachePayload = string | Buffer;

• RedisCache loads that payload and previously discarded the encoded representation on a hit:

  DialCache/src/internal/redis-cache.ts

  Lines 90 to 128 in e06d833

  	async getWithResolvedConfig<T>(
  	key: DialCacheKey,
  	layerConfig: ResolvedLayerConfig,
  	readTimeoutMs = this.readTimeoutMs,
  	): Promise<CacheGetResult<T>> {
  	let payload: RedisCachePayload | null;
  	const abortController = new AbortController();
  	try {
  	const redisKey = this.redisKey(key);
  	payload = await withMonotonicDeadline({
  	timeoutMs: readTimeoutMs,
  	operation: () => this.client.read(
  	{
  	valueKey: redisKey,
  	...(key.trackForInvalidation ? { watermarkKey: this.redisWatermarkKeyFromKey(key) } : {}),
  	},
  	{ timeoutMs: readTimeoutMs, signal: abortController.signal },
  	),
  	onTimeout: () => abortController.abort(),
  	timeoutError: () => new RedisReadTimeoutError(key.useCase, readTimeoutMs),
  	});
  	} catch (error) {
  	this.recordError(key, error instanceof RedisReadTimeoutError ? "cache_read_timeout" : "cache_read");
  	throw error;
  	}
  	if (payload === null) {
  	return { status: "miss", config: layerConfig };
  	}
  	const start = performance.now();
  	try {
  	const value = (await this.serializerFor(key).load(payload)) as T;
  	return { status: "hit", value };
  	} catch {
  	this.recordError(key, "serialization_load");
  	return { status: "miss", config: layerConfig };
  	} finally {
  	this.recordMetric((metrics) => metrics.observeSerialization({ ...labelsFor(key, CacheLayer.REMOTE), operation: "load" }, elapsedSeconds(start)));
  	}

• The Redis-hit branch returns the cached value without invoking the SoT loader:

  DialCache/src/dialcache.ts

  Lines 523 to 536 in e06d833

  	private async finishRedisChain<T>(
  	redisCache: RedisCache,
  	key: DialCacheKey,
  	local: CacheGetResult<T>,
  	remote: RemoteCacheGetResult<T>,
  	fallback: () => Promise<T>,
  	resolvedRemoteConfig?: ResolvedLayerConfig,
  	): Promise<T> {
  	if (remote.status === "hit") {
  	if (local.status === "miss") {
  	await this.putLocalFailOpen(key, remote.value, local.config);
  	}
  	return remote.value;
  	}

• Existing shared-layer ramping assigns exact cache keys to deterministic cohorts:

  DialCache/src/internal/ramp.ts

  Lines 4 to 20 in e06d833

  	/**
  	* Assigns each cache key to a stable per-layer rollout bucket in [0, 100).
  	*
  	* Keep this algorithm stable: changing it reshuffles partial-ramp cohorts
  	* across every DialCache instance after an upgrade.
  	*/
  	export function deterministicRampSample(key: DialCacheKey, layer: CacheLayer): number {
  	return stablePercent(`${key.urn}:${layer}`);
  	}
  	function stablePercent(value: string): number {
  	let hash = 0x811c9dc5;
  	for (let index = 0; index < value.length; index += 1) {
  	hash ^= value.charCodeAt(index);
  	hash = Math.imul(hash, 0x01000193);
  	}
  	return ((hash >>> 0) / 0x1_0000_0000) * 100;

Confirmed design

Eligibility and rollout

• Shadow validation is opt-in through DialCacheKeyConfig.shadowRamp?: number on the existing per-use-case runtime configuration path.
• Omitted or 0 disables validation. 100 selects all otherwise eligible hits. Partial rollout uses a stable deterministic cohort of exact cache keys.
• Validation applies only when trackForInvalidation: true and only after an actual successful Redis hit.
• Request-local/process-local hits, Redis misses/errors/timeouts, initial serializer-load failures, disabled calls, untracked values, and ramped-out operations do not spawn shadow work.
• A metrics adapter must implement the optional shadowValidation hook; DialCache does not perform an unobservable SoT read.
• Coalesced followers share the Redis-hit leader and do not fan out validations.

Request-path isolation

• The caller receives the successfully loaded cached value normally.
• Eligibility checks and bounded slot reservation occur on the hit path, then an unreferenced setImmediate starts all extra work.
• The SoT read, detached cached-payload deserialization, comparison, and outcome metric are not awaited by the caller and cannot turn a successful hit into a rejection.
• Shadow execution runs the underlying loader with DialCache disabled so it cannot recursively satisfy itself from the same cache.
• Shadow work is best effort and is not guaranteed to finish during process shutdown.

Coherence definition

A match means application-level equality between an independent decoded snapshot of the Redis value and the raw current value returned by SoT.

1. Retain the semantic string | Buffer payload from the successful Redis read internally.
2. Return the already-decoded cached value to the caller.
3. In detached work, read the current value from SoT.
4. Call load(retainedPayload) again on the same effective serializer to create an independent cached snapshot.
5. Compare that cached value with the raw SoT value.

The default comparator is Node's util.isDeepStrictEqual. This avoids representation-only mismatches such as plain-object property insertion order while retaining strict value, array-order, prototype, constructor, Buffer, Map, and Set semantics.

cached() and getOrLoad() accept an optional typed per-operation comparator:

export type ShadowComparator<T> = (
  cachedValue: T,
  sourceValue: T,
) => boolean;

shadowComparator is use-case equality policy, not serializer behavior or runtime rollout configuration. It must synchronously return a boolean and must be deterministic, side-effect-free, non-mutating, and bounded. A throw or non-boolean return reports comparison_error, not mismatch. If untyped JavaScript accidentally returns a Promise, DialCache consumes its settlement without treating it as a valid result or allowing it to escape the concurrency cap.

The comparator never receives the raw Redis payload or the object already returned to the caller. Re-loading the retained payload prevents caller mutation after the hit from contaminating validation. The raw SoT value is intentionally not dump/load normalized: lossy serialization remains observable as a mismatch unless the use case supplies a comparator that defines the normalization as equivalent.

Capacity and liveness

• DialCacheConfig.shadowMaxInFlight?: number is a positive safe integer and defaults to 1 per DialCache instance.
• There is no queue. Exact-key duplicates and work above the per-instance cap emit dropped.
• A single monotonic deadline covers the SoT read, detached serializer.load, and comparison.
• A finite fallbackTimeoutMs is reused as the shadow budget. With fallbackTimeoutMs: null, normal fallbacks are unbounded but shadow work uses the internal 60-second default.
• Timeout releases DialCache's retained payload reference and prevents later phases from starting. JavaScript cannot cancel already-running source/serializer/comparator work, so its slot remains occupied until that work settles.
• Scheduler and deadline handles are unreferenced. A synchronous source, serializer, or comparator can still occupy the Node event loop after the caller has returned and must therefore remain bounded.

Observability and side effects

Bounded outcomes:

• match
• mismatch
• source_error
• deserialization_error
• comparison_error
• timeout
• dropped

Metrics may include the existing namespace, use case, key type, and outcome, but never cache IDs, cached/SoT values, serialized payloads, Redis keys, or raw exception messages.

Validation is observational only. No outcome writes Redis, refreshes TTL, advances a watermark, invalidates, repairs, evicts local state, or changes the value returned to the caller.

Serializer and client contracts

• The effective serializer's load() may run twice for a sampled hit and must be repeatable, non-mutating, and return independently usable values.
• Serializer.load() receives borrowed immutable input and must not mutate a Buffer.
• A custom DialCacheRedisClient must return an operation-owned payload whose contents remain stable after read() settles; DialCache does not add a payload-size-linear defensive copy on the request path.
• The wrapped function or inline loader must be safe to invoke as an additional side-effect-free SoT read.
• Detached cached() arguments and getOrLoad() captures must remain immutable or be snapshotted so the later SoT read still corresponds to the exact key built on the request path.

Architecture boundary

• Keep DialCacheRedisClient, Redis Lua, frame format, value keys, watermarks, and bundled adapter protocol unchanged.
• Carry the semantic payload only through an internal remote-hit result alongside the decoded value.
• Keep serializer selection and detached deserialization behind the existing Redis/serializer boundary; do not expose payloads publicly.
• Model validation as a post-successful-Redis-hit observational branch, not another cache layer.
• Preserve request/process single-flight behavior.

Acceptance criteria

• Shadow absent or 0 leaves behavior unchanged and performs no SoT read, extra deserialization, or comparison.
• A selected tracked Redis hit returns before shadow work starts.
• Stable exact-key cohorts cover partial rollout, boundaries, sparse runtime overlays, and malformed configuration.
• Untracked hits and every non-Redis-hit path skip shadow work.
• Coalesced followers produce at most one validation for the Redis-hit leader.
• The default comparison treats equivalent JSON objects with different property insertion order as a match.
• A typed custom comparator works for both cached() and getOrLoad().
• Caller mutation of the returned cached object cannot affect the detached comparison.
• SoT failure, second-load failure, comparator throw/non-boolean/accidental Promise, telemetry failure, and timeout cannot affect the original request or escape as unhandled rejections.
• In-flight work is bounded per instance; excess and exact-key duplicate work does not queue.
• Timed-out underlying work retains its slot until settlement while DialCache releases the retained Redis payload.
• Metrics are bounded and exclude sensitive values and identifiers.
• Shadow validation never writes, repairs, invalidates, refreshes TTL, or mutates local cache state.
• Existing custom Redis clients and metrics adapters remain source-compatible when shadow validation is disabled.
• Unit, package-consumer, Redis/Valkey integration, and normal-path benchmark coverage is included.

Out of scope

• Local or request-local hit validation.
• Automatic repair, refresh, invalidation, or cache mutation based on comparison results.
• Redis protocol/frame/key changes or another Redis round trip.
• Worker threads or a guaranteed background-job delivery system.
• Guaranteed completion during process shutdown.
• Shadowing untracked cache use cases.
• Async/I/O-performing comparators.

Implementation

Draft PR: #105

Related issues

• Make tracked local caching consistent with the invalidation contract #28 — targeted invalidation is intentionally shared-Redis-only.
• Add a repeatable performance and scale benchmark suite #35 — comparable performance and scale measurement.
• Bound metrics error labels with a backend-neutral taxonomy #45 — bounded backend-neutral metrics error taxonomy.
• Reject out-of-range runtime ramps instead of activating cache layers #102 — strict runtime ramp validation.
• Consume rejected promises from logger and metrics adapters #103 — rejected logger/metrics promise isolation.