fix: tolerate matched docs with missing field payload (Redis 8.8 expiry race)

[vishal-bala]

Summary

Harden RedisVL result parsing so a matched document with a missing field payload is tolerated (skipped) instead of raising or surfacing a partial record. This is client-side future-proofing for a Redis 8.8+ server-side race and the durable fix behind the recent CI flakiness on redis:latest.

Background

Redis 8.8 changed a RediSearch default: the query worker pool now runs a nonzero number of background worker threads. With a background executor active, a document (or an indexed hash field) that expires via TTL/HPEXPIRE at the exact instant a background FT.SEARCH materializes it is returned as a matched id with a nil field array instead of being dropped server-side. redis-py collapses that into a Document carrying only {"id": ..., "payload": None} — no exception — so the malformed record lands in RedisVL, which previously raised KeyError in the vector-normalize branch or leaked a partial record into CacheHit/ChatMessage/RouteMatch. The prior mitigation (--search-workers 0 in CI) only hid this; real users on Redis 8.8 with TTLs are still exposed.

What changed

• Core parser (redisvl/index/index.py): process_results now skips a matched doc whose field payload is missing, detected only on zero-false-positive signals — a vector/range query missing its always-present vector_distance, or a JSON full-object unpack missing its json key. A plain hash FilterQuery/TextQuery doc is passed through as an id-only dict rather than dropped, so legitimate id-only projections and INDEXMISSING results are never over-skipped. process_aggregate_results and the hybrid/SQL paths drop entirely-empty rows.
• Extension guards: the semantic cache, message history, and semantic router independently tolerate an id-only/incomplete record (skipping it) for the paths the core rule intentionally doesn't cover.
• Completeness signal: SearchIndex.query() now returns a SearchResults object — a drop-in list subclass that additionally exposes dropped_count and complete, so callers that need completeness guarantees (audit, eval harnesses, agents) can detect and react to a race-shortened result set. Fully backward compatible: it behaves exactly like a list everywhere else.
• Observability: skips are logged at WARNING level, aggregated once per query.
• Docs: a new "Results and expiring documents" section in docs/concepts/queries.md explains the behavior, the detection boundaries, and how to use results.complete / dropped_count.

Behavior and compatibility

Backward compatible for every caller not hitting the race. Under the race, result counts become best-effort: query() may return fewer than num_results, paginate() fewer than page_size, and CountQuery.total can exceed the number of materialized documents (it reflects the server's match count). Silent skip is the default; no config knob is added.

Testing

Deterministic unit tests simulate the malformed response (a Result with a None fields slot / hand-built Document), covering every raise-site and skip branch, an over-skip guard proving legitimate id-only results survive, the SearchResults signal, and the extension guards. CI stays pinned at --search-workers 0; a live REDIS_SEARCH_WORKERS>0 reproduction is documented as a manual step. make check-types and make format are clean; unit and integration query tests pass.

Follow-ups (out of scope)

A process-level skip metric/callback and semantic-cache race-miss vs. real-miss decomposition, a nightly workers>0 integration test to guard the undocumented server shape, optional router detect-and-retry for the top-1 mis-route case, and raw=True message-history hardening.

---

Note

Medium Risk
Changes default query result handling and completeness semantics across all search paths; behavior is backward compatible for healthy results but can shorten result sets under TTL-heavy workloads on Redis 8.8+.

Overview
Hardens RedisVL against a Redis 8.8+ race where background FT.SEARCH can return a matched document id with an empty field payload when a key or field expires mid-query, instead of raising or leaking partial records.

Core parsing (process_results, aggregates, hybrid/SQL): skips matches only when the query type guarantees fields on a healthy hit (e.g. missing vector_distance on vector queries, missing json on JSON unpack); hash filter/text id-only matches are still passed through. Returns a SearchResults list subclass with dropped_count and complete for optional completeness checks.

Extensions (semantic cache, message history, semantic router): defense-in-depth skips invalid/incomplete rows so CacheHit / ChatMessage / RouteMatch construction does not fail.

CI/docs: fixes --search-workers 0 in docker-compose via real redis-server args (replacing a no-op REDIS_ARGS env var); documents behavior in docs/concepts/queries.md; adds unit/integration test coverage and stabilizes vector ordering assertions in flow tests.

^{Reviewed by Cursor Bugbot for commit 995a52c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6d065e1. Configure here.}

[nkanu17]

(Removed. Fixes for the points raised here are folded into #661, which now builds on this branch.)

[nkanu17]

Reviewed against a live Redis. Solid PR, the tricky part is handled well. I found three things and have already fixed them in #661, which is stacked on this branch, so nothing here needs action from you beyond a look at that PR.

Heads-up first: I rebased this branch onto main and force-pushed it (8dbf9fd -> 995a52c).