Routing table retains stale entries for short-lived shards

[ncoiffier-celonis]

Describe the bug

In my setup, I run a suite of unit tests, which does the following:

• create an index
• ingest some logs
• query some logs
• delete the index
• run more tests with the same setup

On the second test (and all following tests), I'm getting a failure with a 503 ingest service is unavailable (no shards available)

Steps to reproduce

Here is a small script to reproduce the problem:

reproduce-503.sh

#!/usr/bin/env bash
# Reproduces the 503 "no shards available" bug (commits 92a526b → e1732a7).
# Creates index → ingests → deletes → re-creates → ingests again (should 503 on buggy builds).
set -euo pipefail

IMAGE="${QUICKWIT_IMAGE:-quickwit/quickwit:v0.9.0-rc}"
CONTAINER="qw-repro-503-$$"
URL="http://localhost:7280"
INDEX="test-index-repro"

trap 'echo "Cleaning up..."; docker rm -f "$CONTAINER" >/dev/null 2>&1 || true' EXIT

NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
PAYLOAD='{"timestamp":"'"$NOW"'","message":"msg1"}
{"timestamp":"'"$NOW"'","message":"msg2"}
{"timestamp":"'"$NOW"'","message":"msg3"}'

INDEX_CONFIG='{"version":"0.8","index_id":"'"$INDEX"'","doc_mapping":{"field_mappings":[{"name":"timestamp","type":"datetime","input_formats":["iso8601"],"fast":true},{"name":"message","type":"text"}],"timestamp_field":"timestamp"}}'

# Start Quickwit
echo "Starting Quickwit ($IMAGE)..."
docker run -d --name "$CONTAINER" -p 7280:7280 -e QW_DISABLE_TELEMETRY=true RUST_LOG=debug "$IMAGE" run >/dev/null

echo "Waiting for readiness..."
for i in $(seq 1 60); do
    curl -sf "$URL/health/readyz" >/dev/null 2>&1 && break
    [ "$i" -eq 60 ] && { echo "FAIL: not ready after 60s"; exit 1; }
    sleep 1
done
echo "Ready."

# Round 1: create → ingest → delete
echo "--- Round 1 ---"
curl -sf -X POST "$URL/api/v1/indexes" -H "Content-Type: application/json" -d "$INDEX_CONFIG" >/dev/null
echo "Index created."

curl -sf -X POST "$URL/api/v1/$INDEX/ingest" -H "Content-Type: application/x-ndjson" -d "$PAYLOAD" >/dev/null
echo "Ingest OK."

curl -sf -X DELETE "$URL/api/v1/indexes/$INDEX" >/dev/null
echo "Index deleted."

# Wait
echo "Waiting 10s..."
sleep 10

# Round 2: re-create → ingest (expected to fail with bug)
echo "--- Round 2 ---"
curl -sf -X POST "$URL/api/v1/indexes" -H "Content-Type: application/json" -d "$INDEX_CONFIG" >/dev/null
echo "Index re-created."

# If we wait here, the bug doesn't reproduce (maybe because the shards have time to be marked as failed and removed from the cluster state?).
# sleep 10

echo "Ingesting (round 2)..."
curl -sv -X POST "$URL/api/v1/$INDEX/ingest" -H "Content-Type: application/x-ndjson" -d "$PAYLOAD"

Expected behavior
No 503 for short-lived shards.

Additional information

• If I add a sleep 10 between the index creation and the insert, then the problem disappears (see comment in the reproduction script).
• The problem was introduced between commits 92a526b and e1732a7.
• From my understanding, I believe it might be related to the recent routing change introduced by @nadav-govari
  with Merge feature node based routing #6203.
• I believe the problem could be an edge-case between 2 ticks of the BroadcastIngesterCapacityScoreTask background task.
• I'm not sure if such short-lived shard can happen in a "real" production setup (for example with really low commitTimeOut), or if it's just a unit-test only type of problem?
• here are some logs of the problem: logs-debug.txt.zip

Configuration:

quickwit version: 0.9.0 (aarch64-unknown-linux-gnu 2026-04-19T08:54:33Z e1732a7)

[guilload]

Thanks for the detailed report @ncoiffier-celonis. @nadav-govari will take a look on Wednesday.

[fmassot]

@nadav-govari I dug a bit into the issue and here is my understanding.

TLDR: When a user deletes an index and recreates it with the same name, the index ID stays the same but the index UUID changes (v1 → v2). The router caches the routing entry by index ID, so it can keep routing ingest requests to the old UUID (v1). This leads to 2 issues:

1. Multiple rejected requests for up to ~5 s as observed by @ncoiffier-celonis: the router routes to v1 and the ingester returns NoShardsAvailable since there are no more shards for that UUID.
2. Silent data loss if the ingester never receives the RetainShardsRequest (e.g. network issue): writes succeed into v1's WAL but the indexer can't publish them to a deleted index.

If I'm right, I'm open to doing a PR on this one.

Issue 1 — root cause

The router's routing table retains a stale entry keyed by (index_id, source_id) with seeded_from_cp = true, open_shard_count > 0. The router learns the index_id → index_uid mapping from the CP response or from the Chitchat broadcast and caches it. There is no direct notification to the router on index deletion — it has to wait for the next BroadcastIngesterCapacityScoreTask tick (5 s in production). Until then:

• seeded_from_cp stays true: set by merge_from_shards on the first CP response, only reset when a newer incarnation arrives — which deletion never triggers.
• open_shard_count stays positive: the CP does notify the ingester via a fire-and-forget RetainShardsRequest which deletes the shards, but the broadcast that zeros the router's entry only fires on the next tick.

has_open_nodes() therefore returns true, the router skips the CP query and routes to v1. The ingester has no shard for v1 and returns NoShardsAvailable. Recovery via the piggybacked RoutingUpdate fails because get_shard_snapshot() only iterates live shards — deleted shards are absent from both source_shard_updates and closed_shards, so the stale entry is never zeroed. All MAX_PERSIST_ATTEMPTS (5) route to the same dead node → 503.

The 10 s sleep workaround is enough time for the next broadcast tick to clear the entry.

Issue 2 — data loss

If RetainShardsRequest is lost (network issue), the ingester still has v1 open. The persist with index_uid=v1 succeeds — find_most_capacity_shard_mut matches on the full UUID and finds the open shard. Data is written to v1's WAL; the broadcast keeps advertising v1 as open so the router never re-queries the CP for v2. When the indexer tries to publish, v1 no longer exists in the metastore → data loss.

Even if we can say that we are ok with this data loss for this delete/create index scenario, we could improve a bit the situation:

• Synchronous deletion: the CP waits for all ingesters to confirm shard closure before returning from the delete request. Hard guarantee, small cost since deletion is not a hot path.
• Graceful IndexNotFound at publish time: when the indexer gets IndexNotFound on publish, we acknowledges and discards the WAL records cleanly rather than failing (I did not check what we are doing today).

We can open an issue on this one + add some public docs. My approach would be to synchronously wait for all ingesters to ack the deletion. If we have a timeout, we return an error to the index delete request with the list of ingesters who timeout. Up to the user to clean up the situation.

A note on the ambiguity in PersistFailureReason::NoShardsAvailable

The ingester currently returns NoShardsAvailable in two distinct situations:

1. find_most_capacity_shard_mut returns None — no open shard exists for the source.
2. The rate limiter rejects the write — a shard exists but can't accept the request right now.

I would distinguish the 2 cases.

Proposed fix 1 — clear the stale routing entry on NoShardsAvailable

In process_persist_results, when a persist failure of type NoShardsAvailable is received, call routing_table.apply_capacity_update(leader, index_uid, source, 0, 0) before applying the piggybacked routing update. On the next attempt, has_open_nodes() returns false triggering a CP query that returns v2 shards.

Proposed fix 2 — add PersistFailureReason::ShardRateLimited

Add PERSIST_FAILURE_REASON_SHARD_RATE_LIMITED = 8 to the proto and use it for the rate-limiter path. After this:

• NoShardsAvailable: no open shard for the (index_uid, source_id) pair.
• ShardRateLimited: a shard exists but the rate limiter rejected the write.

The router zeroes the routing entry only for NoShardsAvailable (fix 1), leaves ShardRateLimited alone, and the existing IngestFailureReason::ShardRateLimited mapping works cleanly.