nightly

docs(metastore): the startup probe checks both preconditions

The configuration and operating notes still described the check as "write a throwaway object twice
with `If-None-Match` and require the second write to be rejected". The probe also reads the
object's version, overwrites it with different content, and requires a write carrying the version
it read to be rejected, so an endpoint that enforces only the first precondition (localstack up to
4.0.2) now refuses to start -- and the notes would have told an operator that such an endpoint was
fine.

Review: walgit qw-conditional-write-probe-covers-if-match, oid d32ebb6c (needs-changes, B1).

v0.9.1-metastore1 — S3/R2 shared metastore

Superseded by v0.9.1-metastore2 (f4df32219): this tag predates the fix that makes the shared metastore's startup probe check both preconditions a commit uses (If-None-Match and a stale If-Match). Prefer the newer tag.

[Unreleased]

Added

• Azure Blob Storage: support custom endpoints via endpoint and endpoint_suffix configuration options for sovereign clouds (#6624)
• Storage: an S3 request now gives up after 5 minutes (storage.s3.read_timeout moves that bound, 0s
  waits forever as before). The AWS SDK sets no read timeout, so a connection that went away without a reset —
  a NAT or a proxy that forgets the flow — held the request forever: a metastore call that never returns hangs
  the publisher or the GC instead of failing so that the retry and the replay can run. Measured: a request with
  no timeout sat in mio::poll for ten minutes with its connection ESTABLISHED and no bytes moving, while the
  endpoint answered a fresh connection in 0.22 s. The bound covers the request up to its response headers,
  sending the body included — measured against the pinned SDK, a healthy 32 MiB upload that takes 22 s fails
  with a 2 s bound — which is why the default is generous: it keeps the default 256 MiB split inside the bound
  down to roughly 7 Mbps. Widen it, or set 0s, for a slower link. (walgit: qw-s3-request-timeout)
• Storage: conditional writes (put_if_absent, put_if_version_matches, get_all_with_version) with a
  PreconditionFailed error kind, and a Cloudflare R2 storage flavor (storage.s3.flavor: r2, alias
  cloudflare) that sets region: auto, path-style access and Content-MD5 checksums. Verified against the
  real R2 endpoint: it enforces If-None-Match/If-Match and accepts both the Content-MD5 and the default
  crc32c upload checksums. (walgit: qw-dist-metastore-s3-r2, qw-r2-flavor-and-storage-polish, qw-real-r2-verification)
  Note for implementors of Storage: the three methods have default implementations, so an existing
  implementation keeps compiling, but ObjectMetadata gains a public field and StorageErrorKind two
  variants, which breaks exhaustive matches and struct literals outside this repository.
• Metastore: a shared (S3-compatible) file-backed metastore now reports
  quickwit_metastore_file_backed_cas_conflicts_total (writes that lost a compare-and-swap race; the ones
  that ran out of the replay budget are counted again by the next counter) and ..._exhausted_total
  (mutations that failed after the replay budget ran out), so contention and the
  mutations that ran out of budget are visible from Prometheus. A third counter, ..._replay_tolerated_splits_total, counts the
  split state changes a replayed publish found already applied, so the tolerance granted by is_replay is
  visible instead of silent. (walgit: qw-metastore-cas-observability, qw-metastore-replay-visibility)
• Monitoring: the metastore Grafana dashboard now panels those counters — compare-and-swap conflicts, the
  mutations that failed after the replay budget, replayed publishes, manifest and shard folds with their failures, shard
  objects a reader skipped, and the manifest reads a listing pays for adoption — so a shared object-storage
  metastore's contention, and the mutations that ran out of the replay budget, are visible where an
  operator looks, not only in Prometheus.
  The same dashboard's three request panels and its instance variable had kept the pre-0.9 gRPC metric names,
  which no longer exist, so they showed nothing: they now query quickwit_grpc_requests_total and
  quickwit_grpc_request_duration_seconds with service="metastore", the names the upgrade notes give.
  (walgit: qw-metastore-dashboard-counters)
• Metastore: a sharded layout for indexes too large to rewrite on every publish.
  QW_METASTORE_SHARDED_LAYOUT=true makes a node create indexes whose split metadata lives in one slot file
  per slot (v2/splits/slots/00042.json), with a view that names, per slot, the segment and the slot version
  folded into it. A publish rewrites only the slots it touched instead of the whole split map, and a reader
  fetches the view and the slot files whose version changed; segments are folded per slot against the view's
  generation, and two generations are kept so a reader holding an older view still resolves. The layout is
  recorded in the objects, so a node reads an index whichever layout created it, and the slot count is fixed
  when the index is created. Measured with the same workload at 180 splits — 64 slots and a fold threshold of
  8, before a fold has to rewrite a large segment, not the 256/512 defaults — a publish rewrites 1.6 KB → 5.6 KB
  against 6.5 KB → 123.9 KB for the single-object layout. A mutation commits the slots it touched before
  its index metadata, which is why a failure between the two costs duplicated documents rather than lost ones
  (see the commit-order entry under Fixed). (walgit: qw-metastore-sharded-layout,
  qw-metastore-sharded-split-store, qw-sharded-partial-commit, qw-sharded-fold-on-s3)
• Metastore: a third layout for very large indexes, where the manifest holds references instead of the
  split map. QW_METASTORE_MANIFEST_LAYOUT=true makes a node create indexes with one manifest per stripe
  (v3/manifest-<stripe>.json), an immutable WAL object per published batch and one segment per time bucket,
  so a read fetches the query's time window rather than the whole index and a publish rewrites only the splits
  it changes. The layout is recorded in the objects, so a node reads an index whichever layout created it, and
  the stripe count is sized from the writer count (QW_METASTORE_MANIFEST_STRIPES, default 32). Measured on a
  real R2 bucket: a publish writes 1 380 bytes, and 12 writers reach 3.8 publishes/s with 0.03 conflicts per
  publish on 32 stripes (1.4/s and 0.68 on eight). Segments and WAL objects are collected per stripe against
  that stripe's own fold generation, and a publish replayed after a partial commit finishes; one revision at a
  time has to serve a prefix in this layout, as for the shared metastore as a whole.
  (walgit: qw-metastore-manifest-layout, qw-metastore-manifest-r2-cost, qw-metastore-manifest-fold-removal,
  qw-metastore-manifest-publish-replay, qw-metastore-manifest-gc-collection)

Changed

• An S3-compatible file-backed metastore can be shared by several nodes. Metadata writes reload the file
  together with its version and write it back with If-Match; a lost race is replayed within a bounded budget
  (16 attempts, the delay doubling from 10 ms to a 2 s cap) instead of overwriting the
  winner. The metastore probes the endpoint for conditional-write support at startup and refuses to run in
  shared mode when the endpoint would silently ignore the preconditions
  (QW_METASTORE_ALLOW_UNSAFE_STORAGE=true opts into single-writer mode on such an endpoint). file://,
  gs:// and azure:// metastores keep the single-writer behaviour, and the startup probe that decides
  between the two modes is covered by the same thread. (walgit: qw-dist-metastore-s3-r2,
  qw-metastore-distributed-mode)
  The escape hatch covers only what the probe can prove — an endpoint that accepts a conditional write it
  should have rejected. A storage that does not implement conditional writes, or a probe that could not run,
  still stops the node instead of silently starting in single-writer mode.
• Documentation: how to upgrade and roll back a cluster that shares an object-storage metastore (including the
  lost-update hazard of mixing versions), and what a metadata write costs in requests and latency.
  (walgit: qw-metastore-rollback-drill, qw-metastore-perf-bench)
• Documentation: for an index at the scale where the whole-index layouts stop fitting, the choice between
  the object-storage metastore and PostgreSQL is now written as a locality decision — object storage in the
  manifest layout when the nodes sit next to the bucket, PostgreSQL (or move the nodes) when they do not —
  with the round-trip arithmetic and the measured numbers behind it. (walgit: qw-metastore-backend-guidance)

Fixed

• Metastore: a publish that the pipeline replays because its first attempt committed and the response was
  lost now finishes. The replay carries the checkpoint delta the first attempt applied; that delta is not
  incompatible, it is done, so the replay skips it instead of failing on it. Skipping it is only sound for
  the publish that is the replay's own, and the checkpoint cannot say whose delta moved it, so a replay
  proves it first: the splits it publishes are published already (only the caller's earlier attempt can have
  published them — an empty set proves nothing). On the ingest-v2 shard API the request must also carry the
  shard's publish token, which says the caller still holds the shard; it is not ownership of the delta by
  itself, because a shard that changed hands has a new token. A fresh request that re-sends an applied delta,
  a competing writer whose delta lands on the same position, and a writer that took the shard over and
  replays an overlapping delta all keep getting the precondition failure.
  (walgit: qw-replay-tolerates-applied-delta)
• Metastore: the sharded layout now commits the slots it touches before root.json, the object that
  carries the checkpoint. A slot commit that fails therefore leaves the checkpoint where it was — the
  slots that committed before the failure stay published, a mutation touches several of them — and the
  metastore's own retry finishes the mutation; the other order used to leave the
  checkpoint ahead of a split that was never published, so a single transient slot failure stranded that
  split's documents — nothing re-read the window, because the checkpoint said it had been read. A fa...