v0.19.0

[0.19.0] - 2026-09-27

Fixed

• The chaos lane reads labeled metrics by label and passes multi-shard
  commit tokens one per parameter (issue #534). The scripts/chaos/
  parser now scans a sample's label set quote-aware, reads the value before
  any timestamp, and takes label=value selectors, and the mid-flush trigger
  selects signal="metrics". A comma-joined x-ravel-commit-token header is
  split into its opaque tokens, and the header name is matched without the
  gawk-only IGNORECASE. Both scenarios now exit 3 on a setup error:
  scenario 1 used to exit 1 there, the same code as its strict-ack oracle
  failure, and scenario 2's exit 2 stays the release-blocking verdict.
  scripts/chaos/lib.test.sh covers the
  helpers on every pull request, and the nightly job annotates each
  scenario's exit code.

• OTLP and Remote Write ingest now take the in-flight permit and check the
  tenant credential before the request body is read or decoded (issue
  #1705).
  Every ingest handler took the --max-inflight-ingest-requests permit and
  resolved the tenant as its first two statements, but on both transports
  those statements ran too late to bound anything: on the HTTP surfaces the
  handler's body argument is an extractor, which axum runs before the handler
  itself, so the whole body was buffered (and a gzip body inflated) first,
  and on the gRPC surfaces the handler runs only after tonic has read and
  decoded the message. An unauthenticated caller could therefore make a
  process read and decode a full request body no matter how far over the
  ceiling it was. For OTLP metrics, logs and traces on both transports and
  for Remote Write, both decisions now happen on the request head, through
  middleware on the HTTP ingest routes and a tower layer on the gRPC
  listener, so a refused request costs the bytes of one request head. The
  OTAP stream is only partly covered: the gRPC layer checks its credential
  on the stream's head, before any frame is read, but its permit is still
  taken per batch after tonic has decoded that batch's protobuf frame, so an
  authenticated OTAP client over the ceiling still costs one decoded frame
  per shed batch. Because the permit is now taken before the body arrives,
  the ceiling counts concurrent uploads as well as concurrent decodes, and
  the body wait is bounded: a body that has not fully arrived 30 seconds
  after admission is refused with 503 and Retry-After on HTTP or
  DEADLINE_EXCEEDED for a unary gRPC export, and its permit is released, so
  a trickled upload holds its slot for at most 30 seconds. The OTAP stream holds no permit while it
  waits for a frame and is not bounded this way. The gRPC layer claims only
  the ingest services the listener registers, so a process without OTLP
  ingest (for example --mode query) still answers UNIMPLEMENTED on those
  paths. No HTTP/2 stream cap is derived from the ceiling, so the Flight SQL
  and fragment listeners keep their previous stream limits. The ceiling, its
  flag, and the ravel_ingest_concurrency_shed_total counter are unchanged,
  and a refusal keeps the status and message it had: 429 with Retry-After
  and 401 for bad credentials on HTTP, RESOURCE_EXHAUSTED and
  UNAUTHENTICATED ("invalid or missing tenant credentials") on gRPC.

• CI no longer pulls the AWS CLI image from ECR Public to create a test
  bucket (issue #2036). ECR Public caps anonymous pulls per source IP, and
  hitting that cap failed bench-smoke, quickstart and
  object-store-contract. Those jobs and the MetricsBench nightly now create
  their RustFS bucket with the runner's own aws CLI through
  scripts/ci-create-bucket.sh, which retries only the create call's own
  transient failures. The CI quickstart applies a CI-only compose override,
  deploy/docker-compose/ci-host-bucket.yml, that turns createbucket into a
  no-op on the locally built image; the documented quickstart command is
  unchanged. demo/kill-and-recover.sh now restarts ravel-server with
  --no-deps, so replacing the server no longer re-runs the stack's one-shot
  services.

• Retention no longer deletes logs or spans objects whose format version
  this build cannot read (issue #530). The physical sweep's version hold
  covered metrics only, so after a binary rollback across an RLOG or RSPAN
  format bump, a tombstoned logs or spans bucket holding objects written at
  the newer version was deleted once its protection horizon elapsed, even
  though the newer build could still read them. The sweep now reads each
  data object's trailer through the gate of the bucket's own format (one
  16-byte ranged GET per object, the cost metrics already paid) and, if any
  object is outside this build's reader window, deletes nothing in the
  bucket, logs a warning, and counts the objects on the same in-process
  held-object counter, exactly as it does for metrics. A corrupt trailer is still swept. ADR-0531 and ADR-0066
  carry dated amendments recording the wider hold.

• A PromQL alert rule raises one alert per matching series (issue #117,
  ADR-0117). A rule used to collapse its result vector into one alert with
  the rule's labels; each series that satisfies the condition is now its own
  alert, whose labels are the series labels without __name__ overlaid by
  the rule labels, and whose identity hashes those labels. The Alertmanager
  sink's alertname stays the rule id, or the rule's own alertname label
  when it sets one; a series label named alertname never replaces it and,
  when the rule sets no alertname, is sent as exported_alertname (with
  Prometheus' exported_exported_ rule on a further conflict) so series
  differing only in that label stay distinct Alertmanager alerts. A
  series that stops matching resolves only its own alert, and
  repeat_interval applies to each alert on its own. A rule matching more
  than 1000 series fails the tick with TooManyAlerts, and two series
  merging to one label set fail it with DuplicateAlertIdentity, whose
  message names the label names and the colliding alert_id but no label
  value; neither writes a record, and both count in
  ravel_alert_rules_failed_total. A scalar query and a
  SQL rule keep their existing single alert and identity. Two rules in one
  tenant may no longer share a rule_id, even with different labels: such a
  rules file now fails startup. Per-series rules over churning label sets
  should wait for alert state pruning (#1438). A new
  ravel_alert_undelivered_notifications gauge reports how many
  notifications wait for every sink to accept them; while a sink keeps
  failing it grows by one for every alert identity that transitions. A rule
  whose write fails partway through its alerts now counts the records
  already written in ravel_alert_records_written_total.

  Upgrade notes:

  • Expect a notification burst on the first tick. A PromQL rule whose
    matching series carry labels besides __name__ that the rule labels do
    not override gets a new identity for each series. If its single
    rule-level alert is pending or firing at upgrade, that tick writes one
    Resolved transition for it and one new transition per matching series.
    The webhook sink is notified of every one of them, and the Alertmanager
    sink of every one except a pending transition.
  • A rule that fires today can start failing every tick. It fails with
    DuplicateAlertIdentity when two matching series merge to one label
    set, for example a selector over several metric names
    ({__name__=~"a|b"}) whose series differ only in __name__, or a rule
    label that overrides the series label that told them apart. It fails
    with TooManyAlerts when more than 1000 series match. On every tick it
    fails, the rule writes no record, and its existing alerts keep their
    state.
  • A rules file in which two rules of one tenant share a rule_id now
    fails startup with rule id "<id>" is used by more than one rule in tenant "<tenant>". Give each rule its own rule_id before upgrading.
  • Repeat notifications multiply by the firing series. repeat_interval
    now applies to each alert, so a rule with 500 firing series sends 500
    repeat notifications per interval to every sink where it used to send
    one. Raise repeat_interval, or set it to 0s, on rules that match many
    series.

Added

• ravel-server builds a read and a write CPU gate and reports their
  queueing on /metrics (ADR-1702, issue #1702). A new crate,
  ravel-cpu-gate, holds the gate: a job at or above its inline floor
  (256 KiB, or 100,000 samples for a PromQL evaluation) waits for one of a
  fixed number of permits, then runs on the tokio blocking pool and keeps the
  permit until it returns; a smaller job runs on the calling thread.
  --cpu-gate-read-permits and --cpu-gate-write-permits size the two gates.
  Unset, they are max(1, cores - 1) and max(1, cores / 2), and 0 is
  refused at startup. /metrics renders ravel_cpu_gate_permits,
  ravel_cpu_gate_running, ravel_cpu_gate_queued, the
  ravel_cpu_gate_wait_seconds and ravel_cpu_gate_run_seconds sum and count
  pairs and ravel_cpu_gate_abandoned_total per gate, plus
  ravel_cpu_gate_jobs_total and ravel_cpu_gate_inline_total per gate and
  site, and the tokio runtime's ravel_runtime_workers,
  ravel_runtime_alive_tasks, ravel_runtime_global_queue_depth and
  ravel_runtime_worker_busy_seconds_total. No decode or encode path submits
  work to the gates yet; the later ADR-1702 tasks move them.

• ravel-cli export --signal logs writes a tenant's stored logs back out to
  a Parquet file ravel-cli load reads in (ADR-1751, issue #1712). The
  command takes the store and tenancy flags the other read commands take, plus
  --tenant, an RFC 3339 --start/--end window (Z or any numeric
  offset, converted to UTC), --parquet for the output path, and the same
  --mapping TOML a load uses to decide which column each field lands in. It
  resolves the catalog once, reads and decodes the RLOG objects that snapshot
  names, and writes the rows sorted by event time. The
  window is half-open, so exporting adjoining ranges writes no row twice and
  drops none between them. Exclusion follows the query path rather than a
  second copy of the rules: retention tombstones and compaction supersession
  are already applied by the catalog resolve, and erasure predicates from that
  same snapshot are handed to the segment fetcher and then matched against
  each decoded record's merged resource, scope and record attributes by the
  same function the SQL log scan calls, so a record a query cannot see,
  including one whose erased subject is only a resource or scope attribute, is
  a record the export does not write. The mapping's typed attribute
  columns round-trip, and the optional attrs_map_column adds one map column
  carrying every record attribute no typed column covers. --signal has no
  default and today accepts only logs; metrics and spans are refused
  with the bulk-import follow-up each one waits on, because an exported file
  no command can load back is not an export. --parquet is replaced only
  once the export finishes, by renaming a temporary file written beside it,
  so a failed export leaves an existing file untouched; a path under /dev
  or an existing directory is refused before anything is read.
  --max-ingest-lag passes the deployment's own ravel-server --max-ingest-lag through when it differs from the 2h default, and refuses
  zero as the server does. The whole window is decoded into memory
  before the first row is written, so a wide range wants several narrower
  exports. The ingest guide's bulk-export section covers that and the
  round-trip caveats, including ts_unit truncation and the fact that load
  does not read attrs_map_column back, and that no mapping key names a
  record's observed time, flags, instrumentation scope name and version, or
  scope attributes, so export drops them. Export costs ravel-cli a normal
  dependency on ravel-query, which is what sharing the query path's
  exclusion rules rather than copying them is worth (ADR-1751 amendment,
  2026-09-26).

• ravel-server --listen-health <addr> serves liveness and readiness from
  its own thread (ADR-1702). The listener runs on a dedicated
  single-threaded runtime and serves only /healthz, /readyz, /-/healthy
  and /-/ready, so a node whose main-runtime workers are all busy still
  answers its probes. A heartbeat task on the main runtime keeps it honest:
  /healthz there returns 503 once the heartbeat is older than 60 s, and
  /readyz once it is older than 30 s or any existing readiness condition
  fails. The flag is unset by default, so nothing binds, and the routes on
  --listen-http are unchanged. Startup refuses a --listen-health address
  equal to any other listener's, and shutdown waits at most 5 s for open
  health connections before dropping them, plus a 1 s join margin, so the
  worst case from SIGTERM to exit grows by up to 6 s (32.5 s to 38.5 s). A
  hand-written manifest that sets the flag needs a termination grace period
  to match.

• The operator can point both probes at a dedicated health port with
  spec.probes.dedicatedHealthPort (ADR-1702, issue #1702). Set to true,
  every gateway, query, and maintain container gains
  --listen-health 0.0.0.0:4316, a container port named health on 4316,
  and liveness and readiness probes on that port instead of 4318, with the
  same paths, period, timeout, and failure threshold as before. The health
  listener runs on its own thread, so a main runtime busy decoding segments
  can no longer let a probe time out and have the kubelet restart the pod.
  Those pods also get terminationGracePeriodSeconds 51 instead of 45, to
  cover the health listener's longer shutdown.
  The same routes stay on 4318 as well, so anything already probing the HTTP
  port is unaffected, and the ravel-ingest-router Deployment keeps its
  probes on 8080. The field defaults to false in this release and flips to
  true one release later; setting it needs a ravel-server image from this
  release or newer, since an older server rejects the unknown flag and
  restart-loops.

• spans.links decodes span links into a structured, filterable column
  (issue #1710). Symmetric to events, it is built from the plain
  attrs["_links_raw"] protobuf blob at scan time, on every RSPAN version,
  since RSPAN is a frozen persistent format and promoting links into a
  nested on-disk column would need an ADR and a version bump. NULL when a
  span carries no link, or when its _links_raw value is malformed (bad
  hex, bad framing, a link chunk missing a well-formed trace_id or
  span_id, a non-UTF-8 trace_state, or a trace_id, span_id or
  trace_state field that is not a length-delimited value), never an empty
  list or a fabricated field; one malformed link makes the span's whole
  links value NULL. Selecting events or links turns off the columnar
  fast path. On a single node, a query that selects neither never builds
  those columns; under distributed execution each worker still builds both
  for every row it returns and the coordinator drops them. A bare single-node
  SELECT count(*) FROM spans now returns the row count, with or without a
  pending erasure, instead of failing with "must either specify a row count or at least
  one column".

• /metrics now renders a per-shard ingest skew family (issue #1692).
  ravel_ingest_shard_messages_enqueued_total,
  ravel_ingest_shard_messages_processed_total,
  ravel_ingest_shard_queue_depth,
  ravel_ingest_shard_on_actor_seconds_total,
  ravel_ingest_shard_flush_permit_wait_seconds_total, and
  ravel_ingest_shard_off_actor_seconds_total render one series per
  configured shard, labelled mode, signal, and shard, with idle shards
  reading zero rather than being omitted. Zero-fill covers shards 0 to
  --shards - 1; a tenant resharded above --shards renders its extra
  shards once they record activity. The span pipeline now counts enqueued
  messages in span_router.rs and records on-actor and off-actor time in
  span_shard.rs, matching the metrics and log pipelines, so all six series
  carry live figures on all three signals.

Changed

• A transient object-store error on an audit-record PUT is retried before the
  batch fails closed (issue #2035). write_audit_batch used to fail an
  entire batch of queries on a single object-store timeout or throttle
  response, even though the object store's own client-side retry never covers
  a conditional PUT (PutOptions::create_if_absent(), the mode both the data
  object and the commit record use, is never marked idempotent by the S3
  client, so a Timeout on it skips the client's retry loop entirely). Each
  PUT now retries up to two more times with a short jittered backoff when the
  error is one already classified as transient, and only a non-retryable
  error or one that keeps failing across every attempt still fails the batch
  closed, exactly as before. A 30 s budget bounds each tenant group's
  writes in a flush, both PUTs and every retry, so a hung store still fails
  closed rather than stretching out for the full retry ladder. A flush
  writes its tenant groups one after another, so against a hung store a
  query can wait up to 30 s for each group ahead of its own. A throttle
  response's retry-after hint is honoured when it fits the budget, and a
  commit record found already stored on a retry counts as written when its
  bytes match, since the earlier attempt landed and only its acknowledgement
  was lost. Every retried attempt is now counted on /metrics as
  ravel_audit_put_retries_total.
• The default query request budget is now derived from the unsealed tail a
  healthy catalog carries plus the fold-stall alert window (ADR-1306). At 4
  shards and a 2 s flush cadence it gives 343,400 requests instead of 15,800,
  so a wide query is not refused for fold lag before the fold-stall alert
  pages. Each unsealed flush is budgeted at 8 requests, which covers flushes
  above the fetcher's 512 KiB whole-object threshold: one fetch of an L0
  segment now issues at most 4 page-range GETs, bridging the smallest gaps
  between the page runs a query selects when there are more, so it costs at
  most 7 GETs plus its commit-record GET. The bridged gap bytes are fetched,
  reserved against the process fetch memory budget and charged to
  max_bytes_scanned, so a selective query over large L0 flushes can read up
  to about the object size per segment, the same order as the existing
  whole-object fallback. The per-flush figure holds per selector: each
  selector fetches the segment again, so an N-selector query can spend up to
  1 + 7N requests per flush, which the derived budget does not scale for. An
  explicit --max-s3-requests is still used as given. The seal margin the
  span is built from is the one the server's own catalog folds and resolves
  with, read off the CatalogConfig its build_catalog constructs rather
  than from a compiled-in reference. No flag moves that margin today; when
  one does, the budget follows it with no hand recomputation. The
  shipped RavelCatalogFoldStalled alert is now held to the same span by a
  test: its threshold must be that seal margin and its for: the 600 s the
  derivation budgets, so raising either in
  deploy/prometheus/ravel.rules.yaml fails a gate rather than silently
  moving the first refusal ahead of the page. ravel-server logs the resolved
  budget once at startup with the span it is measured against
  (max_s3_requests, source, covered_span_secs, seal_margin_secs),
  derived or explicit alike, so an operator who pins --max-s3-requests can
  see which span their value undercuts. Scalar and histogram
  pages are now fetched in one batch, so a page_fetch tracing span can
  carry page_kind="mixed", where it used to be only scalar or
  histogram.
• The scheduled catalog fold now runs only in --mode maintain and
  --mode all, and a maintain fleet partitions it across its replicas
  (ADR-1693, issue #1693). A maintain process folds only the
  (tenant, signal) pairs it owns under the rendezvous hash and heartbeat
  live set that already distribute maintenance units, and it tests ownership
  before every per-tenant read, the pair's lifecycle t/<hash>/config record
  and its HEAD peek alike, so a non-owner issues no object-store request at
  all for a pair it does not own. The only request a tick makes that no owned
  pair accounts for is the single delimited listing of t/ that discovers the
  tenants. Scaling maintain to N replicas therefore divides the fold's
  request cost N ways instead of running N full copies of it, and a replica
  leaving the fleet hands its pairs to the survivors within 570 s at the
  defaults: the 180 s liveness window, plus the survivor's next 60 s heartbeat
  tick, which is when it re-lists and recomputes the live set, plus its next
  fold tick, 300 s with up to 10% jitter. That bounds the survivor's first
  tick over a pair, not always its re-fold: a pair the departed replica
  folded just before leaving keeps a HEAD younger than the 300 s fold
  interval, a survivor tick before then skips it as fresh, and the next tick
  is at most 330 s later, so the re-fold lands within 300 + 330 = 630 s of
  departure. The liveness window alone bounds when the departure becomes
  visible, not when the pairs are folded again.
  --mode all computes its live set as itself alone and keeps folding
  everything. gateway and query processes no longer fold on a timer. A
  query process keeps the on-demand POST /api/v1/admin/fold route, which
  is unchanged; a gateway process mounts no fold route at all, as before. A
  deployment running neither maintain nor all no longer folds on a timer
  at all, and must add a maintain process. --disable-fold and
  --fold-interval-secs are now refused at startup in --mode gateway and
  --mode query instead of being accepted and ignored, and the operator
  renders them on the maintain Deployment: spec.maintain.fold replaces
  spec.gateway.fold, which is now refused with a Degraded condition
  (reason GatewayFoldUnsupported) naming the field that replaces it.
  ravel_catalog_fold_last_success_timestamp_seconds now renders only in the
  two modes that fold on a schedule, so no process publishes a permanently
  stale gauge and RavelCatalogFoldStalled fires through its absent() arm
  on a fleet with no scheduled fold; ravel_catalog_fold_cycles_total and
  ravel_catalog_fold_failures_total render wherever a fold can run, the
  query mode's on-demand route included, so an on-demand fold's failures
  stay visible. The fold also reads the injected maintain clock rather than
  the system clock. Each signal's fold loop now runs under a supervisor: a
  panic in a tick body is caught, counted on the new
  ravel_catalog_fold_loop_restarts_total{signal} counter, logged at error
  level, and the loop is respawned after a bounded backoff doubling from 1 s
  to 60 s and resetting after a completed tick; the respawned loop ticks as
  soon as its backoff ends, so a loop panicking on every tick restarts 20
  times in its first 15 minutes and 15 times in every 15 minutes after at the
  defaults. The supervision is not
  optional under a partitioned fold: a replica whose loop dies keeps
  heartbeating, so it stays in the live set, keeps its pairs, and leaves them
  unfolded while its peers' fresh gauges hold RavelCatalogFoldStalled
  (max by (signal)) under its threshold. The new
  RavelCatalogFoldLoopCrashLooping rule in
  deploy/prometheus/ravel.rules.yaml fires on more than 5 restarts in 15m,
  unaggregated, because the condition is about one replica. It carries
  for: 15m: a loop panicking on every tick crosses the threshold 31 s after
  its first panic and pages about 15.5 minutes after it, while a single
  transient panic counts one restart. Supervision covers panics only: a tick
  that hangs on a store call that never returns moves no restart counter, and
  no shipped alert catches it on one replica of several, since only that
  replica's own ravel_catalog_fold_last_success_timestamp_seconds stops
  moving and RavelCatalogFoldStalled reads the fleet-wide maximum.
  On upgrade, a
  RavelCluster with spec.gateway.fold set now fails the render before any
  tier renders, so the whole cluster stops reconciling until the field moves
  to spec.maintain.fold; a hand-written manifest passing --disable-fold or
  --fold-interval-secs to a gateway or query container now fails at startup
  instead of ignoring the flag; and during a rolling upgrade an old-version
  maintain pod sits in the live set without folding, so the pairs the hash
  gives it stay unfolded until the rollout completes.
• A distributed-query worker resolves each pinned segment from that segment's
  own commit record instead of re-resolving its catalog (issue #1721). Every
  fragment request used to re-resolve the worker's catalog to map the
  coordinator's pins back to segments, so a compaction committed between the
  coordinator's resolve and the worker's fetch could invalidate the slice and
  cost a full re-resolve and re-dispatch. The worker now rebuilds the key of
  the pinned segment's own record from the identity the coordinator shipped
  with the ADR-0010 key builders: the commit record for an L0 segment, and the
  compaction record (or, when none exists, the erasure rewrite record) for a
  compacted L1 segment. It GETs that one record, checks that the record's own
  fields address the key it was read from, verifies the data-object key with
  verify_object_key (or the L1 object-key reconstruction), and compares the
  full 32-byte content hash, the object size, and for L1 the full input-set
  hash and part_index against the identity. The segment ref it reads is
  built from the verified record alone. A worker lists nothing and reads no
  manifest on the intra-cluster fetch path; it issues one record GET per
  pinned L0 segment, one per pinned L1 segment, and two for an L1 segment only
  a rewrite record describes. Each record GET holds a permit of the worker's
  process-wide GET limiter, the one its data-object GETs use. Record GETs are
  not charged to the slice's accounting, so they count toward neither
  max_bytes_scanned nor max_s3_requests on the worker or the coordinator,
  and a query that fits its budget locally also fits it distributed; they are
  also not in the query's reported cost, which the slice summary carries as
  one pooled figure. Off the budget is not unreported: a worker exports the
  totals as ravel_distrib_fragment_record_get_requests_total and
  ravel_distrib_fragment_record_get_bytes_total, process-wide counters
  beside the other ravel_distrib_fragment_* series, carrying only the
  mode label. A record that is missing, unreadable, fails verification,
  or disagrees with the identity fails the fragment with UNSUPPORTED, and the
  coordinator runs the query locally; a malformed identity is BAD_DATA. A
  throttled, timed-out or transient error on the record GET fails an inbound
  fragment with UNAVAILABLE, and the coordinator re-dispatches that slice to
  another worker. On a slice the coordinator runs itself there is no other
  worker to re-dispatch to and UNAVAILABLE would be terminal, so that same
  failure fails the slice SNAPSHOT_INVALIDATED instead and the coordinator
  re-resolves and retries once, keeping the recovery the catalog re-resolve
  had. That covers both local arms: a self-mapped or unroutable slice, which
  is dispatched straight to local execution with no remote attempt at all, and
  the fallback after a remote worker and its one re-dispatch both failed. Only
  the resolve phase moves; a fetch-phase UNAVAILABLE still means the segment
  reads themselves are failing and stays terminal.
  Records and the objects they name are immutable, so the pinned read is
  unaffected by whatever the catalog says by then. The queryfrag wire and
  PROTOCOL_VERSION are unchanged, and cross-cluster federation is unchanged:
  a resolve-scope request still resolves the remote cluster's own snapshot.
• A scrub tick's request count follows its budget, not the corpus size
  (issue #1686, ADR-1686). Each content-tier tick used to LIST a shard's whole
  commit prefix and GET every record in it before verifying its slice. It now
  lists strictly after a start-after marker held in the per-shard cursor and
  stops once the tick's budget is filled; the rotation rolls over when the
  listing runs out. The budget is a pair of caps, one on listing entries and
  one on requests, recomputed every tick from what the walk has actually
  observed: every rotation opens with one LIST-only count of the shard's
  entries, and every later tick recounts the last two ingest hours the previous
  count met, plus anything after them, and adds their growth, so a commit from
  any writer on the shard is counted. The per-tick entry cap is the share
  needed to reach the rotation's deadline, floored at the rotation's sustained
  rate and capped at four times that rate. The deadline is the smaller of
  --scrub-period and half the tenant's retention window, so a rotation that
  keeps pace reaches each object by about half its retained life. When the entries per
  tick the deadline needs exceed four times the sustained rate, the tick
  increments ravel_scrub_behind_total{signal} and logs the entries per tick
  needed beside the number allowed, with the deadline and the period. Every
  listing page the walk draws and every record GET attempt is charged against
  the request cap; the LIST-only count passes are not. A unit where a record or
  object GET failed with a retryable error (throttled, timeout, transient) does
  not move the marker past it and counts nothing from it, so the next tick
  retries the whole unit. That hold is capped at six consecutive held ticks on
  one unit, six hours of tick cadence at the default one-hour tick (at most 6.6
  with the jitter the loop adds to every sleep): the next tick that reaches the
  unit moves past it and counts each record or object that still fails
  retryably on
  ravel_scrub_unreadable_total{reason="retry_exhausted"}, and the new gauge
  ravel_scrub_marker_held_ticks{signal} reads the current held count of the
  signal's worst shard. A record or object GET that fails with any other
  error except not-found, and a record whose bytes do not decode, is counted
  once on the new ravel_scrub_unreadable_total{signal, level, reason} (with
  reason="access_denied" or reason="permanent", at the level of the object
  or record that failed), and the marker moves on, so one unreadable record
  cannot pin the rotation. Neither is counted on
  ravel_scrub_checksum_mismatch_total, which counts only bytes that were read
  and did not match; a new RavelScrubUnreadable warning alert fires on any
  increase of the unreadable counter over an hour. A compaction record that
  lands in an hour the marker has already passed is still judged against that
  whole hour, and a tick whose cursor GET fails for any reason other than NotFound
  is skipped with the stored cursor left alone. The cursor gains serde-default
  fields, so a cursor from an earlier release loads with defaults and its
  progress restarts: the next tick starts a fresh rotation. During a
  mixed-version rolling upgrade each version's cursor write drops the other's
  new fields, so the rotation restarts each time a shard's ownership flips
  between versions; nothing is corrupted and the cursor stays in object
  storage. A retention window whose half fits in one tick gives every tick the
  budget of a whole rotation, so every tick normally walks a full rotation and
  pays the LIST-only count of the whole commit prefix that opens it, whose cost
  is not charged against the budget.
  ravel_scrub_cursor_position is now a fraction of listing entries rather
  than of data objects. Every object the walk reaches and can read is still
  verified once per rotation, except that a retried unit's records and objects
  are fetched again,
  and the hour re-list that judges a late compaction record GETs that hour's
  compaction and rewrite records already consumed a second time. A record committed behind the marker waits for
  the next rotation.
• PromQL fetches now reserve against the same process-wide memory budget
  as SQL execution (issue #1255). ravel-server hands the PromQL engine
  and the startup cache warm pass the one MemoryBudget its SQL executor
  already uses, and with --distributed-query on it hands the same budget
  to the fragment service that runs metrics slices, both for remote workers
  and for the coordinator's own no-hop local path.
  An RSEG or RLOG fetch for a PromQL query that needs more than the budget's
  remainder fails with FetchMemoryExhausted (HTTP 503 on the PromQL API)
  instead of running unbounded, and the next query is admitted as before. A
  cache-warm fetch the budget refuses is skipped and logged rather than
  failing startup.
  The /metrics gauges now report real values:
  ravel_memory_reserved_bytes{component="fetch"} is the bytes held by live
  fetch reservations, component="sql" is the rest of the budget's reserved
  total rather than all of it, and ravel_memory_handoff_overlap_bytes is
  the part of the fetch share whose bytes went through the read cache, hit
  or miss, whether or not the cache kept them. The SQL path's own RSEG,
  RLOG and RSPAN fetchers still reserve against private unlimited budgets.
• ravel-server resolves cost-based logs fetching on every deployment
  again, including a --store s3 deployment against a loopback
  --s3-endpoint (ADR-2023 decision 1, issue #2023). 0.18.0's loopback
  byte-minimal default (ADR-2014) is withdrawn: under ten concurrent
  queries on the ClickBench reference machine it cut throughput to about a
  third of cost-based's (0.123 against 0.400 queries per second).
  --logs-fetch-policy byte-minimal remains available as
  an explicit opt-in; only the unset default changes. The startup line's
  policy_source now reads only flag or default; the
  derived-loopback-endpoint value is gone.
• ravel-server's catalog byte cache is sized independently of
  --cache-max-bytes (ADR-2023, issue #2023). --cache-max-bytes now
  bounds the query fetcher cache only; the catalog byte cache derives its
  own share of the process memory budget (CATALOG_CACHE_MEMORY_PERCENT,
  5%) regardless of --cache-max-bytes, or takes a new
  --catalog-cache-max-bytes <BYTES> flag explicitly. A deployment that
  relied on --cache-max-bytes to grow the catalog byte cache above its
  default now also needs --catalog-cache-max-bytes set explicitly to get
  the same catalog cache size. The reverse also changes: a deployment that
  set --cache-max-bytes low to constrain memory used to get a catalog
  byte cache that small too, and now gets the catalog's own 5% share (about
  1.5 GB on a 30 GiB host) unless --catalog-cache-max-bytes is also set.
  Startup still refuses to start when the two caches' resolved ceilings
  together reach or exceed the process memory budget, exempting
  --disable-cache as before. With --cache-dir set, each cache's disk tier
  is bounded by that cache's own RAM ceiling, so the catalog disk tier no
  longer follows --cache-max-bytes.
• ravel-server derives a larger fetcher-cache share on a loopback S3
  store (ADR-2023, issue #2023). Unset, --cache-max-bytes used to
  always derive 25% of the process memory budget; now, a --store s3
  deployment whose --s3-endpoint is loopback (the predicate that also
  gates a plaintext endpoint) derives 40% instead, so a fetch cache holding
  the working set's whole objects serves repeated statements without going
  back to the store's disk. An
  explicit --cache-max-bytes always wins, and every other deployment
  keeps the 25% share. The resolved value's source (budget-carve-loopback)
  is logged on the performance default resolved startup line alongside
  cache_max_bytes. With --cache-dir set, the fetcher cache's disk tier
  grows with it, on the same disk the store reads from.

What's Changed

• docs: add ADR-1702 and ADR-1306, amending ADR-0075 by @pmoust in #2013
• test(query): measure cold requests per unsealed flush by phase by @pmoust in #2019
• test: move cache watchdog helpers into ravel-test-support by @pmoust in #2020
• feat(metrics): render per-shard ingest skew for every signal by @pmoust in #2021
• feat(sql): decode span links into a structured spans.links column by @pmoust in #2026
• feat(server): charge PromQL fetches to the process memory budget by @pmoust in #2025
• feat(scrub): resume from a start-after marker with a per-tick budget by @pmoust in #2030
• feat(query): resolve pinned segments from their own records on workers by @pmoust in #2029
• feat(server): fetch whole objects on loopback again, split cache flags by @pmoust in #2024
• feat(server): run the catalog fold on the maintain tier, partitioned by @pmoust in #2028
• feat(cli): add export --signal logs by @pmoust in #2022
• feat(server): add the isolated health listener and runtime heartbeat by @pmoust in #2027
• feat(query): size the request budget for the unsealed tail by @pmoust in #2033
• fix(ci): create RustFS buckets with the runner's aws CLI by @pmoust in #2039
• fix(maintain): hold logs and spans buckets on an unreadable version by @pmoust in #2042
• docs(adr): accept ADR-2023 with its acceptance figures by @pmoust in #2045
• feat(operator): probe a dedicated health port behind a CRD field by @pmoust in #2041
• fix(server): admit and authenticate ingest before reading the body by @pmoust in #2034
• fix(maintain): retry a transient audit PUT before failing the batch closed by @pmoust in #2037
• fix(chaos): make the chaos lane's helpers and exit codes hold by @pmoust in #2038
• feat(cpu-gate): add the CPU gate crate and its server metrics by @pmoust in #2047
• Narrow the postings-absent assertion to POSTINGS fields by @pmoust in #1941
• feat(server): size the request budget from the running seal margin by @pmoust in #2058
• feat(alerting): raise one alert per matching series by @pmoust in #2018
• chore(release): prepare 0.19.0 by @pmoust in #2061

Full Changelog: v0.18.0...v0.19.0

Downloads

The attached binaries were extracted unmodified from the signed container
images this release published, so each one is byte-identical to the file inside
the image you can pull. They are glibc-dynamic, built against Debian 12
(the runtime image is gcr.io/distroless/cc-debian12), so they need glibc 2.36
or newer. On an older distribution, run the container image instead:

docker pull ghcr.io/nofireai/ravel-server:0.19.0

One binary per platform, named <binary>-<os>-<arch>:

• ravel-server-linux-amd64, ravel-server-linux-arm64
• ravel-cli-linux-amd64, ravel-cli-linux-arm64
• ravel-operator-linux-amd64, ravel-operator-linux-arm64
• ravel-ingest-router-linux-amd64, ravel-ingest-router-linux-arm64

Verifying the downloads

SHA256SUMS covers every attached binary and symbol archive. Check the files
you downloaded, then verify SHA256SUMS itself against the signature and
certificate beside it. The signature is keyless, bound to this release
workflow's identity, so there is no key to obtain:

sha256sum --check --ignore-missing SHA256SUMS

cosign verify-blob \
  --certificate SHA256SUMS.pem \
  --signature SHA256SUMS.sig \
  --certificate-identity 'https://github.com/NOFireAI/ravel/.github/workflows/publish-images.yml@refs/tags/v0.19.0' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  SHA256SUMS

Debug symbols

The shipped binaries are stripped of debug info but keep their symbol table,
so a backtrace resolves function names with nothing extra installed. Line
numbers come from the matching symbols archive. Each binary records the
basename of its symbol file, so unpacking the archive next to the binary is
enough and no symbol-file step is needed:

tar -xzf ravel-debug-symbols-linux-amd64.tar.gz
gdb ./ravel-server-linux-amd64

gdb looks for ravel-server.debug in the same directory (and in
.debug/), which is the name inside the archive. Do not rename the extracted
.debug files.