Stellar Index v0.16.3

[v0.16.3] — 2026-07-15

Operations

• Repo migrated to the Stellar-Index GitHub org (from the billing-locked
  StellarIndex org). Go module path is now
  github.com/Stellar-Index/StellarIndex — a pure rename across all imports,
  CI/config refs, monitoring runbook_urls, and docs; build/vet/mod-verify green.
• run-heavy-job.sh gained a root-disk watchdog (archival-node ansible role,
  applied to r1): a background loop stops the job's systemd scope when root free
  space drops below 2 GiB, closing the 2026-06-11 incident follow-up — a heavy job
  flooding logs or scratch space could fill the 49 GB root and wedge service
  logging channels. The remaining durable fix (move /var/log/swap onto ZFS or
  resize root) is recorded as accepted operator-gated debt in the post-mortem.
  The watchdog subshell runs with errexit/pipefail disabled and skips a
  non-numeric df reading (2026-07-13 review finding): a single transient df
  failure must not silently kill the watchdog and fail the guard open.
• stellar.account_movements gained a balance_id skip index (idx_cb_balance_id,
  bloom_filter(0.01) on JSONExtractString(attributes, 'balance_id'), GRANULARITY 4) —
  applied directly to r1's production ClickHouse via ALTER TABLE ... ADD INDEX
  (2026-07-12; mutation complete). Now codified in both DDL sites — deploy/clickhouse/ tier1_schema.sql and internal/storage/clickhouse/account_movements.go's
  EnsureAccountMovementsTable — so a fresh install gets it from the start; note that
  CREATE TABLE IF NOT EXISTS does not retrofit it onto an already-existing table
  (irrelevant for r1, already applied there directly).

Changed

• stellarindex_dex_nonstandard_decimals_detected downgraded from severity: ticket to
  severity: informational. Per its runbook, every serving path has auto-normalized
  non-7-decimal Soroban tokens via the nonstandard_decimals_assets correction table
  since 2026-07-10, so a detected token is now an expected, handled condition — the
  latching detector was sitting in the firing list forever as noise, not an action item.
  Added a new severity: ticket alert, stellarindex_nonstandard_decimals_correction_failing
  (deploy/monitoring/rules/aggregator.yml + configs/prometheus/rules.r1/aggregator.yml),
  for the genuine failure mode: the correction sweep erroring
  (stellarindex_nonstandard_decimals_cache_refresh_failures_total) or a serving path
  actively declining (stellarindex_price_serve_declined_nonstandard_decimals_total) —
  either means newly-detected tokens don't get corrected and silently revert to serving
  the skewed raw ratio. Runbook docs/operations/runbooks/dex-nonstandard-decimals.md
  extended to cover both alerts.

Fixed

• classic_movements: sponsored account creations (CAP-33, Protocol 15+) are no longer
  dropped as malformed. CreateAccount with startingBalance = 0 is legal once a
  sponsor covers the reserve; the decoder rejected any non-positive balance, silently
  skipping every sponsored creation — caught live on the 2026-07-12 archive backfill at
  ledger ~37.12M (a sponsorship-bot storm produced a decode-error flood). Zero now emits
  a real zero-amount create_account movement; only NEGATIVE balances are malformed.
  The already-derived ranges need one fixed-binary re-pass without -resume (idempotent,
  ReplacingMergeTree) to pick up the dropped creations.
• classic-movements-backfill: per-window deadline with one retry, closing the
  2026-07-12 half-dead-connection stall. A half-dead ClickHouse native connection left
  the backfill loop blocked in a network read for ~2h at zero CPU; the driver's
  ReadTimeout alone did not unwedge it. Every ClickHouse call inside a window (both
  StreamClassicOps passes, both StreamEntryChanges calls, FindClaimableBalanceCreates,
  InsertAccountMovements, VerifyAccountMovementsWindow) now runs under a single
  20-minute per-window context.WithTimeout; a window that exceeds it is retried exactly
  once on fresh connections before being treated as a real error. The window body was
  extracted into classicMovementsAttemptWindow, which touches only window-local state
  (a new windowResult), so a failed attempt's partial batch/counts are discarded rather
  than merged into the run — the retry starts clean, including a discard-and-redecode of
  the shared decoder's pending claimable-balance refs so stale state from the failed
  attempt can't leak into the retry.
• classicmovements.Decoder's in-run claimable-balance-create index is now bounded.
  The index had no eviction and could grow across a multi-million-ledger run toward the
  full historical CreateClaimableBalance row count (research §5: ~1.5B), which
  contributed to an earlier OOM. It's now capped at 2,000,000 entries with FIFO eviction;
  a miss on an evicted balance_id falls through to classic-movements-backfill's existing
  ClickHouse lookup (FindClaimableBalanceCreates) the same way an out-of-range create
  already does, so eviction never produces a wrong or guessed amount.
• classic-movements-backfill's claimable-balance fallback lookups are now batched,
  one query per window instead of one query per ref. The claimable-balance-bot era
  (ledgers ~34M-40M) surfaces thousands of pending claim/clawback refs per window; each
  serial FindClaimableBalanceCreate call was, before the idx_cb_balance_id skip index
  above existed, a 6.5s full scan of stellar.account_movements' 973M rows — with
  thousands of refs per window, the drain was crawling. The index alone brought a single
  lookup to ~84ms (~77x); the retired per-ref function is replaced by
  clickhouse.FindClaimableBalanceCreates, which resolves an entire window's misses (after
  the free in-memory index pass) in ONE IN (?) query. A batch-query error degrades the
  whole miss-set to unresolved with one stderr line, matching the previous per-ref
  behavior of counting a failed lookup as unresolved rather than failing the window.
• FindClaimableBalanceCreates now passes lookup ids as a ClickHouse EXTERNAL TABLE
  instead of any inlined IN list — the terminal fix after two same-day regressions
  (2026-07-12/13). The failure chain: (1) a single unchunked IN (?) query was the
  original batched-lookup shape; the claimable-balance spam era around ledger 49.3M
  produces windows with well over a million pending refs, and the clickhouse-go driver
  inlines a bound []string directly into the SQL text rather than sending it
  server-side — real production failures on 2026-07-13 (58,714, 853,775, 1,292,177, and
  1,393,786 ids in a single call) all blew past ClickHouse's max_query_size (256 KiB
  default, code: 62) past ~3,400 ids, failing the whole lookup and leaving every claim
  in that window's batch unresolved. (2) The first fix chunked the inlined IN list at
  2,000 ids per query — which shipped, then broke production an hour later: with 2,000
  ids inlined per chunk, idx_cb_balance_id's bloom_filter(0.01) skip index stops
  helping and starts hurting, because a granule's false-positive probability compounds
  with probe count — across the table's ~119k granules, 1-(1-0.01)^2000 ≈ 1, so
  effectively every granule looks like a possible match and each chunk degenerates into a
  near-full parallel scan of the wide attributes column over the 973M-row table,
  blowing the connection's max_memory_usage (code: 241, Query memory limit exceeded, would use 10.00 GiB). Single-id point lookups were never affected by either failure
  (~84ms, confirmed live) — only the batch path. (3) The terminal fix: ids are now sent
  as a server-side external table (clickhouse.WithExternalTable, native-protocol side
  channel, not SQL text) and matched via JSONExtractString(attributes, 'balance_id') IN cb_ids — a hash-set semijoin whose cost is O(ids), immune to both the SQL-text-size
  ceiling and the bloom-filter false-positive-rate blowup, bounded by a SQL-text
  SETTINGS clause (use_skip_indexes=0, max_threads=4, max_memory_usage=8 GB) —
  SQL-text because per-query WithSettings was observed not reaching the server next to
  WithExternalTable, and use_skip_indexes=0 because evaluating the bloom index against
  a large IN-set matches every granule and itself blew the 10 GiB query-memory ceiling. The external table itself is still chunked
  (cbLookupExtTableChunkSize, 1,000,000 ids) as a footprint safety bound, not to dodge a
  ceiling — real windows up to 1.4M ids now resolve in one or two queries.
  idx_cb_balance_id is unaffected and still serves true point lookups (a literal = ?
  or a small hand-written IN (?)) — the indexed WHERE expression
  (JSONExtractString(attributes, 'balance_id')) must stay textually exact for either
  access pattern, or ClickHouse silently falls back to a full scan.
• classic_movements: the claimable-balance index cap was raised 2M → 8M entries. The
  2021 claimable-balance spam era creates 2–3 million balances per 10k-ledger window, so
  the 2M FIFO cap evicted a window's own creates before its resolution pass ran, zeroing
  in-memory resolution and dumping 600k+ refs per window onto the ClickHouse fallback
  (each such scan ~2.5 min over 695M cb-create rows). 8M (~3 GB) restores ~3 spam windows
  of locality; spam claims land seconds after their create, so the fallback is rare again.
• stellarindex-ops bounded-backfill walkers (ch-backfill, wasm-history,
  verify-archive) now use a small, explicit, parallelism-scaled read-ahead buffer instead
  of the SDK's large default — the fix for the 2026-07-15 -parallel OOM.
  opsutil.NewBoundedLedgerStreamConfig left ledgerstream.Config.Buffered nil, so every
  bounded walk fell through to ingest.DefaultBufferedStorageBackendConfig(1)
  (BufferSize=10000, NumWorkers=10 — the SDK's "small files" branch, since Galexie's
  1-ledger-per-file schema was never threaded into DataStore.Schema). Each
  ledgerstream.Stream call builds its own BufferedStorageBackend with an independent
  prefetch queue, so N concurrent walkers (each subcommand's -parallel/-workers splits
  a bounded range into N chunks, one goroutine per chunk) multiplied that queue depth by N.
  On r1, ch-backfill -parallel 2 and -parallel 4 both OOM-killed the 20G
  run-heavy-job.sh cap within ~1000 ledgers; -parallel 1 was stable at ~12GB on the
  same 10000-deep default — the single walker is IO-latency-bound (serial MinIO fetches;
  CPU idle), so parallelism is the right throughput lever, it just needed a bounded buffer
  to use it safely. NewBoundedLedgerStreamConfig gained a parallel int parameter and
  now returns an explicit Buffered override sized 200/parallel ledgers (floored at 32),
  fixed NumWorkers=4 — so per-walker read-ahead shrinks as -parallel grows and total
  buffer memory across all walkers stays roughly constant instead of scaling with N. This
  unblocks -parallel N as the intended throughput lever for historical backfills
  (ADR-0047 Phase 0). The indexer's live-tail path (internal/pipeline.LedgerstreamConfig)
  is untouched — it runs exactly one walker and legitimately wants the larger default.