Stellar Index v0.51.0
[v0.51.0] — 2026-08-31
Operator action required: no — restart and done, but see the two
served-behaviour changes below before you deploy.
Tested against pubnet protocol 23.
Migration notes: none. No new migrations; schema head is unchanged at
0150.
Two changes alter what public endpoints SERVE, and are the reason to
read this section rather than skim it:
/v1/oracle/prices,/v1/assets/{id}'schange_24h_pct, and the
change-summary worker now fold BOTH stored market directions. The
SEP-40 series will return more buckets than before, and some
previously-served values move to the volume-weighted union price.
That is the correct answer — the old readings were computed from one
leg of a two-sided market — but it is a visible change on a published
oracle surface./v1/vwapand/v1/twapnow withhold (404price-withheld) for
a directory-scam-flagged issuer, matching every other price surface.
Previously they served a price at 200.
Fixed
-
Five guards that did not check what they claimed — all mine, all
found by an adversarial review of the 2026-08-31 merge range, each
fix proven by mutation.lint-rule-structure.py's metric-label regex crossed declaration
boundaries. A non-greedy gap scanned past a label-less
NewGauge/NewCounteruntil it found the NEXT metric's
[]string{…}and credited it to the wrong metric — 19 metrics
affected. So the new fixture-realism check ran against fabricated
"declared" sets and its error message named labels the emitter does
not have:stellarindex_anomaly_freeze_activeis a bareNewGauge
with zero labels, yet the lint credited it with{op}and passed a
fixture writing a series production can never emit.lint-unit-failed-baseline-test.shwas executed by nothing. I
wrote it, its commit trailer claimedverify.shran it, and it was
wired into neitherverify.shnor CI — andcheck-verify-parity.sh
only enforces CI→verify, so a script in neither is invisible. Now
wired to both. Its check was also one-directional: it walked
baseline→regex only, so a unit added to the exclusion regex with no
baseline entry was silently exempted from the catch-all and had
no dedicated alert. The reverse direction now fires.lint-deploy-systemd-authority.shasked two independent
questions. "Is- <unit>a list item anywhere in tasks/" AND "does
deploy/systemd/{{ item }}appear anywhere in tasks/" — the second
being a repo-wide constant satisfied by one unrelated task. So
classification collapsed to the first, and adding a unit to an
ansible.builtin.systemdenable loop (installing nothing) passed
as authoritative — the exact "enable a unit you don't install"
footgun the lint's own header describes. Both facts are now required
of the SAME task, via a parser rather than greps.- The orphan-branches footer never rendered when it was the only
thing to report. The header promised dispositioned branches are
"still COUNTED, in a footer line", but the issue step was gated on
the orphan count alone — so in the header's own worked example (the
17 landedfix/issue-*branches clearing the grace with no other
orphan) the close step fired instead and the count appeared only in
a job log. - The CS-017 freshness test re-implemented the rule instead of
calling it. Deleting the> r.freshnessWindow()term left the
suite green while/v1/priceresumed serving months-old buckets
withstale=false. The rule is extracted to
storePriceReader.bucketIsStaleand the test now calls it;
deleting the term fails the suite.
-
Every R1 deploy was failing its config-apply gate and skipping the
post-deploy smoke test.deploy.ymlread the host's version
sidecars withcat …/deployed-versions/stellarindex-*, but those
files are written byansible.builtin.copy: content: "{{ version }}",
which writes no trailing newline. Six binaries therefore
concatenated into one token (v0.46.1v0.44.7v0.28.1…); the
^-anchored version filter matched it andsort -V | head -1passed
it straight through. The gate could not resolve that to a commit and
failed CLOSED — so the binaries deployed fine, the job went red, and
theServed-path smokestep (which has noif:) never ran.Now reads with
awk 1, which emits each record with a trailing
newline whether or not the file had one, and the version filter is
anchored at BOTH ends so any future malformed sidecar is rejected
rather than mistaken for a version.The gate's self-test gained three checks, each proven to fail on its
own defect: the newline-safe read, the end-anchored filter, and — a
separate latent hole — that the baseline is read before the
playbook. That last one previously asserted the ordering in its
message while only grepping for two strings independently, so moving
the baseline step below the deploy would have kept it green while
making the gate permanently vacuous (the "live" version would be the
version just deployed). -
A withholding guard I deleted let the MSP-07 regression back in.
The review sweep replacedTestPriceServingSeamsAreGated's weak
two-entry subject list with a derived one — correctly — but removed
TestWithholdingGatesAreSpelledOnlyAtTheChokepointin the same
change, while that commit's own message said "the MSP-07 half (drift
WITHIN a seam) was always real".main.gowent on citing the
deleted test by name.The two guards answer different questions. The surviving one asks
"does every serving seam consult the gates at all" — a method with two
arms satisfies it as soon as ONE arm callspriceWithheld(). The
deleted one asks "is the withholding decision spelled in exactly one
place", which is the only way to catch a single arm drifting.Verified: reverting the last-trade arm to
!r.substance.Allowed(…)—
the literal MSP-07 code, which drops the scam gate — passed CI
before this restore and fails by file and line after it. That
regression matters because an operator setting
disable_substance_gate=trueto diagnose a coverage complaint would
silently publish a directory-flagged issuer's last trade as its price,
reversing an owner-level trust decision they never touched. -
The SDK's spec-coverage table was bound to nothing (wave-D
F-SDK-10).TestSDKCoversSpecreconcilescoveredOperationsagainst
the OpenAPI spec, so it catches an endpoint the SDK forgot — but it
cannot catch the table drifting from the SDK. Renaming or deleting a
Clientmethod left the table naming the old identifier while the
reconciliation stayed green. A new guard binds it both ways: every
tabledsdkMethodmust resolve on*Client, and every exported
*Clientmethod must appear in the table or in an exemption set with
a stated reason. The second direction is the load-bearing one —
without it the table only ever describes the subset someone
remembered to add.docs/audit/recipe.mdalso recordedpkg/clientas never
adversarially audited with no behavioural coverage. It was audited on
2026-08-04 and has 156 tests at 78.6% statement coverage, and
"retries" cannot be a gap because there IS no retry layer. Corrected,
and pointed at where the SDK is actually weak. That entry was itself
an instance of the class it now warns about. -
Three price reads served ONE of the two stored market directions
(wave-D UNAUTH-DOS-9). The decoder files each trade in the venue's
observed base/quote ordering and deliberately does not normalise, so
a two-sided market lands in the CAGGs as BOTH(A,B)and(B,A)
rows — asdirVWAP's own doc says, "every serving read has to fold
the two together itself". Three did not:RecentClosedVWAP1mForPair(/v1/oracle/prices) dropped every
minute the market traded only the other way. The SEP-40 series went
silently sparse, and for a predominantly-flipped pair the endpoint
returned200 []for an asset/v1/oracle/lastpricepriced
without difficulty — two endpoints on the same declared SEP-40
surface disagreeing about whether the asset had any history.ClosedVWAP1mAtOrBefore(/v1/assets/{id}change_24h_pct)
understated the 24-hours-ago anchor two ways: for a two-sided
bucket it returned one leg's VWAP as if it were the bucket's,
ignoring the other leg's volume; for a flipped-only bucket it
matched nothing and the percentage vanished.TimedVWAPs1mForChangeSummary(the change-summary worker), the
same defect in the aggregator's series read.
All three now read both orientations and fold them with the exact
volume-weighted unioncombineDirVWAPalready serves — the
inversion stays in Go, never in SQL, where1.0/vwapwould round
the flipped leg before it was weighted (ADR-0003).This is the same bug for the third time:
LatestClosedVWAP1mForPair
was fixed for it in audit-2026-07-23 (MNY-06), and that fixer
reportedRecentClosedVWAP1mForPairas R-076 — recorded in the
audit's remediation state and never dispositioned. Each earlier fix
shipped a test pinning that one function's query, and none could see
the next reader. The new guard is written against the CLASS instead:
it parses every string literal in the package and fails on any
pair-bound CAGG read that filters a single orientation. It found the
third instance immediately — one that neither the finding nor its
skeptic had spotted — and it fails loudly if its own subject set ever
comes back empty. -
/v1/vwapand/v1/twapserved a scam-flagged issuer's aggregated
price, and the guard's own docs said they didn't (wave-D
MSP-02/EXR-04). Every other price surface withheld it —
/v1/price,/v1/price/tip,/v1/price/batch, the SEP-40 oracle,
the asset headline — while these two returned 200 with a number.
Reproduced live against a directory-flagged issuer before the fix.The gap was documented as fixed:
pricingguard/scam.goclaimed the
gate sat "at the price-reader seam so every reader-backed surface
(…,/v1/twap,/v1/vwap, …) is covered by ONE gate", and PR #182's
merged body repeated it verbatim. Neither endpoint goes through the
price reader at all — both compute from raw trades via their own
fetch — so the claim was never true, and no test contradicted it. The
doc now states the six real call sites and says plainly that no
single seam covers them all.Gated in the two HANDLERS, deliberately not in the shared
tradesInRangeWithStablecoinFallback: that helper is also the fetch
behind the single-bar/v1/ohlc, which the guard's own docs, the
config reference, and the withheld problem's own guidance text all
promise stays visible — gating there would make our error message's
escape-hatch advice a lie.Scam gate only, not the substance gate. The scam gate is targeted
(flagged issuers) and directly implements the 2026-08-25 decision.
Applying the substance gate here would newly 404 every thin pair —
a breaking change, and arguably wrong on principle, since ADR-0015
andVWAPResult's own doc position/v1/vwapas the "narrow the
window and compute it yourself" surface opposite/v1/price. That
is an owner decision, not something to smuggle in with a scam fix. -
A withheld price verdict reached on the stablecoin-proxy leg was
swallowed, so the API said "no price data" (wave-D MSP-06). The
direct fiat read misses for the dominant on-chain shape (trades are
quoted in issuer stablecoins, notfiat:USD), the handler enters
priceFallback, the proxy loop'sLatestPrice(asset, <peg>)returns
ErrPriceWithheld— and the loop's barecontinue, written to skip
an INACTIVE peg, discarded that verdict along with the miss. The
response waserrors/price-not-found.Those are different answers. "No price data" tells a customer to look
nowhere; the withheld problem names/v1/observations,/v1/ohlcand
/v1/history, where the data IS available — that guidance is the
whole reason the distinct problem type exists. Same swallow affected
/v1/oracle/lastpriceand/v1/oracle/x_last_price.Withheld is now sticky, not terminal: a later peg that can serve
still wins (a real price beats a 404), and the verdict surfaces only
when nothing serves. The batch path keeps dropping it, deliberately —
that envelope has no per-row problem shape.The two existing withheld tests were left untouched. They are
load-bearing — mutation testing confirms deleting the direct-read
short-circuit fails both — and the finding's suggestion to rewrite
them would have deleted live coverage. -
stellarindex_oracle_stalecould never fire, for any oracle —
and its test passed anyway (wave-D ALERT-02). The rule compared
stellarindex_oracle_last_update_unix(labels{source, asset})
againststellarindex_oracle_resolution_seconds(labels{source})
with noon()/ignoring(). A vector-to-vector operation requires
identical label sets, so no pair ever matched and a silent oracle
raised nothing. Now joined withon (source) group_left(), which
keeps the left side's per-asset cardinality so one silent asset still
tickets.The promtool case covering it was green only because it fabricated an
assetlabel on the resolution series — a label the emitter is
structurally incapable of producing (WithLabelValueswith two
arguments on a one-label vector panics). With a realistic series the
rule returns nothing at all, which is what the corrected test now
proves. A second case pins the fan-out, so a regression to a bare
comparison fails rather than silently matching nothing. -
Both usd_volume coverage alerts went silent at exactly 100% NULL —
their worst case (wave-D ALERT-04). Each divides a "priced" counter
by a "total" counter, but theusd_volume_populated="yes"child is
created only by the first priced insert (there is no zero
pre-initialisation). So when a source prices nothing at all, that
series does not exist, an aggregation over it is the empty vector
rather than zero, and the division yields no sample: total pricing
failure was the one condition these alerts could not see. Partial
failure fired correctly, which is why it was never noticed.Both arms now substitute an explicit zero for a source present in the
denominator, so 0/N = 0 and the alert fires. Neither alert had any
promtool coverage; there is now a test file for both, including
full-coverage controls so the fix cannot over-fire. Proven red against
the pre-fix rules — both cases returned no alert whatsoever. -
A scam-flagged issuer still published an all-time-high dollar price,
and a declared-peg asset still served the price series the listing
refused (wave-D MSP-05 / MSP-04).suppressScamIssuerPricingnulled
six fields but notath, so a directory-flagged token returned
"price_usd": nullnext to"ath": {"usd": "0.0091"}— a published
USD valuation, from the same USD-quoted CAGG, for an asset the
platform had just decided must publish none. Separately, the listing's
sparkline rule and the detail path's series suppression asserted the
same product question in two places and drifted: the listing excluded
declared-peg rows, the detail path did not, so/v1/assetsdrew no
sparkline while/v1/assets/{id}served a full
price_history_24h/_7dcharted from the dust market the substance
gate had refused.Both paths now share one predicate,
priceSeriesPublishable, which
answers "may this payload carry a derived price-over-time claim?" for
sparklines,price_history_*andathalike. The peg price itself is
untouched — the peg is the published claim; only the market series
charted beside it goes. -
Two more alerts that could not fire (wave-D ALERT-03, ALERT-06),
both instances of the same class as #389/#390 — an expression nothing
evaluated against what the emitter can actually produce.stellarindex_external_poller_error_rate_highsummed
rate(…{outcome="success"}) + rate(…{outcome="error"})with no
matcher. PromQL's one-to-one matching compares the full label
signature andoutcomediffers on every candidate pair, so the+
yielded the empty vector unconditionally: the alert could not fire at
any error rate. Nowsum without (outcome)joined with
ignoring(outcome), which keepsjob/instancefor the annotation.
Thesuccess|errorselector is deliberate — a third outcome,
skipped(post-429 cooldown), is excluded on purpose, and summing all
outcomes instead would leave the alert dead during throttling, the
exact incident shape it exists for.stellarindex_divergence_no_referenceand
…_refresh_error_dominantcompare a failure outcome's rate against
theokoutcome's, butstellarindex_divergence_refresh_totalwas
the one alert-referenced outcome counter missing from
seedBoundedLabelSeries. A counter child does not exist until its
first increment, so an aggregator that had never completed a
successful refresh — every reference unreachable and the process
restarted mid-outage, as deploys routinely cause — had nookseries,
making both comparisons empty and both alerts silent while
flags.divergence_warningserved frozen and a live depeg went
unflagged.The seed guard derives its subject set from the emitter source, so a
new outcome added without a matching seed fails on the day it is
written. -
The verify-archive staleness page measured when the timer last
fired, not when verification last succeeded (wave-D ALERT-10).
stellarindex_verify_archive_run_staleis aseverity: page
guarding R1's role as integrity leader (ADR-0016) — the nightly
chain verification R2/R3 trust. It read
node_systemd_timer_last_trigger_seconds, which systemd updates on
every firing regardless of how the triggered service exits. So a job
that failed every single night kept the gauge perfectly fresh, and
the page was defeated by exactly the scenario its own description
names: "either the timer isn't enabled or every recent attempt
failed." The Tier B ticket had the same defect.Both now read
stellarindex_verify_archive_last_success_unix, a new
per-tier gauge the verify-archive binary writes into its existing
node_exporter textfile and advances only on a clean exit; a
failed run carries the prior value forward. A host that has never
completed a run emits an explicit0rather than no series at all.The runbook had already documented the limitation as a known caveat
— a NOTE under Symptoms and a state-table row marked "Shouldn't
happen". That row describes a reachable, expected state now, and both
are rewritten. Writing a limitation into a runbook makes it
survivable; it does not make the page work. -
The total-ingestion-loss SEV-1 sent a RESOLVED while ingestion was
still completely down (wave-D ALERT-01).
stellarindex_ingestion_all_sources_stoppedmatched on
sum(rate(stellarindex_source_events_total[5m])) == 0. When the
indexer process dies the series stops: for a few minutes its last
samples are still inside the range window andrate()is 0, so the
page fires correctly — then the samples age out, the series vanishes,
andsum(rate())over nothing is the empty vector rather than zero.
The alert stopped matching and Alertmanager resolved a P1 that was
still fully in progress, which reads to a responder as "it fixed
itself". It also never fired at all when the indexer was already down
as Prometheus started, or when Prometheus restarted mid-outage.An
absent_over_timebranch now takes over at exactly the point
rate()gives up — both use a 5m window — so the page persists until
events resume. This adds no new paging scenario: the rate branch
already fired after 3m of silence, so any restart longer than
for: 3mpages today. What changes is that the page no longer lies
about recovery.The existing test could not have caught it — its fixture is a flat
counter, which is a stalled source with a live process, not a dead
one. A truncated-series case now covers the process actually dying. -
A percent-encoded slash forged the SSE exemption, so 13 routes could
be asked to run with no request deadline at all (wave-D
UNAUTH-DOS-4).RequestTimeoutexempts streaming endpoints by the
/streampath suffix, but tested that suffix againstr.URL.Path—
the decoded path — while Go's mux routes on the escaped form. Those
disagree exactly when a wildcard segment contains%2F:
GET /v1/assets/native%2Fstreamroutes to the ordinary
/v1/assets/{asset_id}handler while its decoded path ends/stream.
The exemption now keys onr.URL.EscapedPath(), which is what the mux
itself routes on, so the two cannot disagree about what a request is.Guarded by a sweep that enumerates the routes from
server.gorather
than listing them: the forgery worked against every trailing-wildcard
route, and new ones are added regularly, so a hand-written table would
pin today's routes and miss tomorrow's. It found 13 pre-fix, and
checks the converse too — genuine SSE routes must keep their
exemption, since a fix that bounded the streams would be worse than
the bug. -
The SEP-40
prices()closed-bucket read was not sargable (wave-D
UNAUTH-DOS-3): it applied a function to the indexed column
(bucket + INTERVAL '1 minute' <= now()), so the planner could
neither use the bucket index nor prune chunks at plan time. Rewritten
tobucket <= now() - INTERVAL '1 minute'— semantically identical,
no change to what is served. Its sibling combined-direction template
already had the correct form.It drifted because it was an inline
const qinside the function
body, invisible to the package's existing sargability guards, which
assert over package-level templates. Hoisting it is the durable half
of the fix and is what makes a guard possible at all.Deliberately not given a literal lower bound or the 14-day existence
gate its neighbours use: both change what a documented public endpoint
serves (a dormant asset's last N closed buckets becoming an empty
array), which is an owner decision rather than a query-shape fix. -
A TimescaleDB counter built to make silent CAGG starvation
alertable had no alert (wave-D ALERT-12), and the lint that should
have noticed could not see its emitter (ALERT-07's class).stellarindex_timescale_job_failures_totalexists because r1 once
failed 37–69% of every CAGG refresh run withfailed to start job—
background-worker starvation — and nothing surfaced it: the jobs got a
slot on a later tick, solast_run_statusreadSuccess, the caggs
were never stale, and both existing alerts stayed correctly quiet. The
only evidence was this counter, which no rule referenced. Now
stellarindex_timescale_job_failures_climbing(informational, >10
failures in 6h sustained 30m) with a runbook that branches on the
actualerr_message, sincefailed to start jobmeans starvation
rather than a broken job body.Adding it exposed the lint problem.
lint-metric-refsrejected the
rule as referencing a metric "nothing emits" — but marking it
KNOWN_INERTwould have been false: it is emitted and scraped every
60s.is_emitted()grepped only*.go/*.sh/*.prom, and several
textfile probes live as inline ansiblecontent:blocks, so their
real metrics had all been parked inKNOWN_INERTwith comments
reading "NOT inert: the probe timer runs every minute on r1". That
made the list mean two incompatible things — "no producer exists" and
"the producer is invisible to this lint" — which is the confusion the
gate exists to prevent.The lint now scans ansible
tasks//handlers/YAML, scoped so a rule
file cannot satisfy its own reference, with comment-stripping intact
so a metric named only in a comment still counts as dead. Its own
stale-inert check then flagged 7 metrics wrongly listed as inert,
including the galexie-catchup and stellar-stack-version probes; all
seven are removed.KNOWN_INERTmeans "no producer" again. -
A promtool fixture could assert against a series production cannot
emit, go green, and certify an alert that could not fire. That is
not hypothetical:stellarindex_oracle_stalewas unfireable for every
oracle while its test passed, because the fixture wrote
stellarindex_oracle_resolution_seconds{…,asset="XLM"}and that
metric is declared with one label —WithLabelValueswith two
arguments panics.lint-rule-structurenow checks every rule-test
series:label set against what the emitter can actually produce.The declared set is the union over every emitter shape, not just the
Go vector:verify-archivedeclares{chunk_idx, reason}in-process
but production scrapes its node_exporter textfile, which carries
{tier, reason}— checking against the Go declaration alone flags
correct tests. Labels attached by the scrape (job,instance, and
r1's staticbinary) are read from the scrape config rather than
hardcoded, so a new target label does not turn the lint into a source
of false failures.It found two real violations: a second copy of the
asset="XLM"
fabrication, in a negative case that passed either way — which is
exactly why it survived when the firing one was corrected — and three
stellarindex_trade_inserts_total{usd_populated="true"}fixtures
inventing both a wrong label name and wrong values
(usd_volume_populated,yes/no).A rule-test coverage-percentage gate was considered and rejected: it
would have caught none of this wave's unfireable alerts, and it
creates pressure to shrink a baseline by writing more fixtures — the
mechanism that produced the false greens in the first place. -
Three ways a malformed
/v1/assetscursor got past validation
(wave-D KP-2), one of which failed silently.isNumericPrefix
required no digit, so a volume prefix of.validated and was bound
as$n::numericfor Postgres to reject; the rank tier was checked
only for being digits, so2147483648reached anint4placeholder;
and an over-int64observation-count prefix passed, then made
parseAssetCursordegenerate the whole cursor to(0, 0, "")— which
matches no rows, so the client got a 200 with an empty page,
indistinguishable from end-of-pagination. All three are now rejected
at the boundary with a 400.markets.gocarried a hand-copied
duplicate of the same digit-less loop and now shares the helper. -
A page whose rows were all folded away stopped pagination dead
(wave-D KP-3). The next-cursor was emitted only when
hasMore && len(out) > 0, butoutshrinks after the query —
suppressCatalogueTwinsdrops rows andfoldAliasTwinscollapses
them. A page that folded to empty therefore emitted no cursor while
more rows remained, and the walk ended early. The cursor is built from
the raw last row, which exists wheneverhasMoreis true, so it is
now emitted onhasMorealone. -
The config-apply gate diffed against the wrong baseline, so a
catch-up deploy was told "the binary deploy is complete" (wave-D
LID-5).deploy.ymlcalledconfig-apply-gate.shwith two arguments,
omitting the host's live version — so the gate fell back to "the
previous release tag by ancestry", which is only correct when the
fleet is exactly one release behind.deployed-versions.mdstates
plainly that a tag cut does not imply the fleet moved to it, and 8 of
23 adjacent tag hops over v0.40.0..v0.49.0 have an empty
config-surface diff — so the green branch is reachable.Measured on real tags: the two-argument form reports "no
config-surface changes between v0.47.1 and v0.47.2 — the binary deploy
is complete" and exits 0, while the three-argument form against a
real host baseline of v0.45.0 finds 13 changed config surfaces and
exits 1.The workflow now reads the host's deployed-versions sidecar before
the playbook runs — afterwards it reports the version being deployed,
which would make the baseline trivially equal and the gate vacuous —
and takes the lowest version across the managed binaries, since config
is unapplied if any binary predates it. Best-effort: an unreachable
sidecar falls back to the documented ancestry default with its
existing warning, because failing the deploy there would trade a
weaker gate for an outage risk.The script and its self-test were always correct; only the caller was
wrong. So the new guards check the caller — a script-level test
cannot catch a caller-level omission. -
The binary version-skew probe scored an absent binary as perfectly
healthy, and its alert claimed to cover that (wave-D LID-2). The
probe globs the install directory, so a file that is not there is
never visited: an entirely absent release binary yieldedskew=0
andprobe_success=1. A present-but-non-executable one was skipped
just as silently by[ -x "$path" ] || continue— while the alert
description named "missing, not executable" among the causes it
covers. Two of its four named causes were false.The non-executable half is fixed: a present, managed, non-executable
binary now marks the run degraded. The parked-build skip
(.prev-/.rolledback-) moved above the executable test, so a
non-executable parked artifact cannot be mistaken for a broken live
binary.Absence is deliberately not fixed here. The obvious approach —
asserting all six managed binaries are present — would pin the alert
permanently red on testnet and futurenet, where fewer are installed by
design, and the probe's own comment already warns that "a
permanently-firing alert is the same as no alert". Detecting absence
needs a host-derived expected set, not a hardcoded count. The
description now states plainly what the probe cannot see and why the
naive fix is wrong: an alert that overstates its coverage is worse
than one with a documented gap, because the first makes you stop
looking. -
An oracle row whose stored asset/quote text would not parse vanished
from the served stream with no signal (wave-D SI-OC-04).
LatestOracleStreamsdropped it with a barecontinue— no log, no
metric, no error — so it was simply absent from/v1/oracle/streams
and the explorer's/oraclespage.The silence mattered most exactly where it was most likely: the
documented remediation for a mislabelled oracle row is an
operator-run raw SQLUPDATEagainst that column, which has no
CHECKconstraint. A typo therefore deleted the row from the served
surface rather than erroring — and the operator would watch it
disappear and reasonably conclude the relabel had worked. Now counted
bystellarindex_oracle_stream_rows_unparsed_total{source,field},
with a ticket alert and a runbook whose diagnosis leads with the
operator-typo shapes (missing prefix, truncated strkey,rwa:/raw:
confusion).Deliberately not pre-seeded, unlike the divergence outcome counters:
seeding matters when a rule compares two children of one metric or
divides by one, and this is a bare threshold on a single series that
appears exactly when the bad thing happens. -
The explorer showed an assumed oracle quote as though it were
observed (wave-D SI-OC-01). An unmapped symbol carries no reliable
denomination, so the decoders record a default —fiat:USDunless the
symbol ends in a recognised fiat suffix. The unmapped panel rendered
that default as a linked quote beside the price, so a hypothetical
wstETH/ETHor barewBTC_FUNDAMENTALwould display "USD" next to a
number that is not dollars. The column is now labelled assumed, is
no longer a link, and carries a hover saying it is a decoder default;
the OpenAPI description says the same for API consumers.Only the presentational half is changed. Broadening the suffix rule
substitutes one guess for another and contradicts the design doc, and
the bare-suffix case is not inferable at all —mapped: falsealready
means the denomination is unknown by design, and that is the contract. -
deploy/systemd/is mixed authority, and four units in it are
installed by nothing (wave-D LID-7). Ansible copies
config-assertions.{service,timer}straight out of that directory, so
editing those changes production — while every other file is either a
reference copy shadowed by a.j2in the role (editing it changes
nothing, which is how several drifted) or an orphan that nothing
installs at all.r1-deployment-state.mddocuments an operator
convention of scp-ing units straight from here, which makes an
undeclared orphan a live footgun: it looks like a deployed unit, it
starts if copied, and nothing ever reconciles it.A new lint classifies all 23 unit files as installed, templated, or a
declared orphan, and fails on anything else — so the three states stay
distinguishable instead of being folded into "files in a directory".
The four orphans are declared with reasons rather than deleted:
ch-live-catchup.{service,timer}are the ClickHouse lake's only
self-healer and their absence is a real gap (LID-1), while
stellarindex-completeness.{service,timer}are superseded — their
ExecStarttarget does not exist anywhere in the repo and the role
shipscompute-completenessfor the same job. -
ch-rebuild -writeleaves the completeness verdict carrying a stale
clean claim over the range it just rewrote (wave-D CV-1).
projector-replayrecords a projection dirty window so the next
compute-completenessre-reconciles the rewound range;ch-rebuild
records nothing, so the nightly verdict keeps its prior clean claim.It now says so, loudly, at the point of use — an operator running
-writeis told to note the window and re-check the affected sources'
reconcile before trusting the next/v1/coverageverdict.The automatic record is deliberately not implemented yet, and the
reason is measured rather than cautious: one source's dirty window
(aquarius) already blew the reconcile pass's 120-minute deadline and
needed a bespoke prefilter, against a 180-minute service timeout.
Recording windows for the eight sources the rebuild script drives over
~12.9M ledgers would force the next nightly to re-reconcile all of
them un-prefiltered — a likely timeout that takes out every
source's verdict, which is worse than the stale claim it fixes. That
needs a bounded per-window re-reconcile and a re-measured pass
wall-clock, neither of which can be established without the real lake. -
A completeness dirty window could be cleared on evidence that
predates the rewrite it certifies (wave-D CV-6). The clear's bounds
(from_ledger >= $2 AND to_ledger <= $3) correctly protect a
widened window — a concurrent replay that grew the range leaves a row
outside the bounds, which survives — but not a subset re-record: a
replay re-recording the same or a narrower range leaves both bounds
satisfied, so the delete erased evidence of a rewind the run never
verified and the next verdict carried a clean claim over it.The guard was already provisioned: migration 0125 declares
updated_atand the upsert maintains it. Comparing it makes the clear
optimistic concurrency — it succeeds only if the row is the one whose
obligation this run discharged. Any re-record, wider or narrower or
identical, bumps the timestamp, the delete matches nothing, and the
window stays pending. Fail-closed, costing at most one extra
reconcile.The bounds are kept: they and the timestamp cover different races, and
a test asserts both survive. Two alternatives were rejected on the
finding's own analysis — tightening the DELETE to equality changes
nothing (a subset re-record leaves both bounds identical), and gating
on the live cursor would make windows recorded at or above it
permanently unclearable, which is the stale-window treadmill this repo
already suffered. -
Two oracle-path comments asserted things the repo's own evidence
contradicts (wave-D SI-OC-05). The external dust-floor constant's
docstring claimed$1is an upper bound across the fiat allow-list.
That was true when written (32 codes, GBP/CHF ≈ $1.3 the richest), but
the list was later widened to 133 and brought in KWD ≈ $3.26,
BHD ≈ $2.65 and OMR ≈ $2.60.The direction of the resulting error was also stated backwards, and is
corrected: under-stating the reference over-states the floor, so a
$3.26 KWD leg gets a floor ~3.3× stricter than intended — the
size-biased-dropping direction the constant exists to prevent, not the
harmless one. The exposure is bounded at compile time (every fiat
quote leg a streamer can see is hard-coded, all ≤ ~$1.35), so the
single constant stays; a real FX table would be false precision for an
order-of-magnitude threshold and would rot.A reflector test comment also stated magnitudes the repo's own
captures contradict (VES 7.3e-6 / XAU 4,100 against a real 2.07e-3 /
4720.90). The constants are left alone deliberately — the decode path
has no magnitude-dependent branch, so restating them buys no coverage
— and the comment now says so and points at the real-fixture test.The behavioural half of the proposed remedy is rejected: returning
no-floor for an unvetted fiat resolves to 1e-8 whole units, which
disables the dust guard for that leg — strictly worse than a
too-strict floor. A third claim, that aParsePaircomment
misdescribes the accepted grammar, is refuted: the comment cites
api-design.md§3 specifically and is accurate about that document. -
Canonical
rwa:assets linked to pages that do not exist (wave-D
SI-OC-02). ADR-0028 ids likerwa:XAUfell through to the bare-code
branch, producing/assets/rwa%3AXAU— the API 404s on the prefixed
id and 400s on the bare code, so both spellings are dead. They now
render as a plain label with the full id in the tooltip, unlinked.Deliberately not "strip the prefix like
fiat:/crypto:": that would
produce/assets/XAU,/assets/BENJI, which the API rejects — one
dead link swapped for another, plus the loss of the namespace signal.
Real per-asset RWA pages are separate work. -
A server response field shipped invisible to every consumer, and
two reconciliation gates could not see it (wave-D F-SDK-04). F-1321
moved the issuer's SEP-1 rounding hint offdecimals— where it
inflatedmarket_cap_usdby up to 10^(7−display_decimals)× and was an
issuer-controlled manipulation vector — onto a newdisplay_decimals
field. That field never entered the OpenAPI spec, sopkg/clientand
the explorer's generated types both dropped it: the remediation's
entire replacement surface was unreachable from the published
product. A wallet had no way to obtain the issuer's stated
preference.It is now documented in the spec, carried by the SDK, and present in
the generated explorer types.Neither existing gate could have caught it.
lint-docscompares
handlers to the spec at route granularity, never fields; the SDK's
contract test compares the SDK to the spec bidirectionally — which is
useful, but reconciles two derived artifacts, so when a field exists
only on the server they agree with each other about being wrong. A new
test compares the handler struct — the source of truth — to the
spec, fails on any undocumented response field, and fails loudly if
its own subject set comes back empty. -
Three SDK docs taught things the server does not do (wave-D
F-SDK-01/02/03).TradeRow.BaseDecimals/QuoteDecimalswere documented as a property
of the asset ("7 for native/classic/fiat"). They are a property of
the row's source, and a single page mixes both: on-chain rows
carry the asset's own scale, while CEX rows carry 8 regardless of
the pair. That error is payload-undetectable —priceis
quote/base and therefore scale-invariant, so nothing in the response
looks wrong — and a reader assuming a constant mis-scaled real rows in
production once already. Corrected in the SDK, the handler, and the
OpenAPI description, since an SDK-only edit would leave the
machine-readable contract still teaching it.MarketsOptions.OrderBynamed the wrong server default: it said
alphabetic, but the default switched to volume-desc in May because
alphabetic surfaced spam tokens at the top. Corrected in all three
places the claim lived, and the godoc now says why a full-catalogue
walker should passpairexplicitly — volume-desc ranks on a
mutable key, so a pair whose volume changes mid-walk can be seen
twice or missed.The SDK deliberately does not start sending
order_by=pair: that
would silently flip every existing caller from top-by-volume to
alphabetically-first spam tokens, and make the SDK disagree with an
equivalent curl. Converting a wrong comment into wrong behaviour is
not a fix. Likewise rejected: wideningparseAPIErrorto keep 256
bytes of an unrecognised body, which would let a proxy inject
arbitrary text into an error string that lands in customer logs. -
The ClickHouse lake's only self-healer was installed by nothing
(wave-D LID-1).ch-live-catchupfills holes in the Tier-1 lake, and
both the script and its systemd units already existed and were
covered by the ops-credential test — but no task inconfigs/ansible
ever installed them, so on a playbook-provisioned host the healer was
simply absent. That is not cosmetic:resolveTipclamps to the
contiguous watermark, so a single unhealed gap stalls the CH-fed
projector permanently rather than degrading. It is reachable wherever
the lake is live — the testnet and futurenet inventories set
run_clickhouse: true, and the config template defaults both the live
sink and the projector source to true.The role now installs the script and both units and enables the timer,
behindch_live_catchup_enabled(default true, and only where the
ClickHouse tasks run at all). Units are copied verbatim from
deploy/systemd/rather than templated: they carry no host-specific
values, and copying keeps the checked-in file the thing that actually
runs.
Reviewed, no change
-
MSP-03 (four more surfaces bypass both gates:
/v1/marketsand
/v1/poolslast_price,/v1/chart?price_type=market_cap, and
windowed/v1/price). Two of the four are an open OWNER decision,
not a broken guard:last_priceis documented as the raw
quote-per-base ratio — the same data class as deliberately-ungated
/v1/ohlc— and issue #366 states the unresolved scope question
verbatim. The windowed tier is additionally unreachable in the
shipped config, whose aggregate pair set is allcrypto:/fiat:,
for which the scam gate returns false immediately. The root cause is
already recorded in #366 and #182. -
ALERT-05 (Alertmanager's inhibit rule keys on
componentalone,
so one page suppresses every ticket of that component). The mechanism
is real and was reproduced against the shipped config, but the harm
claim is not: suppressed tickets are re-delivered ~5 minutes
after the page resolves, not at the next 24hrepeat_interval—
inhibition is applied in the notify pipeline after the dispatcher
flush, so nothing is written to the nflog and the first post-unmute
flush notifies immediately. 14 of the 109 are informational alerts
routed to a receiver with no integrations, so they reach Discord
neither way; and every suppressed alert stays queryable with
inhibitedBy. It is also a duplicate of open audit finding OBS-02,
which records the same title, files and proposed fix. Left for that
finding's owner — an Alertmanager routing change is a production
paging decision — with the measured 5-minute number noted here,
because it materially lowers the severity OBS-02 was filed at. -
The R2/R3 deferral rationale rested on three false cache claims
(wave-D PS-07).multi-region-ha.mdsaid/v1/price,
/v1/oracle/latestand/v1/ledgers/latest"all return
Cache-Control: no-store". None is true at HEAD:/v1/pricereturns
public, max-age=30, s-maxage=60— the SAME switch case as
/v1/assets, which the entry contrasted it against —/v1/oracle/
returnsmax-age=60, s-maxage=300, and/v1/ledgers/latestis not a
route at all (latestbinds{seq}, fails to parse, and 400s). The
policy is the original April 2026 one, four months older than the
text, so "the deployed binary was older" was never available as a
defence. The conclusion survives on the real numbers — a 30-60s edge
TTL is not a substitute for a regional origin, since a Singapore
consumer still pays full origin RTT on every miss — but the argument
now says so from facts, and the micro-cache experiment it proposes
reads as more attractive rather than less. -
PS-03 (restore-drill's ClickHouse stage is opt-in via
DRILL_CH_WINDOW, so the scheduled monthly drill never measures lake
re-derive throughput). Real, but the opt-in IS the shipped documented
design, the gap is disclosed in three docs, and it is already tracked
by open issue #343. The finding's one increment over #343 — that
a metric alone could never populate, because the stage is gated — is
worth recording there, not re-filing. -
PS-04 (ADR-0043 §2.3's "tail insurance" rests on a premise the
repo's own analysis contradicts, and is unimplemented). The premise
really is wrong-as-written, but ADRs are immutable
(docs/adr/README.md) — superseded, not edited — and the corrected
assessment already lives inoff-site-backup-plan.mdwith a drafted
amendment. Nothing to change without a superseding ADR, which is an
owner decision. -
PFR-01 (the supply-divergence alert is unarmed on r1 because
[divergence.supply]is never rendered). Confirmed end to end, and
already stated in the alert rule's own comment plus a registered open
finding. Arming it is an operator/config decision on a production
paging surface, not a repo fix. -
PFR-03 (a narrowed re-run can rewrite
tip_ledgerdownward). The
mechanism reproduces, but the defect CS-083 closed was AUTONOMOUS —
a nightly chunk driver that no longer exists. The trigger now needs
two deliberate operator commands, the first of which must find a real
problem; the end state is detected and annotated on the serving path
(coverageVerdictsStale, and the scenario's regression is 17× that
bound), CI-linted, and pinned by a test using materially identical
numbers. It is a re-report of CS-090's accepted residual. -
PFR-05 (a blocked completeness write is a silent no-op). The
discardedsql.Resultis real; every consequence drawn from it is
wrong for the shipped configuration. The trigger is unreachable on
the deployed path (the driver's tip comes from a strictly monotonic
cursor), the claimed "fresh green verdict" printscomplete=falsein
the only deployed mode, and the backstop claim fails on all three
counts — a 36h per-source staleness gauge alerts on exactly the
column a discarded write leaves unchanged, naming the source, ~2h
after the blocked run rather than a day later. -
The capacity register offered two levers that no longer exist
(wave-D PS-05 / PS-06), on a document whose whole purpose is to be
read during a capacity crunch.Move D (cold-tier enable + bulk LCM trim) was listed as an
unexecuted~3.5 TBoption whose AWS dependency was "not yet
incurred". It executed on 2026-07-26 and reclaimed 1.07 TB —
the estimate was ~3.5× high for a structural reason worth keeping:
early history is sparse, so trimming 78% of the partitions
reclaimed 22% of the estimate; the bytes live in the dense Soroban
era above the cutoff, which was kept. The dependency it was weighed
against is not just incurred but formally accepted (ADR-0043 §2), so
"adds an external dependency" no longer discriminates between the
remaining options. A planner would have added ~3.5 TB of
already-spent runway and ruled the option out on a criterion that no
longer applies. Move G's derived "~4 TB net" and the May-2026
recommendation table are corrected and annotated accordingly.Move E (trades retention) read "Decision status: Lever
available" in a register that marks its dead levers explicitly.
Trades retention is forbidden: migration 0031 removed it, 0031's
own.down.sqlnames re-adding one as "the EXACT mechanism of the
recurring 'rogue retention on trades' data-loss drift", CLAUDE.md
carries it as a standing invariant, Ash re-signed it as launch
decision D5, andtest/integration/migrations_test.gopins it.
Arming it would also trip the completeness verifier immediately —
migration 0116 treats a risingMIN(ledger)on a reconcile target
as loss, unconditionally, "because NO reconcile target has a
retention policy". Marked NOT A LEVER rather than deleted, so a
future reader sees why it was rejected instead of re-proposing it.Also corrected: the
data/postgresrow still cited "ADR-0006
retention", which ADR-0006 itself records as superseded by 0031. -
The launch plan told an operator to mint a credential nothing
reads (wave-D PS-02). W4.5, W4.6 and Recommended-order step 3 all
carried the Gosla-probestack as live code with pending r1-ops
actions — "mint the Partner/Operator-tier key, set
stellarindex_probe_api_keyin the r1 vault" — and cited
10-observability.ymlline ranges as installing units that the same
file now removes. The whole stack was retired on 2026-08-24
(634d4be6, #135): both stacks wrote the same textfile, so the
keyless Go stack's 401/429 runs stomped the wrapper's passing
verdicts. An operator working the plan would have minted a live
operator-tier key with no consumer, whose file the next
--tags observabilityapply deletes — credential sprawl on the exact
surface W6.3 exists to shrink — then hunted a unit file ansible had
already removed. The genuine item, rotating the key exposed in a
2026-08-15 transcript, is preserved and now points at the file that
actually holds it. -
The launch-day migration gate named a version 7 behind HEAD
(wave-D PS-01). §2.8 hardcodedschema_migrations.version = 143when
head was 0150. Made version-agnostic rather than re-pinned — "143 →
150" just reproduces the defect at 0151 — pointing instead at the two
places CI keeps in agreement (migrations/head and
ExpectedSchemaVersion, guarded by
TestExpectedSchemaVersionMatchesMigrationsHead). The gate's floor
semantics (applied >= expected, deliberately not==) are now
stated, since the old text's "≤142 or dirty" phrasing invited reading
a schema AHEAD of the binary as a failure.migrations/README.md's register was also missing rows for 0138-0143,
0145-0147, 0149 and 0150, against the file's own mandate. Backfilled,
andlint-migrations.shgained a third pass that fails on a migration
with no row, a row naming no migration, or its own pattern going
vacuous. The reader this costs is the one the register exists for —
someone bringing up a fresh database, for whom the row is where an
"⚠ operator must re-materialize" warning lives. 0147 is exactly that:
it leaves nine CAGGs empty, and r1 having already run it does nothing
for a new node. -
api.status_serviceswas validated case-insensitively and consumed
case-sensitively (wave-D RD-05). Config validation lower-cased each
entry before checking it against{indexer, aggregator}, but
statusServicesOronly trimmed — and the heartbeat map is keyed by
Prometheusjoblabels with thestellarindex-prefix stripped,
which are always lower-case. Sostatus_services = ["Indexer"]
booted clean and then reported"status": "unknown"on every
/v1/statusrequest forever:overallnever left degraded and the
explorer's status page stayed amber, while the operator debugging it
found a value that passed validation and matched the documented
vocabulary — the exact symptom the list was added (#328) to remove.
Both halves now apply the same transform. -
NetworkUnavailablepromised to self-suppress and never did
(wave-D RD-06). Its docstring said it "renders nothing when the route
IS available here" and pointed at anavailablehelper "below" that
was never written; the component always rendered the empty state.
Nothing was visibly broken — all five callers guard with
if (!routeAvailable(…))first — but the next network-gated surface
written by following that comment would have shipped "Not available
on Mainnet" above its real content, on mainnet (/exchangesand
/bridgesare both inROUTE_CAPABILITYwith no page-level gate).
The component now honours the contract, and the available-route
branch — the case no test covered, which is how the drift went
unnoticed — is now covered. -
The orphan-branch tripwire was about to report 17 non-problems on
its first real fire (wave-D RD-04). This repo's remediation flow is
a worktree fixer pushingfix/issue-<N>, then a BATCH PR squashing
the verified subset (#353, #364) — and a squash-merge leaves no
ancestry, so a landed fix branch is mechanically indistinguishable
from a forgotten one: no PR, stale against main. On the first tick
where they cleared the 24h grace, every one of the 17 surviving
fix/issue-*branches would have been listed, all already landed
with their issues closed. A 17-row table of non-problems on a
tripwire's first real fire is how a tripwire gets ignored forever.A
fix/issue-<N>branch whose issue N is CLOSED is now treated as
dispositioned and kept out of the table — closing the issue is a
human act saying the work was dealt with, which is exactly the signal
this workflow exists to detect the absence of. They are still counted
in a footer naming them as safe to delete, because they are real
clutter, just not lost work; silence would trade one failure mode for
another. The rule applies only to that naming convention, and any
lookup failure falls through to REPORTING the branch — the tripwire
errs toward surfacing work, never toward hiding it.Deliberately not done: loosening the 24h grace (it exists so ordinary
in-session branches don't spam the issue), and closing issue #282 —
that one is a live, unfixed P1 gap on main, and closing it would hide
a real defect. -
The projector's replay-window read ran on the un-timeout'd root
context (wave-D RD-08). Every otherp.storecall in the package
runs undercycleCtx; this one passedRun's ctx straight through,
so a query blocked behind a lock wait parked the watcher goroutine
withstellarindex_projector_replay_window_activeholding whatever
it last published — a stale1keeps suppressing the lag ticket for
a source nobody is replaying, and the suppression's whole
justification is that it stays narrow. Not unbounded even before
(OpenBackgroundSETsstatement_timeouton every connection, 30m
by default), but 30 minutes of a wrongly-suppressing gauge is not a
bound worth relying on when a local one costs two lines. Bounded by
PerSourceTimeout(60s) — deliberately NOT a budget matched to the
refresh interval, which would trip on ordinary DB slowness, zero the
gauge mid-replay and re-armprojector_lag_highfor the whole
catch-up, reinstating the ticket storm #325 removed. -
RD-09 (replay-window upper bound on a widened dirty row). The
arithmetic reproduces, but the scenario is a recorded, ratified
decision —docs/operations/runbooks/projector-replay.mddocuments
it with its operator remedy. Every proposed remedy is worse than the
gap: narrowing the range union, or refusing to record while a window
is pending, trades a tighter alert suppression for a data-integrity
regression on the verifier path — that union is what closed the
2026-07-31 carried-claim invalidation gap (19,366 over-projected
cctp rows), and compute-completeness's forced re-reconcile floor, the
table's primary consumer, depends on it. What was genuinely wrong was
a code comment: it glossed the bound as the pre-rewind position
unconditionally, which holds only for an un-widened row. Corrected,
along with a note on why the union must not be narrowed. -
/v1/assetsacceptedorder_byand never read it, so the home
page's headline ranking was computed over the wrong ten assets
(wave-D RD-02). The explorer requested
?limit=10&order_by=volume_24h_usd_descunder the caption "Ranked by
trailing-24h trading volume across every venue we ingest"; the
handler builtListAssetsOptionswith noOrder, so it was served
the top ten by all-time observation count and re-sorted just
those ten client-side. An asset that traded $2M in the last 24h but
has a modest lifetime count could not enter the candidate set at all,
while a dormant high-lifetime-count asset held a slot and rendered as
a dash.order_by=TOTAL_GARBAGEreturned 200.This was missing WIRING, not a missing feature: the storage layer has
supportedAssetsOrderVolume24hUSDDescthe whole time — its own
ORDER BYbranch, keyset cursor args, cursor predicate and rank-tier
expression, and the unified path already passes it. The handler now
parsesorder_by, threads it into the query and both cursor
calls (the two orders encode different keyset keys, so encoding
under the wrong one skips or repeats rows rather than erroring), and
400s on an unrecognised value the way/v1/marketsalways has.Combining
order_bywithasset_classnow 400s rather than being
silently ignored: those listings rank on their own fixed scheme with
a cursor encoding that scheme's keys. Of the explorer's four callers
only the home page sendsorder_by, and it sends noasset_class.The explorer's client-side re-sort is removed in the same change —
with the server ordering correctly it stopped being a no-op and
became actively wrong, because the API ranks on a concentration-
ADJUSTED volume (so wash / operational assets don't sit atop the
directory) while the payload'svolume_24h_usdis the RAW figure.
Re-sorting the page by the raw column promotes exactly the assets the
server demoted. Native XLM, which/v1/assetsdoes not return, is
now spliced into the server's order instead of triggering a re-rank
of everything. -
pkg/clientSDK:Retry-Afterno longer yields a NEGATIVE
back-off.parseRetryAftermultiplied the header's delta-seconds
into atime.Duration(int64 NANOSECONDS) with no range check, so
any value above ~292 years wrapped —Retry-After: 9223372036854775807
produced-1s, and a caller sleeping on it retried IMMEDIATELY,
the exact opposite of the back-off requested. Out-of-range values
now return0, the field's already-documented absent/unparseable
sentinel. Deliberately not a clamp:APIError.RetryAfteris a
SemVer-stable reporting field, and clamping would make it
misreport the wire (wave-D F-SDK-06). -
pkg/clientSDK: an oversized response body now errors instead
of surfacing as a bogus JSON decode failure. The 16 MiB read cap
usedio.LimitReaderat exactly the limit, andLimitReader
returns(n, nil)AT its limit — so a truncated body was
indistinguishable from a complete one and got parsed, reporting a
confusing error about the payload rather than the truth. Now reads
cap+1and errors naming the cap (wave-D F-SDK-08). -
Docs:
pkg/clientquery-parameter godoc said out-of-range
limit/window_secondsvalues are "clamped". They are
REJECTED with a 400. Read in thestd::clampsense the old wording
told a caller their out-of-range value would be quietly honoured at
the boundary — on a pricing surface, the difference between a VWAP
over a window they never asked for and a loud error. The
genuinely-saturating ADR-0015 closed-bucket adjustment keeps the
word, anddocs/architecture/lexicon.mdnow fixes both meanings so
the two do not re-blur. ADR-0018's copy of the old wording is left
alone — ADRs are immutable (wave-D F-SDK-09). -
Docs: the documented
pkg/*SemVer release mechanism was
inert.semver-policy.mdandrelease-process.mdinstructed
cuttingpkg/client/vX.Y.Ztags, but this repo is a single Go
module (ADR-0005), so such a tag versions nothing — the proxy has
no nested module andgo get …/pkg/client@v0.2.0fails outright.
Both documents now state thatpkg/clientships on the root clock,
that apkg/*break bumps the root minor and MUST be named in the
CHANGELOG (the consumer's only notice), and why adding
pkg/client/go.modwould be a live break for everyone currently
pinned on the root module rather than a fix (wave-D F-SDK-05,
#361 item 8). -
CV-2 (oracle reconcile netting). The finding reads an unwired
vintageBoundaryfield as a live hole; the history is the reverse.
It shipped and changed behaviour, then was retired because its only
subject was upgraded to a stronger position — strict per-ledger with
no netting, proven over 12.5M ledgers with zero mismatches. The
remaining oracle netting is a recorded, deliberately-deferred decision
(F6 / C2-16) with a register entry, a rationale and a superseding
design doc. The implied fix — setting a boundary on the four oracle
sources — is not actionable: no cutover ledger for the legacy backfill
exists anywhere in the repo, and guessing one too low re-opens the
false-positive class the deferral exists to avoid. -
CV-4 (recognition claim unfalsifiable for the sep41 sources). The
mechanical observation is right, but the remedy would be actively
harmful. Lifting the topic exclusion cannot create a falsifiable
check — every watched contract maps back to a sep41 source, so the
guaranteed gaps would pinrecognition_ok=falseand therefore
complete=falsepermanently, while every non-watched SAC's shapes
flood the unattributed bucket. The exclusion's justification is also a
structural truth about the dispatcher this function builds, not the
stale deployment observation the finding assumes. -
Two migration headers cite a different migration than the file they
are in (wave-D CV-7).0125_projection_dirty_windows.up.sqlopens
with0124 upand0096_create_blend_emitter_events.up.sqlwith
0095 up— both real but unrelated migrations, so a reader following
the reference lands somewhere else. A Go comment describing the
dirty-window table citedmigration 0124for the same reason; that
one is corrected.The two migration headers are not, and cannot be: applied
migrations are immutable, and even a comment-only edit changes the
checksum. Immutability is the stronger rule — a migration whose bytes
can change is one whose applied-ness cannot be proven — so the drift
is recorded rather than fixed, and a new lint stops the set growing by
failing any new migration whose header cites the wrong number,
before it ships and freezes.A third citation, in
freeze_events.go, says 0124 and is correct
(0124 really isfreeze_reason_other) — a blanket find-and-replace
would have broken it. The check is therefore deliberately narrow: a
file disagreeing with its own filename, not whether everymigration NNNNmention in the tree points at the right subject.
Added
-
Tests for the CS-017 price-freshness seams (wave-D PFR-04).
storePriceReader'snow func() time.TimeandvwapFreshness
fields exist only to be injected by a test, and nothing did — so the
15-minute staleness rule had no enforcement beyond runtime. Now
pinned: the default window and why it is 15 minutes, the zero-value
sentinel (an explicit0must mean "unset", not "never stale"), the
injected clock, and the staleness boundary mirroringLatestPrice's
real expression including its measure-from-CLOSE+1mand its
lowConfidenceshort-circuit.PFR-04's failure scenario does not survive and was not acted on:
the dormant long tail cannot resume being served a months-old bucket,
because the substance gate runs twelve lines earlier, its window is
trailing-24h, and a dormant pair fails its first comparison — the
read returnsErrPriceWithheldand the staleness expression is never
evaluated. The end-to-end read also remains outside unit-test reach
(storePriceReader.sis a concrete*timescale.Storewith no
injectable constructor); that belongs to the integration harness. -
Composite-reference corroboration of the phase-2 freeze for
structurally single-venue targets (product decision, Ash
2026-08-29; design doc §10.1 amendment). For an allow-listed target
([aggregate.composite_reference], default ON for
crypto:XLM/fiat:GBP+crypto:XLM/fiat:EUR) whose bucket is
single-venue, the aggregator rebuilds the target's triangulation
chain on the CURRENT bucket — this tick's crypto/USD leg publish
(≥min_leg_sourcesreal venues, default 2) × a fresh FX snap
(≤fx_max_age_hours, default 76, FX source class only, never an
oracle) — and reads it against the fresh direct VWAP: agreement
withintolerance_bps(default 75) means the move is market-wide
and the 3-signal-AND fire is suppressed (corroboration_basis= composite); disagreement or an unavailable reference freezes
exactly as before, the reason string naming why
(corroboration_basis=venue composite_unavailable: leg_sources=1 composite_leg_sources={…}). The same sample feeds the confidence
factor (triangulation_checked) and the mid-hold release lens, so
a corroborated genuine move can also release. Hard invariants: the
composite never enters VWAP and never raisessource_count/
effectiveSourceCount; targets with ≥ 2 real venues are
byte-identical to before. New:composite_meta.corroboration_basiscomposite_leg_sources, gauges
stellarindex_aggregator_composite_corroboration{pair,window,verdict}
/..._composite_reference_leg_sources{pair,window,leg}, counter
..._composite_freeze_suppressed_total. Never a prior tick's sample
(the rejected 95da898d mechanism). Tests:
TestCompositeReference_*(manipulation control with the mechanism
ON, market-wide mirror, stale-FX / thin-leg / oracle-FX fail-closed,
multi-venue differential, exact-Rat tolerance boundary, refresh
order) plus the unchanged
TestRouterFreeze_TwoRoutesSuppressSingleSourceFreeze3-tick control
(#246). Verifier advisories (same day): A1 leg-dispersion guard —
every venue's own bucket VWAP on the crypto/USD leg must be within
leg_dispersion_bps(default =tolerance_bps) of the leg VWAP,
elsecomposite_unavailable: leg_dispersion=…(two venues only count
as two when they agree; gauge
stellarindex_aggregator_composite_reference_leg_dispersion_bps);
A2 the mid-hold release lens for a resolved reference uses a
dedicatedrelease_band_pct(default 2.0), not the shared 5 %
cross-oracle band — a held +4 % venue-specific offset no longer
auto-releases (TestCompositeReference_ReleaseBandHoldsVenueOffset,
…_LegDispersionCannotCorroborate,…_LegDispersionBoundary,
TestLegDispersion_MeasuresWorstVenue). A3/A4: config-bound tests
(TestValidate_CompositeReferenceBounds) and the guard fails CLOSED
when a venue VWAP cannot be computed (leg_dispersion=uncomputable,
TestCompositeReference_UncomputableDispersionFailsClosed).
-
Rolling ZFS snapshots of the ClickHouse lake + Postgres on r1
(decision 2026-08-29).scripts/ops/zfs-snapshot.sh(installed by
the archival-node role, new tagzfs-snapshots,zfs-snapshot.timer
daily 01:45 UTC) takesauto-YYYYMMDD-HHMMsnapshots of
data/clickhouse(3 d retention) anddata/postgres(7 d) — the
minutes-scale answer to a logical fault (bad migration,DROP, bad
re-derive) alongside pgBackRest's hours-scale PITR. Hard min-free
guard (zfs_snapshot_min_free_bytes, default 2 TiB): below it the
job prunes its oldestauto-*snapshots (never a dataset's newest,
never any non-auto-*name) and, if still below, skips the snapshot
and reportsstellarindex_zfs_snapshot_guard_skipped=1. Textfile
gauges (stellarindex_zfs_pool_free_bytes,
stellarindex_zfs_snapshot_{latest_unix,count,used_bytes}), alerts
in both rule trees (zfs-snapshots.yml: pool free < 2.5 TiB ticket
/ < 1.5 TiB page, snapshot > 36 h stale) with promtool tests, runbook
docs/operations/runbooks/zfs-snapshots.md(honest crash-consistent
semantics for ClickHouse and Postgres, clone-and-copy / rollback
procedures, vs pgBackRest PITR), and
scripts/ops/zfs-snapshot-now.sh <dataset> [--keep <label>]for the
fresh-snapshot precondition of the ClickHouse destructive-DDL
runbook. The guard is fail-closed: unreadablezpoolfree space
(command failure / non-number) aborts the run before any destroy or
snapshot, exits non-zero and emits
stellarindex_zfs_snapshot_pool_free_unreadable=1(own ticket).
Invariants pinned red-first byscripts/ci/zfs-snapshot-test.sh
against a stubbedzfs, including the destroy choke point directly. -
No-orphan-work contract + daily
orphan-branchestripwire. On
2026-08-27fix/priceless-structural-unpriceablewas pushed with no
PR and no backlog line; on 2026-08-28 a different agent re-diagnosed
stellarindex_assets_popular_pricelessfrom scratch and fixed it
differently (#254), and the orphan (plus a postmortem branch, now
#255) surfaced only via a manual branch audit. The contract is stated
once in AGENTS.md (push ⇒ PR same session; prior-art check via
gh pr list --state all --search,git branch -r, backlog + runbook
grep; PR names the alert and root cause vs symptom; supersede by
closing with a comment) and cross-referenced from CONTRIBUTING.md and
CLAUDE.md. The PR template gains Alert / finding and Prior
art fields..github/workflows/orphan-branches.yml(daily +
workflow_dispatch,contents:read/pull-requests:read/
issues:write) lists every remote branch other thanmain/
old-*/archive*with no open-or-merged PR and a last commit24h old, and opens/updates a single "Orphan branches (no PR)" issue
(closes it when the list is empty). -
stellarindex_ingest_gap_detector_silentthird clause (both rule
trees). The alert'sabsent_over_time(runs_total[15m])clause is
satisfied by theoutcome="error"counter, and a target that has never
once succeeded in a process life emits nolast_success_unixstamp to
age — so a scan failing every cycle (the 2026-08-28 r1
soroban_eventsstatement_timeout loop, found verifying #258) fired
nothing. The rule now also fires when the target's error counter is
present now and 8h ago with no last-success stamp seen in 8h. First
promtool unit tests for the alert
(deploy/monitoring/rule-tests/ingestion_test.yml): fresh stamp
silent, stale stamp fires, never-succeeded fires, stamp-within-8h
suppresses, aggregator-absent fires. -
Ops scripts honour a ClickHouse ops user
(scripts/ops/ch-ops-user-test.sh).ch-live-catchup.sh,
ch-supply-flows-seed.sh,d2-ordinal-reproject.sh,
d3-lecur-v2-rebuild.shandch-backfill-monitor.shran
clickhouse-clientas the default user with no way to supply
credentials. They now honourSTELLARINDEX_CLICKHOUSE_OPS_USER/
STELLARINDEX_CLICKHOUSE_OPS_PASSWORD(e.g. from
/etc/default/stellarindex-ops), handed to the client through its
CLICKHOUSE_USER/CLICKHOUSE_PASSWORDenvironment — never argv,
whichpsand the journal would show. The monitor resolves them on
the HOST side of its ssh (newOPS_ENV, default
/etc/default/stellarindex-ops) for the same reason. Unset ⇒
byte-identical invocations; the new stub-backed test pins both the
credential hand-off and the unchanged argv per script, and runs from
scripts/dev/verify.sh. -
AWS Public Blockchain dataset drift monitor (audit 2026-08-29,
backup-restore-6). r1's galexie-archive was trimmed below ledger
49,984,000 on 2026-07-26, so the second raw-LCM archive ADR-0043
relies on is the third-partyaws-public-blockchainpubnet dataset
— and nothing watched it..github/workflows/public-dataset-check.yml
(weekly + dispatch, no credentials,--no-sign-request, first-party
actions only) now asserts contiguous 64,000-ledger coverage from
genesis to ≥ tip − 2 partitions, theHEX--start-endnaming, an
unchanged.config.jsonmanifest and the trimmed range
[64000, 49983999]fully present; drift opens/updates ONE "AWS
Public Blockchain dataset drift" issue (auto-closed when intact) and
never fails the scheduled run red. Decision core
scripts/ci/check-public-dataset.sh, fixture-tested inci
(check-public-dataset-test.sh: gap inside/above the trimmed range,
misnamed partition, manifest change, stalled publication all RED).
Changed
-
The two frozen planning inventories are retired, and
verify-launch-readycan no longer certify a retired document
(#321).docs/architecture/launch-readiness-backlog.mdhad gained
zero rows since 2026-05-13 and contained none of the actual v1 gate
(W6.1 paging, W6.3 rotations, W4 backups, the W8 correctness
backlog, ToS/Privacy #237), yet every L1–L5 row still carried its
last-written ✅ — so the weeklylaunch-readiness.ymlworkflow
republished "✓ Engineering surface ready" over a document that had
stopped tracking reality, and got more confident the staler it got.
The doc is now formally superseded bydocs/operations/v1-launch-plan.md
(frontmatterstatus: superseded+ a banner mapping its still-open
rows: L4.14–L4.17 + L5.8 → W9 gated on D2, L5.6 → W6.2, L6.4 → §2.8,
L6.6/L6.7 → W6.7),.github/workflows/launch-readiness.ymlis
deleted, andscripts/ci/verify-launch-readynow reads the
frontmatter and emits no verdict at all (new exit code 3) for any
document declaring itself superseded/retired — a retired doc's rows
are history, and neither a green nor a red computed from them means
anything. The prior 2026-07-24 staleness banner was itself wrong
twice over (it claimed the gate was unwired the day after it was
wired in068ec709, and named a "current source of truth" that was
superseded on 2026-07-27); both corrections are recorded in the new
banner.docs/operations/open-fixes-inventory-2026-08-08.mdis
superseded on the same terms: 24 of its 35 rows were done and never
struck — rows 1 and 19 closed on the day it was compiled
(d1cd18ac,a1c5c2e5), rows 2 and 5 two days later (f75ab4b2,
ef278218) — and its genuinely-open threads are carried into the
launch plan's §5. The public company page's "roadmap that gets us to
v1" link now points atv1-launch-plan.mdinstead of the retired
backlog. Tests:TestRealBacklog_IsRetired,
TestVerdictLine_RetiredDocNeverCertifiesReady,
TestSupersession_ReadsFrontmatterOnly, and a company-page case
pinning the roadmap href (red-proof: the pre-fix binary run against
the now-retired doc still printed "✓ Engineering surface ready
(subset gate)" and exited 0). -
ADR-0043 §2 amended (2026-08-29): "two independent raw-LCM
archives" now explicitly = our recent range + the AWS Public
Blockchain dataset; dependency accepted and monitored rather than
duplicated, with the one-time cross-region copy (≈ $80 + $3–4/mo)
recorded as the not-taken option.off-site-backup-plan.mdstatus
carries the same note. -
Public status page shows backup freshness (Ash, 2026-08-29). New
read-onlyGET /v1/diagnostics/backups(experimental) reports the
pgBackRest last full / diff / WAL-archive age, the per-repository
newest backup (repo 1 on-host, repo 2 encrypted S3 off-site), the
monthly restore drill's last run + pass/fail, and the ClickHouse
schema+state snapshot age — each with afreshnessverdict
(ok/stale/unknown) against SLOs the API echoes inslo
(full ≤ 8 d, diff ≤ 36 h, WAL ≤ 15 m, off-site ≤ 8 d, drill ≤ 35 d,
snapshot ≤ 36 h). Source of truth is Prometheus — the same
pgbackrest_exporter / node_exporter textfile series the alert rules
read; the API never shells out to pgbackrest. Every timestamp is
nullable and an absent series isnull+unknown, never a fresh
zero;source_statuscarries the document's trust tri-state and
flags.stalemirrors the roll-up. Cached 60 s, no secrets or paths;
503 where noapi.prometheus_urlis configured. The explorer
/statuspage mounts a Backups panel (BackupsPanel.tsx,
mainnet only) that renders green within SLO, red with the real age
past it, grey "no data" for absent sources, and a "verdicts not
trustworthy" marker when the API's Prometheus reads failed — ages
come from the API'sage_seconds, never the browser clock. Reserved
nulls (documented in the spec): reporetention, drill
restored_backup_ts/duration_s,zfs_snapshot_latest_ts,
replica_lag_s— no producer exports them yet. Tests:
internal/api/v1/diagnostics_backups_test.go,
web/explorer/src/app/status/BackupsPanel.test.tsx. -
stellarindex_backup_offsite_stale(P3, both rule trees). The
existing backup alerts readpgbackrest_backup_since_last_completion_seconds,
which the exporter computes ACROSS repos — a host whose on-host repo1
is fresh while every repo2 (S3) write fails stayed green and the one
copy that survives host loss aged out silently. The new rule fires
per UP exporter instance with nopgbackrest_backup_info{repo_key="2"}
series younger than 8 d (x unless x offset 8d— a new backup is a
new series), which also covers repo2 never written. promtool tests
indeploy/monitoring/rule-tests/backup-offsite_test.yml(red-proof:
wideningrepo_keyto all repos makes the repo1-fresh/repo2-stale
case stop firing); runbookrunbooks/backup-offsite-stale.md. -
/v1/assetsnow rejects a malformed catalogue cursor instead of
silently serving page 1 (wave-D KP-4).catalogue:abc,
catalogue:-7and anAtoi-overflow were swallowed and treated as
"no cursor", while every sibling paginated surface 400s on the same
input. This is a wire-behaviour change (200 → 400), though unreachable
through any shipped client: the explorer clampslimitto
{50,100,200,500} and acatalogue:cursor is only emitted below
limit 11. -
/v1/assetsdeclaredlimittwice (wave-D KP-5) — an inline
parameter with no default, alongside$ref: Limitwhich defaults to
100. Generators pick one arbitrarily, so the rendered reference, the
Postman collection and the explorer's generated types could each
disagree about the same field. The inline copy is removed and its one
unique fact (page 1 fills from the classic stream when the catalogue
is shorter than the limit) folded into the operation description; all
three spec-derived artifacts are regenerated.Spectral does flag this, as
operation-parametersat severity
warn — CI simply runs the action at its default
--fail-severity=error, so it never failed the build.lint-docsnow
enforces resolved-parameter uniqueness as a hard gate, which avoids
re-tuning Spectral's global severity floor and lighting up unrelated
warnings. -
Follow-ups from the adversarial review of the wave-D merges
(2026-08-31 sweep — the merges were self-reviewed, so this pass
existed to catch what that misses; it did).stellarindex_oracle_stale's join usedon (source), which discards
job/instancefrom the match. Two scrape targets exporting the same
sourcemake the right side non-unique and the rule fails evaluation
outright — "found duplicate series for the match group … many-to-many
matching not allowed" — which is the pre-fix silence plus noise. The
prometheus job template buildsstellarindex_indexerby looping its
host group, so a second indexer host (R2, ADR-0004/0016) produces
exactly that shape. Nowignoring(asset) group_left(), with a
two-target regression case.The price-withholding seam guard advertised that "a new read seam
cannot forget it" while enumerating two hard-coded seam names — so
a brand-new ungated reader passed it silently, and the property it
claimed did not exist. It now derives its subject set by finding every
method that calls a closed-VWAP store read (a reader must call one to
serve a price at all), with two documented exemptions and a fatal on an
empty subject set. Two sibling guards carried the same overclaim: the
keyset-ordering test now enumerates the orderings, and the explorer
truncation guard's scope is corrected in its comment rather than
widened — proving that class repo-wide needs dataflow analysis, and a
widened regex flags correct code.withholdPriceSeriesWhenUnpricedalso nullsathfor declared-peg
assets, not only scam-flagged ones. That is intended —GetAssetATH
reads the asset's own USD-quoted market, which for a declared-peg
asset is the dust market the substance gate refused, while the
published headline comes from the peg — but it shipped unpinned and
undescribed. Now pinned by a test that says why.The runbook write-gate lint matched flagset names with
[a-z0-9-]+,
which cannot match a space, so five two-word write-gated
subcommands (supply snapshot,supply seed-observations,
supply seed-sac-balances,supply seed-claimable-balances,
supply seed-sep41-genesis) were silently absent from the gated set: a
runbook telling a responder to run one without-writeproduced no
finding, and the command reportserrors=0having written nothing.
Widening it surfaced a false positive on a deliberate-dry-run
preview, so explicit-dry-runis now exempt — flagging those would
train responders to ignore the check. -
~20 systemd oneshot timers had no failure alert, including the sole
writer of the table the scam-pricing gate reads (wave-D LID-6).
directory-syncpopulatesaccount_directory, which the gate
consults on every aggregated price serve. It isType=oneshotwith no
OnFailureand emits no metric — so if it stopped, the table froze at
its last good snapshot, a newly-flagged scam issuer was never learned,
and the gate kept serving. That reproduces the incident the gate was
built to stop, with the gate present, correct, and reading stale
input.Naming units individually is how the gap opened, so
stellarindex_systemd_unit_failedinverts it: every unit is covered
unless it has a dedicated alert with better triage. The five
exclusions live inscripts/ci/unit-failed-dedicated.baseline, and a
self-test asserts each is genuinely named in a rule file — an
exclusion cannot become a silent suppression, and an empty baseline
fails rather than passing vacuously.The runbook leads with the property that makes these failures hard to
spot: these units write data something else reads and then trusts,
so a failed sync surfaces as a consumer serving stale data
confidently, elsewhere, possibly days later — not as an error. -
Corrected the
/v1/price/batchfan-out comment (wave-D
UNAUTH-DOS-2). It claimed 16-wide parallelism stays "well inside the
DB connection pool's headroom even with several batches in flight",
which does not hold for a 1000-id POST batch. The constant itself is
unchanged and should not be lowered as a throughput control: narrowing
it removes zero database work while lengthening how long each request
holds its connections, re-creating the regression it was raised to
fix. The bound that matters is the rate limiter, and charging it per
id rather than per request is the sound way to close the
amplification. -
An unauthenticated client could leave unbounded price-tip compute
loops running (wave-D UNAUTH-DOS-1). The SSE caps count
connections, but a/v1/price/tip/streamconnection also mints a
detached producer: its context comes fromcontext.Background(),
it outlives the request by design, and it survives the connection's
release for a 30-second linger. So opening and immediately aborting
streams in a loop left a growing set of compute loops, each polling
the database on its own ticker, with no connection left for the
connection cap to see. The Hub's topic reaper could not shed them
either — it evicts only subscriber-less topics, and a live producer
recreates its topic every window.Cheap to drive, because the producer key includes a client-chosen
window_secondsin[1,60]: the key space is pairs × 60, so an
attacker needs no distinct assets at all. The regression test
enumerates exactly that, and pre-fix left 32 producers running from a
single pair.Distinct producers are now capped (default 512,
SetMaxTipProducers
to tune, negative to disable). Two deliberate properties: a pair that
already has a producer still admits new subscribers at the ceiling
— those cost nothing extra to serve, and refusing them would penalise
a popular pair's own audience — and a refused stream returns
503 + Retry-Afterrather than falling through to the legacy
per-connection loop, which is the unbounded compute the ceiling
exists to prevent. -
The
/assets"#" column restarted at 1 on every cursor page
(wave-D EXR-06), so the 101st asset was labelled#1under a header
that reads as a global rank. The counter is per-page, and cursor
pagination keeps only the opaque cursor in the URL — there is no page
depth to recover. The rank is now shown only on the unpaginated first
page.Deliberately suppression rather than arithmetic: deriving
depth * limit + iwould print a different wrong number, because
suppressCatalogueTwinsandfoldAliasTwinsdrop rows after the
query so pages under-fill (measured 81/96/99/96 atlimit=100). A
rank the data cannot back is better omitted than guessed.The test mock hardcoded an empty query string, so every existing case
ran on page 1 — which is why nothing caught this. It is now settable,
and a case pins the paged behaviour. -
Every asset link pointed at the bare CODE, so a link could resolve
to a different issuer's asset than the row clicked (wave-D EXR-02).
assetSlugtruncated a canonicalCODE-GISSUER…id at the dash, and
/assets/USDCis shared by every USDC-alike — so a link built from a
scam issuer's row could land on the legitimate asset's page, or the
reverse.AssetLinkandAssetTextnow link the full canonical id,
as/markets/[pair]already did for the same reason (AM-09).The docstring justifying the truncation ("long-form ids are NOT in
generateStaticParams … so linking to them hard-404s") had outlived its
constraint: canonicalasset_idroutes are emitted for exactly the
same asset set as the short slugs, and anything outside that set falls
to the client shell under both spellings — so the canonical form never
links worse and always links precisely.Display labels are unchanged:
shortAssetTextstays short, because
these are dense analytics rows and a 56-char id would blow out every
cell and chart legend.AssetTextcarries the canonical id intitle
instead, so the issuer is recoverable on hover without spending row
width.Guarded three ways: the repo-walk pack gains a rule against building
an/assets/href from a code-truncated id, andassetSlug— which
decides where every asset reference in the explorer points, and had no
behavioural test at all — now has one, including the property that two
issuers sharing a code get different links. All proven red pre-fix. -
A scam-flagged asset outside the pre-rendered top 500 showed no scam
warning at all (wave-D EXR-01)./assets/[slug]has two render
paths: the build-time pre-render for the top 500, and a client shell
(AssetPathView) for everything else. Both fetch the same
/v1/assets/{id}payload, carrying the sameissuer_directory_tags
andissuer_scam_reason— but the banner ("Do not trust this asset,
establish trustlines, or execute the prices below…") was inlined in
the pre-rendered page only, and the shell simply ignored those fields.
The path serving the long tail, which is where a scam token actually
sits, was the one rendering without the warning. The banner is now one
sharedAssetScamCalloutmounted by both paths.Also closes EXR-05 / part of #335:
AssetSwap'sTokenIcon
rendered an issuer-controlled image URL without the SEC-10
isSafePublicImageUrlhost check that the other two<img>sites
apply.Both were the same failure mode — a trust-critical rendering
obligation enforced by convention at some call sites and silently
absent at one — so both are now pinned by a new guard pack,
src/lib/trust-surface-guards.test.ts, which enumerates the call
sites fromsrc/at test time. A fourth<img>site or a third asset
view fails on the day it is written. Both guards were verified red
against the pre-fix tree, each naming its offending file. -
One extra path segment defeated the price-withholding gates
(wave-D MSP-01)./v1/pricecorrectly returned
errors/price-withheldfor a directory-flagged scam issuer or a
market too thin to aggregate — while/v1/price/atand
/v1/price/changespublished the identical closed-bucket VWAP,
becausestorePriceAtReadercarried neither gate. Every
price-serving read seam now routes through a single chokepoint,
priceWithheld(), which is the only place in the binary where either
gate is spelled.The same change closes MSP-07: the last-trade arm of
LatestPriceconsulted the thin-market gate but not the scam gate,
so an operator settingpricing_guard.disable_substance_gate=trueto
diagnose a pricing-coverage complaint silently also un-withheld every
directory-flagged issuer's last trade — reversing a separate,
owner-level trust decision they never touched.Two AST guards pin this, both proven red against the pre-fix state:
one enumerates the price-serving seams and fails when a seam does not
route through the chokepoint; the other fails on any
substance.Allowed/scam.Withheldcall outside it. The second is
the one that catches MSP-07 — a seam can call the chokepoint on one
arm and still hand-roll half the decision on another.Deliberately unchanged:
/v1/twap,/v1/vwap,/v1/ohlcand
/v1/chartremain ungated. The raw-trade surfaces are documented as
deliberately visible (pricingguard/scam.go), and which quote sets
constitute a "price claim" is the open scope question in #366. -
GET /v1/assetssilently truncated its own pagination on any tie
(wave-D KP-1 / RD-01 — the same bug found twice, independently). The
keyset cursor predicate for the default observation-count ordering
compared(observation_count, asset_id) < ($n, $m)— a SQL row
constructor, which compares every element in the same direction —
against anORDER BY observation_count DESC, ca.asset_id ASC, which is
mixed-direction. On a tie inobservation_countthe tie-break half
therefore read asasset_id < $m, re-selecting rows the walk had
already served while skipping the ones it had not. A client paging the
full asset list received some assets twice, never received others, and
was then toldhas_more: falseas though the walk had completed. Ties
are the norm in the long tail, where most assets share a small
observation count. The volume ordering always spelled the comparison
out correctly; only this arm was wrong.The existing pagination regression test could not catch it: its fixture
gives every row a distinct observation count and a distinct volume,
so the walk never crossed a tie. It now also seeds rows that tie on
both sort keys. A source-derived unit invariant additionally asserts
that any ordering whoseORDER BYbreaks ties onasset_id ASC
resumes withasset_id > $nand uses no row constructor — so a third
ordering added later is covered on the day it is written, rather than
when someone notices missing rows. -
make verifywas red onmain, and CI could not see it. The
ClickHouse ops-credential contract (scripts/ops/ch-ops-user-test.sh)
has been failing since #286 gaved2-ordinal-reproject.sha
destructive-DDL acknowledgement (D2_FORCE_DROP) that exits before
the script's first query — so the harness never got a stubbed
clickhouse-clientinvocation to assert on, and reported
clickhouse-client was never invoked. The canonical pre-push gate has
therefore been failing for every contributor who ran it. The harness
now acknowledges the guard (safe: the stub fails the first, read-only,
query, so the script bails long before anyREPLACE PARTITIONor
DROP, andCH_FLAGS_DIRis redirected away from the real flags
directory). Root cause of the silence, now also fixed: this contract
ran only inscripts/dev/verify.shand in no CI job, so it shipped
red — it is now wired into the ops self-test step alongside the
restore-drill contracts. 15/15 passing. -
Runbook re-verification wave K: eight alert runbooks re-derived
against HEAD (7 broken, 1 stale), plus a lint so the worst class
cannot recur.ledgerstream-tier-both-missing.md— a P1 page —
told responders to "pull the missing range from R2 or R3's mirror"
withrehydrate-galexie-archive … --source vultr. There is no peer
selector and never has been: the command only ever reads the
CONFIGURED cold tier (storage.s3_cold_*), so during an AWS Open
Data outage — the exact scenario the step sat under — it cannot
route around anything. Its-writefail-closed note was correct
(the sharedopsutilwrite gate makes dry run the default) and is
kept, now with the trap that motivated the re-verification spelled
out: a dry run buckets every not-in-hot path ascopiedWITHOUT
asking cold whether it holds the object, so a forgotten-write
logscopied=N missing_in_cold=0 errors=0and exits 0 — a
success-shaped report having rehydrated nothing. It also cited two
gauges that have never been registered.
postgres-ping-failing.mdstill documented the> 0.5/sthreshold
that was unreachable by 30× at the 60 s probe cadence (corrected to
> 0in both rule trees on 2026-08-04) and a
trade_inserts_total{outcome="error"}label that does not exist.
source-stopped.mddescribed ONE 30 m × 15 m alert; the shipped
shape is the F-1208 three-way split (high-volume 30 m/15 m,
low-volume DEX 24 h/30 m, daily publisher 30 h/1 h) all sharing one
runbook_url.external-poller-stale.mdblanket-claimed "30
minutes", misdescribing the 12 h ECB rule by 24×.
ingestion-duplicate-flood.mdstill saidON CONFLICT DO NOTHING
(INV-3 / migration 0109 made it a generation-guardedDO UPDATE, so
a corrective re-derive now reproduces the alert's exact signature)
and used-sources, which does not parse (-source, singular,
comma-separated).
decode-errors.mdasserted a pre-P23 operations+effects fallback
that has never existed, a pre-ADR-0035/0040 comet topic-only match,
and a redstone length-mismatch that refuses without recovery.
sev-status-page-update.mdstill pointed atweb/status/, now a
redirect-only stub — the live page is the explorer's
web/explorer/src/app/status/(CLAUDE.md's copy of the same drift
is fixed by #326, above).exporter-down.md
attributed r1's exporters to the redis-sentinel role, which r1 never
runs. Guard:lint-docs.sh§11's runbook metric-name check was
scoped tostellarindex_source_*only, so both phantom gauges above
were invisible to CI; it now covers every obs-owned namespace a
runbook cites (source/cursor/indexer/backfill/trade/postgres_ping)
and resolves histogram_bucket/_sum/_countchildren (ansible
inventory variables sharing the namespace are subtracted, derived
fromconfigs/ansible/, so a runbook naming
stellarindex_backfill_from_ledgercan't red CI as a false
positive). A second guard, §11b, fails CI on any
run-heavy-job.sh … stellarindex-ops <sub>invocation in a runbook
where<sub>registers the shared write gate but the command omits
-write— the wrapper is the COMMIT path, so a dry run under it is
always a bug. The gated-subcommand set is derived from the Go
source. Both checks verified red against the pre-fix runbook text. -
The P1 archive-divergence page can actually fire: verify-archive now
exports its mismatch counter through node_exporter (#282).
stellarindex_stellar_archive_divergence(severitypage) selects
stellarindex_verify_archive_mismatches_total, which the chain /
checkpoint walk increments — but the counter's only export path was the
opt-in-metrics-listenHTTP endpoint, which neither
verify-archive-tier-anor-tier-bpassed and
configs/prometheus/prometheus.r1.ymlhas no scrape job for (a one-shot
job is gone between scrapes anyway). The metric had no producer in the
deployed topology, so a genuine archive-correctness event opened a P3
ticket (stellarindex_verify_archive_unit_failed) and paged nobody; the
2026-06-11 F-1329 repoint had fixed the metric NAME but not the export
path. New-textfile-output PATHflag writes the counter into
node_exporter's textfile_collector dir, wired into both units (ansible
templates + thedeploy/systemdreference copies) with per-unit.prom
files. Three properties make it usable byincrease(): totals are
CUMULATIVE across runs (a clean run re-emits, never resets), all three
reasonvalues are ZERO-SEEDED on every run (a series that first appears
at 1 and stays flat yieldsincrease() == 0— the same F-0033 /
C4-038 "absence reads as health" trap as the gap-detector fix below),
and the series is labelled bytierrather thanchunk_idx(a per-run
worker slot with no cross-run meaning, and two units exposing an
identical label set through one node_exporter target is a duplicate-
metric scrape error). The rule's lookback also widened1h → 26h:
against a NIGHTLY producer a 1h window showed the step for one hour in
twenty-four, so a SEV-1 correctness page self-resolved before the
morning. Pinned bydeploy/monitoring/rule-tests/stellar_test.yml
(fires immediately AND is still firing 24h later — the assertion the 1h
window fails),verify_archive_textfile_test.go(seeding, accumulation,
tier isolation, atomic rename) andverify_archive_unit_wiring_test.go
(the deployed units must wire an export path — the Go↔systemd seam
lint-metric-refs.shcannot see). Runbook, alerts-catalog and metrics
reference corrected; the never-existent producer
scripts/ops/archive-cross-check.shis flagged as design-intent in
multi-region-topology.md. Requires an ansible apply
(--tags ops-jobs) on r1 — a binary-only deploy ships this dead.
Adversarial verification then found the apply procedure delivered only
HALF the fix and its confirm step read green anyway, so three further
corrections land with it: (1) the tier-b install/remove blocks in
14-stellarindex-services.ymlwere the only verify-archive tasks
withouttags: [ops-jobs](they sit in their own
verify_archive_tier_b_enabledconditional, added after the tag was
introduced), so the documented apply rendered tier-a's unit and
silently skipped tier-b's — leaving the CHECKPOINT tier, the
cross-archive anchor the page's own summary describes, permanently
unwired; they are tagged now and pinned by
TestVerifyArchiveUnits_ReachableUnderOpsJobsTag. (2) The runbook's
confirm is fail-CLOSED on a half-apply: it loops over both tiers and
exits non-zero naming the missing one, instead of showingtier="chain"
at 0 and deferring the other with "once tier-b has run". (3) The two
divergence checks that run OUTSIDE the per-chunk walk — cross-chunk
boundaries (stitchChunks, ~11 per 12-worker run) and the cross-run
resume seam (checkResumeFromHash) — returned their errors without
incrementing the counter, so a break landing on a chunk boundary still
paged nobody; both now increment under the samereasontaxonomy
(TestStitchChunks_BoundaryBreakIsPageable,
TestCheckResumeFromHash_MismatchIsPageable, which also pins that a
malformed-resume-from-hash— operator input, not divergence — must
NOT move a severity-page counter). The runbook gained a Known blind
spots section for what is still uncovered: Tier D / Tier E emit no
metric at all, and the first run on a host with no.promfile yet
publishes a series that appears at its final value, so a break found by
that very first run readsincrease() == 0until the next run — which
is why the apply procedure now primes both units by hand. -
The 7d chart column on
/assetsis back for the assets that matter
— and a withheld price no longer gets published as a picture of
itself. Rows 1–11 of the directory (XLM, USDC, PYUSD, EURC, AQUA,
yXLM, SHX, VELO, BLND, PHO, yUSDC) rendered—in the 7D CHART column
while the unverified long tail below them charted fine. Those eleven
are exactly the catalogue-projected rows, whose wireasset_idis the
catalogue SLUG (projectCatalogueRowsetsAssetID = vc.Slug), so the
listing askedGetAssetsPriceHistory7dBatchfor a series underxlm/
aqua— ids that can never match aprices_1mrow. The batch query
answered with itswant × daysskeleton: seven buckets, every price
null, indistinguishable on the wire from "this asset has never traded",
which is why it shipped unnoticed. The series now keys on the row's
Stellar twinasset_id— the SAME id itsprice_usdand
change_7d_pctalready come from (fillCatalogueStatsForPage,
fillGlobalPriceFromOnChain) — so the chart and the price can no
longer disagree about whether data exists.?include=sparkline7dis
also honoured on the default/v1/assetslisting, where it had been a
silent no-op (the parameter was wired only into the catalogue/classic
phases, so the issue's own repro returned a byte-identical response
with and without it). Two honesty rules go with it, in both directions:
a row with no published price gets no chart and is not even looked up —
the scam-issuer suppression and the thin-market substance gate both
leaveprice_usdnil before the attach runs — and on the asset DETAIL
payloadprice_history_24h/price_history_7dare dropped whenever
the headline price is withheld (measured on r1 2026-08-29: the flagged
JFKBANK2 and RIO details servedprice_usd: nullbeside 24 hourly and
7 daily priced points, and their listing rows drew a full sparkline
next to a—price cell; the last bucket of a price series IS the
number being withheld). New counter
stellarindex_api_sparkline7d_rows_total{result="served"|"empty"}plus
a once-per-request warn when every priced row on a page comes back
empty: a map hit from the batch reader is not evidence of data, and
nothing anywhere reported the dead column. (#355) -
r1's ZFS
datapool is raidz1 everywhere, and a lint keeps it that
way. The pool is SINGLE parity — live-verified 2026-07-17 and
corroborated by arithmetic that needs no host access (the ~16.8 TB
footprint measured that day cannot fit the ~13.85 TB two parity drives
would leave on these four devices) — but the 2026-07-17 correction only
ever reached the two rule trees and two runbooks.r1-deployment-state,
self-hosting,storage-considerations,multi-region-topology,
multi-region-cutover,r3-deployment-state,lcm-cache-tiering,
ADR-0016, ADR-0027 and the ansible per-region comment all still said
raidz2, i.e. promised an operator a second drive of failure margin that
does not exist and sized capacity plans off a usable figure ~4.5 TB too
low. Sharpest of them:configs/ansible/inventory/r1.example.ymlsaid
zfs_data_pool_type: "raidz2", so a rebuild from the codified inventory
would have laid down a pool too small for r1's own data. That value is
nowraidz1and is the machine-readable authorityscripts/ci/lint-docs.sh
§18 lints every r1-scoped file against (paragraph-level: naming another
raidz level is allowed only alongside the live one, so dated history
survives and bare contradictions do not). The role DEFAULT stays raidz2
— deliberately, it is the right shape for a fresh archival node — and
now says so. TheTODO(ash)inzfs-degraded.mdis closed with the
evidence rather than another ssh request. Dated decision records
(ADR-0016/0027, the superseded first-node runbook, the 2026-07-16 audit
assessment that inferred raidz2 from docs while stating it had no live
access) keep their text and carry inline corrections. (#289) -
Account transaction history no longer truncates itself: the keyset
merge dedupes the two arms BEFORE taking its LIMIT.
/v1/accounts/{g_strkey}/transactionsresolves its page keys from a
UNION ALLof the sourced (stellar.ops_by_source) and participant
(stellar.operation_participants) arms, and a tx the account SOURCED
that ALSO carries it as a non-source participant of one of its
operations is emitted by BOTH arms. The merge took itsLIMIT ?over
those still-duplicated rows and only the hydration pass deduped, so
every overlapping tx cost a page slot: the page came back SHORT while
older history remained, and the handler emitsnext_cursoronly on a
FULL page (the documented "absent on the last page" contract), so a
client's history walk stopped there with older transactions
unreached. Measured on the live-ClickHouse fixture (600 txs, page
size 7): the pre-fix query served 100 non-final short pages out of
101 — a walker stopped on page 1 having seen 6 of 600 txs — the fixed
query serves 86 full pages and 0 short ones.LIMIT 1 BY ledger_seq, tx_indexnow runs at the merge too, making the keyset exactly
min(limit, distinct keys older than the cursor); rows and their
order are unchanged (the integration walk keeps the pre-fix SQL as a
differential oracle over the whole history, and now requires it to
still produce a short page so the fullness assertion cannot go
vacuous). The sibling operations listing needs no such dedupe — its
arms are disjoint at op granularity because participants exclude
their op's own source — and a key cannot hydrate to nothing either,
sinceSink.Flushwrites transactions before operations and
participants. (#290) -
The
ops_batchClickHouse identity can no longer reach a live
daemon on adeploy/systemdself-host. The three reference units
(stellarindex-{indexer,aggregator,api}.service) source
/etc/default/stellarindex-ops— the indexer for its MinIO creds, the
API for its SEP-10 seed — and
docs/operations/clickhouse-ops-batch-profile.mdprescribes writing
STELLARINDEX_CLICKHOUSE_OPS_USER/_PASSWORDinto exactly that file.
Since #243 those two vars set the identity of every ClickHouse
connectioninternal/storage/clickhouseopens, so a self-hoster
following the doc demoted the LIVE ledger sink
(NewLiveSink→Open) and the aggregator's supply readers
(NewExplorerReader) to the lowest-priority batch tier — the precise
inverse of the 2026-08-28 r1 incident the profile exists to prevent.
The only guard was a unit-file comment (config-assertions.sh's third
leg checks/etc/default/stellarindex, which does not exist on such a
host). Each live-daemon unit now carries
UnsetEnvironment=STELLARINDEX_CLICKHOUSE_OPS_USER STELLARINDEX_CLICKHOUSE_OPS_PASSWORD, which systemd applies after
everyEnvironment=/EnvironmentFile=(systemd ≥ 235; the Ubuntu
22.04/24.04 targets ship 249/255), so the guarantee holds whatever the
operator puts in the file. The batch one-shots that share the file
(verify-archive-tier-*,ch-schema-*,restore-drill) deliberately
do NOT strip it.TestOpsBatchIdentityNeverReachesLiveDaemonsresolves
the environment each unit indeploy/systemd/and the archival-node
role would hand its process, feeds it toopsAuthFromand pins both
halves — CHdefaultfor the live daemons,ops_batchfor the batch
units — so neither direction can drift. Ansible-managed hosts were
never affected (their daemons read/etc/default/stellarindex) and
nothing about r1's rendered units changes. (#292) -
stellarindex_ingestion_duplicate_floodcan fire in its own target
scenario. The rule joinedrate(...{outcome="duplicate"}[10m]) > 0.5
withand on (source) rate(...{outcome="new"}[10m]) == 0, and an
andjoin needs the right-hand series to EXIST.
stellarindex_trade_insert_outcome_totalis call-site-seeded
(WithLabelValueson the trade-insert path) and itssourcelabel is
config-dependent, so internal/obs does not pre-seed it — a source
whose every insert since process start hit the conflict path never
creates theoutcome="new"child, the join matched nothing, and the
alert stayed silent in exactly the post-restart cursor-replay flood it
exists for. Nowunless on (source) rate(...{outcome="new"}[10m]) > 0,
which reads an absent child and a zero rate identically (the
absent-series idiom already used by theinsert_stalesibling). Both
rule trees. The promtool case that pinned the gap as a KNOWN GAP now
asserts the alert fires, with a companion guard proving a
below-threshold duplicate rate with the same absent child stays silent
(red on the pre-fix rule:got:[]). (#302) -
/v1/livez/lakesingle-flighted, and its 503 no longer publishes the
ClickHouse endpoint (#310, audit 2026-08-29). #266 gave the ADR-0050
lake-route LB probe readyz's infra exemptions — no auth, no anonymous
rate limit — but readyz's safety under those exemptions comes from its
single-flight cache, which this route never had: every anonymous
request ran a freshLakeTipLedgerquery against ClickHouse under a 5s
timeout, so unmetered concurrent probes amplified straight onto the
lake, worst exactly when the lake was already struggling. Concurrent
callers now share ONE ping round per second (livezLakeTTL, the same
budget as readyz;as_ofreports when the lake was actually pinged),
and the round runs on a detached context so one prober's disconnect
can't cancel it for everyone. The 503 body'sdata.detailis now a
fixed operator hint instead oferr.Error()— which on the real
checker is a dial error naming the ClickHouse host:port, served
unauthenticated during an outage; the underlying error goes to the
server log (once per round). OpenAPI 503 contract + the generated
Postman/TS mirrors updated to match. Tests:
TestLivezLake_SingleFlightSharesOnePingPerRound(25 probes → 1 ping;
pre-fix 25),TestLivezLake_UnreadyBodyDoesNotEchoPingError(body
scrubbed, log still carries it),TestLivezLake_RoundRefreshesAfterTTL
(a recovered lake is not pinned 503),TestLivezLake_AbsentLakeFailsClosed. -
A future-dated backup stamp no longer renders a green "fresh" row on
the status page.freshnessVerdict
(internal/api/v1/diagnostics_backups.go) clamped a negative age to
age_seconds: 0for display and then judged the SLO on the clamped
value —*age > slois false for any negative — so a forward-skewed
host clock or a corrupt future-dated pgBackRest label painted an
arbitrarily stale backup"ok"with a 0-second age, and the panel's
roll-up went fully green (its own comment already said "don't reward
it"). A stamp pastbackupClockSkewTolerance(1 min — these ages
cross clocks, so ordinary NTP divergence on a genuinely fresh item
still floors to 0) is now"unknown"carrying its RAW negative age
for diagnosis, which also dragsfreshness.overallandflags.stale
off all-clear. The Backups panel names that state ("stamp from the
future") instead of an ambiguous grey "no data", and its client-side
repositories caption stopped clamping the same future stamp up into
"0s ago" (Math.max(0, …)— the identical bug, one layer up).
Regression tests:TestBuildBackupsSnapshot_FutureDatedOffsite(an
8 d 14 h future-dated repo2 label readok/0 s/ overallok
before the fix) +BackupsPanel.test.tsx. (#311) -
usd-volume-restamp's lifted decompression cap can no longer ride the
pooled connection out of the call.RestampExactTierUSDVolumeraised
timescaledb.max_tuples_decompressed_per_dml_transaction = 0with a
session-levelSETon a borrowed*sql.Conn, and its comment claimed
that was "session-scoped … must not leak into the pool's serving
connections". Mechanically it was the opposite:Conn.Closereturns the
connection TO the pool and pgx v5 stdlib's defaultResetSessionis a
no-op (it pings and discards a conn left mid-transaction; it issues no
DISCARD ALL/RESET ALL), so the lifted cap persisted on that pooled
connection for the process lifetime and any later DML landing on it
would have run uncapped — harmless only becausestellarindex-opsis a
one-shot whose pool never serves the API. The restamp now runs its
window in ONE explicit transaction withSET LOCAL, the same tx-scoped
GUC discipline asFindPerSourceLedgerGaps/
SEP41SupplyEventKindResum: Postgres unwinds it at COMMIT/ROLLBACK, so
it cannot escape even on the error path. Behaviour of the write itself
is unchanged (same predicate, same identity, same INV-3 generation
guard). Unit test
(TestRestampExactTierUSDVolume_DecompressionCapNeverEscapesTheTransaction,
a GUC-scoping driver fake that also models pgx's no-op session reset)- the DB-backed integration test now asserts the pooled conn's cap is
untouched after a restamp and that TimescaleDB honours theSET LOCAL
form inside the transaction. (#312)
- the DB-backed integration test now asserts the pooled conn's cap is
-
The required
lintcheck no longer downloads a JSON schema from
golangci-lint.run on every PR.golangci/golangci-lint-action
defaultsverify: true, which runsgolangci-lint config verify—
and that command fetches
https://golangci-lint.run/jsonschema/golangci.v2.11.jsonschema.json
before it can validate anything. On 2026-08-28 the fetch died with
read: connection reset by peerand took a REQUIRED check red on a
diff containing no Go (PR #275); a rerun passed. Reproduced locally
against v2.11.4 with the network blocked (HTTPS_PROXYto a dead
port):compile schema: failing loading "…golangci.v2.11.jsonschema .json" … connection refused, exit 3. The schema check is NOT
dropped — golangci-lint's own loader silently ignores unknown keys
(a top-levelrunn:block is accepted bygolangci-lint runand
rejected only byconfig verify), so losing it would mean a
misspelt setting is a lint rule that quietly stops applying. Instead
the schema is vendored (scripts/ci/golangci.v2.11.jsonschema.json,
byte-identical to the site's copy) and validated offline by a new
scripts/ci/lint-golangci-configgate wired into thelintjob,
make lint-golangci-configandverify.sh. The gate also enforces
its own preconditions: every golangci-lint-action step must set
verify: false(so the fetch cannot silently return), ci.yml and
the Makefile must pin the same release, and that release must have a
vendored schema (a bump without a re-vendor fails rather than
validating against a stale copy). Fails loudly if the action step
disappears, so it can never pass vacuously. (#317) -
The MEV liquidation-cascade path stops treating unmapped
raw:
oracle rows as evidence — a squash merge had silently reverted the
guard.af5a9d1d(#305, a pgBackRest ansible change whose base
predated2ce680f3) landed a tree that removed PR #248's oracle
capture-totality consumer guards and DELETED their tests. From that
merge until now,OracleUpdatesForMEVScanno longer carried
AND asset NOT LIKE 'raw:%'andbuildCascadeCandidateno longer
calledoracleRefIsMapped, so the oneoracle_updatesconsumer with
NO asset keying — for the cascade correlator, any oracle row inside a
fill's ledger bracket is evidence — was again fed the
orientation-unknownraw:<symbol>rows the totality design records
verbatim. Both guards are restored verbatim, together with the two
deleted regression tests (mev_shape_test.go,cascade_raw_test.go)
and a behavioural assertion on the statement the store actually issues
(TestOracleUpdatesForMEVScan_ExcludesRawRowsFromTheIssuedSQL), which
survives a refactor away from the query const. Red-proven against
origin/main's own files at0f13aa14: twelveraw:NOTACOINrows
and no mapped row at all minted a completeliquidation_cascade
event naming four real accounts on the public/v1/mevfeed.
Still reverted by the same merge and deliberately NOT restored
here — each needs its own change, and the v0.48.0 entry describing
them is ahead of the code until then:internal/divergence/oracle.go's
unmapped-row refusal (+ itsoracle_raw_test.go); the
-- totality: includes unmappedmarkers and the "Unmapped feeds" KPI
ininternal/storage/timescale/{oracle,bespoke_oracle,diagnostics, protocol_stats}.go(+ thebespoke_oracle_shape_test.goassertion);
the repo guardTestOracleUpdatesQueriesDeclareRawRowPolicy
(oracle_updates_query_guard_test.go); and
test/integration/oracle_raw_consumers_test.go. -
ListMEVEvents' doc comment claimed a cap it does not apply. It
said "limit is capped at 500"; an out-of-range limit actually falls
back to the 50-row default, which is the package's convention
(ListIssuers,ListFreezeEvents,ListDivergenceLatest) and is
unreachable from/v1/mevanyway (parseExplorerLimit400s an
These notes are truncated. The full section for v0.51.0 exceeded
GitHub's 125,000-character release-body limit.
Read the complete entry in CHANGELOG.md.