40947f62 - Create the missing Binance custody assets that wedged ledger posting - #4452
Conversation
Every asset held on an exchange needs a second `asset` row with type='Custody' and blockchain=<ExchangeName>, next to its on-chain row. Both were missing for ONDO and ADA on Binance, so LedgerBootstrapService — which iterates over `asset` rows — could never create their CoA accounts. Without the account, exchangeAssetByCcy fail-closes with "Ledger account Binance/ONDO not found (CoA bootstrap missing)" and processForward breaks out of the batch, leaving the exchange_tx watermark frozen: every later row stays unposted, whatever exchange it belongs to. The recurring CoA bootstrap cron cannot heal this — it only creates accounts for `asset` rows that exist. ADA has the same gap and only escaped the wedge because its last Binance trade predates the ledger cutover. The migration is idempotent per uniqueName and takes a transaction-scoped advisory lock first: uniqueName is not DB-unique (the unique index covers (dexName, type, blockchain)) and there is no ENVIRONMENT guard, so two instances starting at once could otherwise both pass the idempotency check and have the second INSERT crash on the index. Prices come from each coin's on-chain row via subquery, with a fail-loud guard instead of a silent NULL priceRuleId.
…ody migration up() claimed to be idempotent per uniqueName but evaluated the price-source guard first, so a re-run against an already-created asset still threw once its source had been renamed or removed. Checking the target first — as AddSavingZchfAsset does — makes the claim true and keeps the guard on the path that actually inserts. Both assets stay independent: an existing ONDO row no longer skips ADA. Also narrows the guard's doc comment to what it really rejects, a missing or renamed source row. It does not require the source to carry a priceRuleId, and an empty one is legitimate — the seed ships Ethereum/ONDO exactly like that, and the subquery carries the NULL over on purpose.
The header claimed dev, loc and CI all mirror the two rows from migration/seed/asset.csv. Only LOC does: main.ts runs the seed exclusively for Environment.LOC and seed.js blocks everything else outright, so dev, CI and prod receive the rows from this migration alone.
|
Three review passes were needed to reach zero findings. Pass 1 raised three points, all on the migration: the price-source guard was documented as preventing a silent The guard itself was left alone on purpose — the seed ships Pass 2 found one more inaccuracy in the same area — the header still claimed dev and CI mirror the rows from the seed CSV, which only |
Problem
Ledger posting for
exchange_txhas been wedged in production since 2026-07-28 20:40 UTC.exchangeAssetByCcyresolves the ledger account by name and fail-closes when it is missing:processForwardcatches that andbreaks out of the batch, deliberately leaving the watermarkuntouched so the row is retried. With a permanently missing account the retry never succeeds, so
ledgerWatermark.exchange_txfroze atlastProcessedId: 145947and every later row stopped posting —regardless of which exchange it belongs to. The content-change scan breaks on the same row, so
status flips of already-booked rows are no longer reversed either.
At the time of writing that is 163 unposted
exchange_txrows worth about CHF 358k, growing byroughly one every few minutes, plus one error per minute from each of the two scans.
Root cause
Every asset held on an exchange needs a second
assetrow withtype='Custody'andblockchain=<ExchangeName>, next to its on-chain row —Monero/XMRhasMEXC/XMRandKraken/XMR,Solana/SOLhasBinance/SOL, and so on.LedgerBootstrapService.bootstrapAssetAccounts()iteratesover
assetrows, so it can only create a CoA account once that row exists.Ethereum/ONDOhas noBinance/ONDOcounterpart, so no account was ever created. The recurringLEDGER_COA_BOOTSTRAPcron cannot heal this: it creates accounts for assets, not assets themselves.ONDO traded on Binance before (May, June) without a wedge because those trades predate the ledger
cutover and are covered by the aggregate opening — 145948 is the first one after it.
Checking every symbol and deposit/withdrawal currency traded over the last 120 days against the
existing custody assets turned up exactly one more gap:
Binance/ADA. It has not wedged yet onlybecause its last Binance trade also predates the cutover. Both are created here so the second one
does not reproduce this incident on the next ADA trade.
Change
A data migration creating the two missing custody rows. Everything else already exists.
uniqueName, so a re-run is a no-op.uniqueNameis not DB-unique — the unique index covers(dexName, type, blockchain)— and this migration has noENVIRONMENTguard, so two instancesstarting at once could both pass the idempotency check and have the second
INSERTcrash on theindex. Same approach as
AddSavingZchfAsset.ENVIRONMENTguard. UnlikeAddBankFrickCustodyAssetsthese rows carry noLiquidityManagementRule, no bank link and no external side effect; they are correct in everyenvironment.
migration/seed/asset.csvmirrors them so a freshly seeded local database matches(the seed runs on
LOConly; CI never touches it).Binance/ONDOinherits fromEthereum/ONDOandBinance/ADAfromCardano/ADA. A fail-loud guardrejects a missing or renamed source row rather than inserting an unpriced asset. It deliberately does
not require the source to carry a
priceRuleId: an empty one is a legitimate state that the subquerycarries over verbatim —
migration/seed/asset.csvshipsEthereum/ONDOexactly like that.LiquidityManagementRule. Neither asset is held as a balance — both are pass-through, andneither has a
liquidity_balancerow. Adding a feed would be a product decision, not part of this fix.refundEnabledisfalse, matching every live custody asset (theTRUEvalues on the olderBinance rows in the seed CSV are a stale snapshot).
Effect
Once deployed, the
LEDGER_COA_BOOTSTRAPcron creates both accounts within five minutes, thewatermark advances past 145948 and the backlog posts through. All backlogged rows already carry a
persisted
amountChf, so they book at their real CHF values rather than needing a mark.Verification
format:check,lint,type-checkand the affected suites run clean.plus a real-Postgres block behind
MIGRATION_TEST_PGfor create / idempotency / rollback, includingthe case where a target row already exists but its price source is gone —
up()resolves instead ofthrowing, and each asset is handled independently of the other.
from production before writing this.