Boost indexing stalls on large DBs - resolved locally with pagination tweak

[TippyFlitsUK]

Checklist

• This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to boost.

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• Other

Boost Version

boostd version 2.4.4+mainnet+git.e91fe37.dirty

Describe the Bug

Over the last few weeks, my Boost node’s (f08403) IPNI indexing has significantly slowed down or has been getting stuck completely.

Recent Lighthouse deals were barely indexing, and older Solana ones were sitting in “indexing” state for months.

When we dug into it, we saw that Boost’s HTTP publisher was hanging on ad entry lookups - taking 10+ minutes to respond and then returning nothing.

With @LexLuthr's help, we traced it to the boostd-data query against Yugabyte - it’s scanning massive result sets with no pagination.

My DB is ~4 TiB with billions of CIDs, so these unpaged queries basically choke the process.

Adding pagination cleared the backlog:

boost/extern/boostd-data/yugabyte/service.go

Line 374 in 2dc284a

iter := s.session.Query(qry, pieceCid.Bytes()).WithContext(ctx).Iter()

- iter := s.session.Query(qry, pieceCid.Bytes()).WithContext(ctx).Iter()
+ iter := s.session.Query(qry, pieceCid.Bytes()).WithContext(ctx).PageSize(10000).Iter()

After this change, all my deals were indexed and announced.

Logging Information

date && boostd index list-multihashes bafyreicb2jgywon37yzwwap6j2mxt6lja72apdb3m5m2uhgfrhzhws4mce | head -n 20 && date
Sat 18 Oct 20:34:49 BST 2025
Found 32261008 multihashes for deal with ID bafyreicb2jgywon37yzwwap6j2mxt6lja72apdb3m5m2uhgfrhzhws4mce:
  122011e2a3f632f2ca1e6d59fd7193210a4bf67ef2f874f2dc2f603340fccae14ca9
  12209dfcd0ad7312052b73eb2b156e20fcd9f47d60c6bbe48bad0574194fb23eed24
  122065dd1b44f28eda6989e44b6a7ffc66b9f720bf711d2e313d6822fdd60c10fc43
  12207d11bd235c6ba27f6b71efb01fa4f9065c07e5677b188af56a3a5aa338896b40
  1220193e9612a0abbedbae5f5d552b631db2c1d4b6a53602189ee701bbc675846ee8
  1220c33444fb1802d95f3da5ab6f83c677d408cc6a349d16a0c2406ed6d9f2554f7f
  12200440c4c086ef8584a89e6a1e8b9b7a0120677d73451fc7363880cb6f747bfb77
  1220918bdb1d90b95cd292cd8e8d5e0a910267587c8fde70f99f1504eb872a43e2df
  122037b1d10603de305d59e220b6168aaf9fc2c87726fd37f8a88de50cf7acdd6a8b
  1220bf4d634099d338d6215938b5efde1e8fa18d9075d7f114a902b6d6e4af048673
  12205a1f84109d70c85e9f819f3070f1c95df14532b36268bae2599aff82fa327e48
  12208874df0e0c285d755af49047e9faf4cd143823847170c1f193be777e9ede847e
  1220cee32ff56281bce88b2cd1f0c6d716af5159c548ab343b83e2db98aadc9405ef
  1220d7141b18170e2af9e0b3288004efc82d9fa8b752db1d1d422c1b23333a572ccc
  122089971ce0921771ec7e8e57fe62c00832e935549f6bbe39818492fed27f872d3b
  1220d9b8f0d056a36b6ed660ad76f1b4922287c5e0a71d2b1ac3b379a29f56bb0271
  12208eda8b2601b1f10be64597ae9f9b0650416074e4667e48b4de1e8bc77fea135b
  1220ec0621383088b1cf5775f4b10486f85cf0b27c307aa42c82ab60b70be6707a4b
  1220fcda1117458439b481d4ebea95f4fcd4b6d6029d6f2618f37b4d5ff770616778
Sun 19 Oct 06:20:31 BST 2025

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[LexLuthr]

Fixed in #2032