getActivePieceCount fails with out-of-gas for large datasets

[frrist]

Problem

getActivePieceCount(uint256 setId) has O(n) complexity where n = nextPieceId[setId]. For datasets with tens of thousands of pieces, this exceeds the gas limit for eth_call requests.

Reproduction

On mainnet verifier (0xBADd0B92C1c71d02E7d520f64c0876538fa2557F):

# Works - small dataset
cast call 0xBADd0B92C1c71d02E7d520f64c0876538fa2557F "getActivePieceCount(uint256)(uint256)" 1 --rpc-url https://api.node.glif.io/rpc/v1
# Returns: 2

# Fails - large dataset (54,568 pieces)
cast call 0xBADd0B92C1c71d02E7d520f64c0876538fa2557F "getActivePieceCount(uint256)(uint256)" 89 --rpc-url https://api.node.glif.io/rpc/v1
# Error: out of gas

Note: getActivePieces with pagination works fine for the same dataset.

[rvagg]

Instead of adding new storage with new invariants, my gentle nudge here would be that since this is for external observation that using a subgraph or similar indexer might be the best way to deal with this? We have paginated active pieces if you want a canonical answer but for efficient lookup just having an external index that knows about pieces would be much more efficient and not push more complexity on chain.

Goldsky is on mainnet and calibnet and we've been using it for a bunch of our services. Chain Love runs a The Graph service that's also available. I use run my own instance of Ponder, indexing all the events and other details for contracts I care about, a Filecoin supported fork I maintain is here: https://github.com/rvagg/filecoin-ponder (you can see the way I'm using it in indexer/ @ https://github.com/filozone/foc-observer). Hugo has written his own Filecoin-focused indexer solution, like a lightweight Ponder that's designed for Filecoin, here: https://github.com/hugomrdias/foxer (early, it works but feature set is being expanded). Further, if this is something that your individual nodes need, Piri could also montior data sets themselves, watch for events, do occsional checking against the full pieces list. I know it's annoying having to do local state vs chain state, but the costs of increasing chain state is pretty high.