feat: fwss dataset/piece index enrichment#100
Conversation
|
FWSS isn't the only user of PDPVerifier. We've held off on doing this because entangling the two creates a false view of the system. PDPVerifier is a utility contract that happens to be used by FWSS to get its work done. On mainnet, there's more volume going through PDPVerifier via the Storacha contract than FWSS at the moment and there's nothing stopping anyone else using it to do whatever they want. We've debated enriching like this because it would be nice on the PDP Explorer to be able to dive deeper into them, and @juliangruber is currently looking at this from the top-down Filecoin Pay perspective. As it stands though this feels like a bit of a hack and maybe we should be thinking more creatively about how to make the subgraph code in here either reusable or extensible such that someone can take it and deploy their own subgraph based on it but either enriched with their custom context, like FWSS, or to fit in with some larger view. |
|
Hi @rvagg, is your concern mainly that I tried to bolt a dealbot usecase onto the pdp-explorer or is the general direction not right? Remember I'm coming from the retrieval++ use case.
Would it work if we took what's in this PR, moved it to the dealbot repository, and deployed a dedicated subgraph for dealbot that performs the aggregation? Currently dealbot seems to reuse this PDP Explorer's subgraph. |
This sounds like a good plan TBH, but i'm not sure if we want to take on owning/deploying our own subgraph. How much work would it be to deploy a dealbot-specific subgraph? |
|
Deploying subgraphs to Goldsky seems very easy (unless I'm missing something). In the PR description you can find two links at the bottom which point to my test deployments. |
Imports the goldsky subgraph mappings from FilOzone/pdp-explorer#100 as an in-tree package. This is the subgraph dealbot will own and deploy for itself (motivated by dealbot#427 anonymous retrieval check). Integrated with pnpm workspace, parameterized over networks.json for mainnet (filecoin) and calibration (filecoin-testnet), and pinned assemblyscript@0.19.23 so matchstick-as@0.6.0 picks up its binary. Biome and root test/build scripts intentionally skip this package — it is AssemblyScript compiled to WASM via graph-cli, and its lifecycle is "rebuild and redeploy to Goldsky", not per-PR. Schema, handlers, and tests are currently the unmodified upstream pdp-explorer content; subsequent commits will trim them to the three queries dealbot actually uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Imports the goldsky subgraph mappings from FilOzone/pdp-explorer#100 as an in-tree package. This is the subgraph dealbot will own and deploy for itself (motivated by dealbot#427 anonymous retrieval check). Integrated with pnpm workspace, parameterized over networks.json for mainnet (filecoin) and calibration (filecoin-testnet), and pinned assemblyscript@0.19.23 so matchstick-as@0.6.0 picks up its binary. Biome and root test/build scripts intentionally skip this package — it is AssemblyScript compiled to WASM via graph-cli, and its lifecycle is "rebuild and redeploy to Goldsky", not per-PR. Schema, handlers, and tests are currently the unmodified upstream pdp-explorer content; subsequent commits will trim them to the three queries dealbot actually uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This PR adds functionality to index the
FilecoinWarmStorageServicecontract alongsidePDPVerifierso a single pdp-explorer query can answer both "what PDP state is this dataset in?" and "what FWSS metadata was it created with?"This is motivated by FilOzone/dealbot#427 (anonymous retrieval check). With this change, dealbot replaces a client-side join of two subgraphs with one native GraphQL query that filters on
withIPFSIndexing,fwssPayer, andrawSizeall at once.Changelog
FilecoinWarmStorageServicecontract added to all three manifests handled in fwss.tsServiceTerminatedflipsDataSet.isActive = false.PDPPaymentTerminateddoes not flipisActive. Clients compare to current epoch themselves. Is this correct? Should it also flip isActive?There was one complication with the event ordering from the contract:
PDPVerifier._createDataSetcallslistener.dataSetCreated(...)before emitting its ownDataSetCreated(seepdp/src/PDPVerifier.sol:589-592), so FWSS's event has the lower log index and fires first. To avoid the FWSS handler running against a missingDataSetentity:fwss.ts:handleFwssDataSetCreated: When first to run, creates a stub with required non-null defaults and sets FWSS fields.pdp-verifier.ts:handleDataSetCreated— changed fromnew DataSet(...)to load-or-create. When the FWSS handler already created a stub, we load it (preserving FWSS fields) rather than overwrite.Only
DataSetCreatedhas this inverted ordering.PiecesAddedandStorageProviderChangedemit PDPVerifier-first, so those FWSS handlers are straightforward patch-the-existing-entity.Downstream example
This unlocks richer filtering on existing dealbot queries. Example:
GET_PROVIDERS_WITH_DATASETSin dealbot (apps/backend/src/pdp-subgraph/queries.ts) currently is:With this PR, it can filter by FWSS metadata in the same query. Anonymous-retrieval piece selection becomes:
All filter dimensions (active state, piece size, IPFS indexing, payer exclusion) are native indexed filters.
I have deployed the new subgraphs for testing to Goldsky:
Transparency: Implementation heavily augmented by Claude Opus 4.7