feat(metrics): track active datasets by provider

[beck-8]

Summary

• expose the configured Dealbot wallet active FWSS dataset count per provider
• expose the configured expected dataset count and inventory freshness timestamp
• reuse the existing dealbot-owned Goldsky subgraph, filtering by configured WALLET_ADDRESS and isActive
• paginate subgraph results with an ID cursor and cache inventory refreshes for five minutes
• preserve previous gauge values when collection fails so freshness alerts can detect stale data
• document the new gauges and the corresponding monitoring expectation

Metrics

• dealbot_active_datasets{network,providerId,providerName,providerStatus}
• dealbot_expected_active_datasets{network}
• dealbot_active_datasets_last_success_timestamp_seconds{network}

Data source

The collector uses the existing per-network SUBGRAPH_ENDPOINT. The dealbot-owned subgraph marks a dataset inactive on FWSS.ServiceTerminated or PDPVerifier.DataSetDeleted, so terminated datasets are removed without waiting for final PDP cleanup.

At the current two-worker/two-network deployment this adds about four Goldsky queries per five-minute refresh, rather than walking thousands of historical datasets over RPC.

Validation

• backend unit tests: 648 passed
• TypeScript typecheck passed
• Biome check passed
• verified the active-dataset GraphQL query against the live calibration SUBGRAPH_ENDPOINT

parts #605

[beck-8]

It seems he has added too many RPC requests.

[Copilot]

Pull request overview

Adds a Prometheus-backed “active dataset inventory” for the configured Dealbot wallet by querying the existing dealbot-owned subgraph and exposing per-provider gauges to support alerting on dataset accumulation and inventory staleness (issue #605).

Changes:

• Introduces a subgraph query + validation for paginated isActive=true datasets filtered by configured WALLET_ADDRESS, grouped by provider.
• Adds a TTL-cached Prometheus collector that publishes active dataset counts per provider, the configured expectation (MIN_NUM_DATASETS_FOR_CHECKS), and a “last successful refresh” timestamp.
• Documents the new gauges and the monitoring/alerting expectation.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/infra.md	Adds an alerting recommendation for excessive active dataset counts and stale inventory.
docs/checks/events-and-metrics.md	Documents the three new dataset-inventory gauges and their semantics.
apps/backend/src/subgraph/types.ts	Adds ActiveDataSetPageResponse type + Joi validation for the active dataset page query.
apps/backend/src/subgraph/types.spec.ts	Adds unit test coverage for active dataset page validation (incl. null provider).
apps/backend/src/subgraph/subgraph.service.ts	Implements fetchActiveDataSetCounts with cursor pagination and provider aggregation.
apps/backend/src/subgraph/subgraph.service.spec.ts	Tests pagination + grouping behavior for fetchActiveDataSetCounts.
apps/backend/src/subgraph/queries.ts	Adds GET_ACTIVE_DEALBOT_DATASETS GraphQL query with cursor pagination.
apps/backend/src/metrics-prometheus/metrics-prometheus.module.ts	Registers new gauges and wires the new collector (plus SubgraphModule).
apps/backend/src/metrics-prometheus/active-datasets.collector.ts	Adds the TTL-cached collector that publishes counts/expectation/freshness per network/provider.
apps/backend/src/metrics-prometheus/active-datasets.collector.spec.ts	Adds unit tests for scrape behavior and “preserve previous values on failure”.

[SgtPooki]

we have more than 2 workers.

1. we should limit this to only running on the backend/api pod, of which there can only ever be one.
2. move the refresh interval off of the /metrics scrape path, that blocks all metrics on goldsky Http queries
3. surface _meta.block.number so we can tell if a subgraph is lagging

[beck-8]

Made some changes and see if they meet expectations. Thank you.

[SgtPooki]

just one thing that needs addressed and then a question

[silent-cipher]

Changes looks good to me...

[rjan90]

@SgtPooki I think Beck has addressed your suggestions, is there anything else needed here before merging?

[SgtPooki]

@SgtPooki I think Beck has addressed your suggestions, is there anything else needed here before merging?

nope, looks good to me, sorry for the delay