feat: filecoin metrics for ucan stream prometheus and updates metrics endpoint

[vasco-santos]

Adds filecoin metrics for ucan stream in the same pattern we track other metrics in https://up.web3.storage/metrics

We want to rely on UCAN Stream metrics table to in a "prometheus style" increment metrics over time for totals. There are some metrics in Firehose that we can rely on for ad-hoc queries and business queries (like monthly usage). More than the price and efficiency in the long run for totals in the Firehose, there are some limitations that are explicit here:

• we use PieceCids to encode Piece size, which we will never be able to handle in Athena level as we cannot decode the CID to get the size
• UCAN invocations (like aggregate/offer) may rely on attached blocks to move attachments with the invocation, while only identifying the CID for them. E.g in aggregate/offer we encode the very large list of pieces to put into the aggregate in a CBOR block, and inline it in the CAR of the invocation, while providing pieces as CID of the block in the nb object.

We will need most measures I wanted to handle, except for tracking execution time between the invocations. We need storacha/w3up#970 to get the power to track execution time of a UCAN effect (e.g. time between receipt of aggregate/offer and aggregate/accept

[seed-deploy]

View stack outputs

• pr290-w3infra-BillingDbStack

  Name	Value
  customerTableName	pr290-w3infra-customer
  spaceDiffTableName	pr290-w3infra-space-diff
  spaceSnapshotTableName	pr290-w3infra-space-snapshot
  usageTable	pr290-w3infra-usage

• pr290-w3infra-BillingStack

  Name	Value
  ApiEndpoint	https://pum9g68234.execute-api.us-east-2.amazonaws.com
  billingCronHandlerURL	https://jszzgutawxjdsafdsx32faxlum0njwnv.lambda-url.us-east-2.on.aws/
  CustomDomain	https://pr290.billing.web3.storage

• pr290-w3infra-CarparkStack

  Name	Value
  BucketName	carpark-pr290-0
  Region	us-east-2

• pr290-w3infra-RoundaboutStack

  Name	Value
  ApiEndpoint	https://gtmdbe0k1b.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr290.roundabout.web3.storage

• pr290-w3infra-SatnavStack

  Name	Value
  BucketName	satnav-pr290-0
  Region	us-east-2

• pr290-w3infra-UploadApiStack

  Name	Value
  ApiEndpoint	https://xbvhe4nw65.execute-api.us-east-2.amazonaws.com
  CustomDomain	https://pr290.up.web3.storage

• pr290-w3infra-BusStack

• pr290-w3infra-FilecoinStack

• pr290-w3infra-ReplicatorStack

• pr290-w3infra-UcanFirehoseStack

• pr290-w3infra-UcanInvocationStack

• pr290-w3infra-UploadDbStack

[vasco-santos]

in https://github.com/web3-storage/w3infra/blob/9897890c65968f7d47a4b3b520f736863343642b/upload-api/ucan-invocation.js#L169

[alanshaw]

Yes but, only for receipts.

I ended up re-typing these definitions in billing to more accurately reflect what's there and not get errors for properties that don't always exist:

https://github.com/web3-storage/w3infra/blob/9897890c65968f7d47a4b3b520f736863343642b/billing/lib/api.ts#L210-L240

[vasco-santos]

This is great @alanshaw thanks!
I will add this as follow up PR

[alanshaw]

LGTM

[alanshaw]

Yes but, only for receipts.

I ended up re-typing these definitions in billing to more accurately reflect what's there and not get errors for properties that don't always exist:

https://github.com/web3-storage/w3infra/blob/9897890c65968f7d47a4b3b520f736863343642b/billing/lib/api.ts#L210-L240

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ DecodeBlockOperationError: missing block: [object Object] staging-w3infra-UcanInvoc-metricsaggregateoffer... View Issue

_{Did you find this useful? React with a 👍 or 👎}