feat: add ucanto aggregation api service #2

vasco-santos · 2023-05-17T13:57:03Z

This PR adds ucanto aggregation API service to POST / route.

It includes the implementation of:

offerStore - bucket keeping track aggregate offers received, so that they can be pulled by Spade
arrangedOfferStore - dynamoDB keeping track of arranged offers received, so that we can kick in cron jobs to poll Spade DB and invoke offer/arrange with status accepted or failed according to what happened. Note that we keep an index with stat so that we can query by stat in queue to then poll Spade on those and see if there are updates for that aggregate
Note that aggregate-store will be a wrapper for Spade DB to query aggregate infos. Therefore, it is not implemented in this PR and will come next.

Decisions:

how often should we be pulled. It influences the way we compute filenames. For ease I just did a 15 minutes multiple in this PR, we can come in follow up with a change once we align on what this number will be.

Note that:

package-lock was wrongly included before, so getting rid of it here
next will come CRON jobs to merge offers into single one (if needed) and

Closes storacha-network/w3infra#11

seed-deploy · 2023-05-17T14:10:09Z

View stack outputs

pr2-spade-proxy-ApiStack

Name Value

ApiEndpoint https://lx6u0g70se.execute-api.us-east-2.amazonaws.com

CustomDomain https://undefined
pr2-spade-proxy-CronStack
pr2-spade-proxy-DataStack

seed-deploy · 2023-05-19T14:19:03Z

Stack outputs updated

pr2-spade-proxy-ApiStack

Name Value

ApiEndpoint https://lx6u0g70se.execute-api.us-east-2.amazonaws.com

CustomDomain https://pr2.up.web3.storage
pr2-spade-proxy-DataStack

Gozala

Made some suggestions, but non I'd consider a blocker.

Gozala · 2023-06-12T18:29:06Z

README.md

+
+_Example:_ `MgCZG7EvaA...1pX9as=`
+
+#### `UCAN_LOG_BASIC_AUTH`


Aside: Do we really need an auth for the UCAN_LOG, they all come from API endpoint and anyone just as well could get messages here by sending there. That is to say maybe we don't need to gate it since we check incoming data is ucanto message.

Created https://github.com/web3-storage/w3infra/issues/203 to discuss

I get the point, but we do have things setup to count things like metrics that would be messed up if someone posted directly here.

...but if anyone sent messages directly to the log it wouldn't be a true representation of what we received and acted on

Gozala · 2023-06-12T18:31:09Z

package.json

    "deploy": "sst deploy",
    "remove": "sst remove",
    "console": "sst console",
-    "typecheck": "tsc --noEmit"
+    "typecheck": "tsc --noEmit && npm run typecheck -w packages/core",


I suggest using tsc --build instead. The --noEmit runs TS in legacy mode that can cause incompatibilities for the code that uses modern TS toolchains.

Good callout.

History here is this was generated with sst project generator https://docs.sst.dev/start/standalone which already had this. I also tried to get it out in favour of --build but than given this is a TS project (which now is the only available by sst project generator) the JS files from build are created inline, which this avoids.

I changed to --build but needed to add noEmit true to tsconfig to not generate the js files

Gozala · 2023-06-12T18:35:17Z

packages/core/tables/aggregate-store.ts

+
+export function useAggregateStore() {
+  return {
+    get: async (commitmentProof) => {


I would highly encourage use of Result types for any IO code se that error types can be captured in the interface.

I totally agree. We need to refactor this at the API code level and not here (and in reality for access-api and upload-api). Would be actually good to do it across the board to be consistent rather than here separately. I created issue for it storacha-network/w3up#818

packages/core/buckets/offer-store.ts

Gozala · 2023-06-12T18:47:28Z

packages/core/buckets/offer-store.ts

+      const putCmd = new PutObjectCommand({
+        Bucket: bucketName,
+        ContentType: 'application/json',
+        Key: `${getNextUtcDateName()} ${commitmentProof.toString()}`,


Reading this I am not entirely sure why this prefixing scheme is used, I think it would really help to have comment explaining it or a pointer to document explaining it.

there is a comment above. It is also part of what was decided with Riba and documented in https://www.notion.so/w3-spade-deal-flow-bd479485bbf64e85aa0dee0ad6fc3019

I can port this to a markdown file if that is helpful

Issue created #10

Gozala · 2023-06-12T18:49:30Z

packages/core/buckets/offer-store.ts

+        })
+      })
+
+      await pRetry(() => s3client.send(putCmd))


I'm also not sure what is the intention here, as in what kind of error it will attempt to recover from. I think it would be a good idea to provide a code comment to make it more clear.

this is for historical reasons where sometimes write fail in these buckets. So we usually wrap with retires across the stack, so that spontaneous issues do not break calls

Maybe we can create a separate put utility function that has the retries and comments about potential write failures it is aiming to recover. I just worry not having that context captured anywhere.

Gozala · 2023-06-12T18:50:08Z

packages/core/buckets/offer-store.ts

+        })
+      })
+
+      await pRetry(() => s3client.send(putCmd))


I'm also not sure what is the intention here, as in what kind of error it will attempt to recover from. I think it would be a good idea to provide a code comment to make it more clear.

this is for historical reasons where sometimes write fail in these buckets. So we usually wrap with retires across the stack, so that spontaneous issues do not break calls

Gozala · 2023-06-12T18:56:25Z

packages/core/buckets/offer-store.ts

+        })
+      })
+
+      await pRetry(() => s3client.send(putCmd))


Not sure what is the intention here, as in what kind of errors it attempts to recover from. I think it would be a good idea to provide a code comment to make it more clear.

this is for historical reasons where sometimes write fail in these buckets. So we usually wrap with retires across the stack, so that spontaneous issues do not break calls

Gozala · 2023-06-12T19:00:21Z

packages/core/buckets/offer-store.ts

+
+      // add commitmentProof for polling
+      // once an aggregate is fulfilled (accepted or rejected) a receipt will be generated.
+      await arrangedOfferStore.set(commitmentProof, 'queued')


Nit: Wonder if we can avoid separate store for this, e.g. if we knew invocation CID we could simply check if we have a receipt.

I spent a lot of time trying to figure it out best solution and so far this is what feels nicer. The issue is that receipt needs to be generated by spade-proxy, so it must keep some state so that a cron wakes up, checks Spade DB and issue receipts for the ones that were already success/failure.

Cool, maybe just write the comment that says that spade-proxy need to keep track of state across cron activations and db is used to ....

vasco-santos · 2023-06-08T12:59:47Z

packages/core/buckets/offer-store.ts

+      const putCmd = new PutObjectCommand({
+        Bucket: bucketName,
+        ContentType: 'application/json',
+        Key: `${getNextUtcDateName()} ${commitmentProof.toString()}`,


Note that we will still align with Spade about this part. Either we have a cron that merged this offers into a new file with format YYYY-MM-DD HH:MM:00, or Spade will be required to perform a list on the bucket with such prefix.

vasco-santos · 2023-06-08T13:06:40Z

packages/core/tables/arranged-offer-store.ts

+import type { Link } from '@ucanto/interface'
+import * as AggregateAPI from '@web3-storage/aggregate-api'
+
+export function createArrangedOfferStore(region: string, tableName: string, options?: ArrangedOfferStoreDbOptions) {


I am not totally happy with this name. Anyone have suggestions?

I named it after what we use this for, to call offer/arrange once we know if aggregate made it to SPs or not

After reading more on the filecoin side of things, it seems to me that this really should be called deal/arrange or deal/create.

vasco-santos · 2023-06-13T09:06:45Z

README.md

+
+_Example:_ `MgCZG7EvaA...1pX9as=`
+
+#### `UCAN_LOG_BASIC_AUTH`


Created https://github.com/web3-storage/w3infra/issues/203 to discuss

vasco-santos · 2023-06-13T09:16:05Z

package.json

    "deploy": "sst deploy",
    "remove": "sst remove",
    "console": "sst console",
-    "typecheck": "tsc --noEmit"
+    "typecheck": "tsc --noEmit && npm run typecheck -w packages/core",


Good callout.

History here is this was generated with sst project generator https://docs.sst.dev/start/standalone which already had this. I also tried to get it out in favour of --build but than given this is a TS project (which now is the only available by sst project generator) the JS files from build are created inline, which this avoids.

I changed to --build but needed to add noEmit true to tsconfig to not generate the js files

packages/core/buckets/offer-store.ts

seed-deploy bot had a problem deploying to pr2 May 17, 2023 13:57 Failure

vasco-santos force-pushed the feat/ucanto-service branch from a990f67 to 2e056ae Compare May 17, 2023 14:01

seed-deploy bot had a problem deploying to pr2 May 17, 2023 14:01 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 2e056ae to 7524d63 Compare May 17, 2023 14:06

seed-deploy bot temporarily deployed to pr2 May 17, 2023 14:06 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from 7524d63 to 8d6e768 Compare May 19, 2023 14:04

seed-deploy bot had a problem deploying to pr2 May 19, 2023 14:04 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 8d6e768 to 5d8fd55 Compare May 19, 2023 14:14

seed-deploy bot temporarily deployed to pr2 May 19, 2023 14:14 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from 5d8fd55 to 0262c9f Compare May 25, 2023 12:57

seed-deploy bot had a problem deploying to pr2 May 25, 2023 12:57 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 0262c9f to 3b16221 Compare May 25, 2023 15:11

seed-deploy bot had a problem deploying to pr2 May 25, 2023 15:11 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 3b16221 to bdef525 Compare May 26, 2023 13:49

seed-deploy bot had a problem deploying to pr2 May 26, 2023 13:49 Failure

vasco-santos force-pushed the feat/ucanto-service branch from bdef525 to 68592ba Compare May 26, 2023 14:46

seed-deploy bot had a problem deploying to pr2 May 26, 2023 14:46 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 68592ba to 2ff25c8 Compare June 1, 2023 13:29

seed-deploy bot had a problem deploying to pr2 June 1, 2023 13:29 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 2ff25c8 to 3537c54 Compare June 6, 2023 07:26

seed-deploy bot had a problem deploying to pr2 June 6, 2023 07:26 Failure

vasco-santos mentioned this pull request Jun 6, 2023

feat: w3 aggregate protocol client and api implementation storacha-network/w3up#787

Merged

1 task

vasco-santos force-pushed the feat/ucanto-service branch from 3537c54 to 9e65abb Compare June 6, 2023 19:27

seed-deploy bot had a problem deploying to pr2 June 6, 2023 19:27 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 9e65abb to 7378ea2 Compare June 8, 2023 07:56

seed-deploy bot had a problem deploying to pr2 June 8, 2023 07:56 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 7378ea2 to 3d19dd3 Compare June 8, 2023 08:00

seed-deploy bot had a problem deploying to pr2 June 8, 2023 08:00 Failure

vasco-santos force-pushed the feat/ucanto-service branch from 751a115 to 9b006b6 Compare June 8, 2023 11:55

seed-deploy bot temporarily deployed to pr2 June 8, 2023 11:55 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from 9b006b6 to e650e45 Compare June 8, 2023 12:06

seed-deploy bot temporarily deployed to pr2 June 8, 2023 12:06 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from e650e45 to 701981f Compare June 8, 2023 12:37

seed-deploy bot temporarily deployed to pr2 June 8, 2023 12:37 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from 701981f to b5822dc Compare June 8, 2023 12:39

seed-deploy bot temporarily deployed to pr2 June 8, 2023 12:39 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from b5822dc to d495ec7 Compare June 8, 2023 12:42

seed-deploy bot temporarily deployed to pr2 June 8, 2023 12:42 Inactive

vasco-santos force-pushed the feat/ucanto-service branch from d495ec7 to f8d9329 Compare June 8, 2023 12:47

seed-deploy bot temporarily deployed to pr2 June 8, 2023 12:47 Inactive

feat: add ucanto aggregation api service

47e6ced

vasco-santos force-pushed the feat/ucanto-service branch from f8d9329 to 47e6ced Compare June 8, 2023 13:02

seed-deploy bot temporarily deployed to pr2 June 8, 2023 13:02 Inactive

vasco-santos marked this pull request as ready for review June 8, 2023 13:04

vasco-santos requested review from Gozala and alanshaw June 8, 2023 13:04

vasco-santos mentioned this pull request Jun 8, 2023

feat: cron #3

Closed

2 tasks

Gozala approved these changes Jun 12, 2023

View reviewed changes

chore: tsc with build option but also with no emit in tsconfig

0a17853

seed-deploy bot temporarily deployed to pr2 June 13, 2023 09:19 Inactive

vasco-santos commented Jun 13, 2023

View reviewed changes

This was referenced Jun 13, 2023

API interfaces with IO should return result type storacha-network/w3up#818

Open

Document flows and offer store key namings #10

Open

feat: integrate aggregation client storacha-network/w3filecoin-infra#22

Closed

vasco-santos mentioned this pull request Jun 21, 2023

Infra for aggregate api to submit offers to Spade storacha-network/w3filecoin-infra#14

Closed

vasco-santos merged commit 084045b into main Jul 11, 2023
2 checks passed

vasco-santos deleted the feat/ucanto-service branch July 11, 2023 14:27

vasco-santos mentioned this pull request Jul 12, 2023

Setup infra and run aggregate-api in API Gateway #11

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add ucanto aggregation api service #2

feat: add ucanto aggregation api service #2

vasco-santos commented May 17, 2023 •

edited

Loading

seed-deploy bot commented May 17, 2023

seed-deploy bot commented May 19, 2023

Gozala left a comment

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

alanshaw Jun 26, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

vasco-santos Jun 13, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

Gozala Jun 26, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

Gozala Jun 12, 2023

vasco-santos Jun 13, 2023

Gozala Jun 26, 2023

vasco-santos Jun 8, 2023

vasco-santos Jun 8, 2023

Gozala Jun 26, 2023

vasco-santos Jun 13, 2023

vasco-santos Jun 13, 2023


		_Example:_ `MgCZG7EvaA...1pX9as=`

		#### `UCAN_LOG_BASIC_AUTH`


		_Example:_ `MgCZG7EvaA...1pX9as=`

		#### `UCAN_LOG_BASIC_AUTH`

feat: add ucanto aggregation api service #2

feat: add ucanto aggregation api service #2

Conversation

vasco-santos commented May 17, 2023 • edited Loading

seed-deploy bot commented May 17, 2023

seed-deploy bot commented May 19, 2023

Gozala left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented May 17, 2023 •

edited

Loading