feat: w3 aggregate protocol client and api implementation #787

vasco-santos · 2023-05-03T12:46:59Z

This PR is a first iteration for w3 aggregate client and API implementations. It follows same patterns we identified while working on upload-api and upload-client and is agnostic from the deployment service. aggregate-api exposes a lib and a set of tests that implementers with infra should follow, as well as the needed interfaces that must be fulfilled. It also includes fx usage for aggregate/offer return value

It follows spec https://github.com/web3-storage/specs/blob/feat/filecoin-spec/w3-aggregation.md

There are two interfaces defined in this service:

offerStore - responsible for receiving offers to be queued. Implementer receives commD of aggregate together with Offer content. These will be used to give SPs. Note that implementer should keep track of the offer's commD so that it can query over time when a deal is done (or failures), so that the fx invocation can be executed and a receipt generated
aggregateStore - store of aggregates already places with SPs (implementor like spade-proxy will have this store hooked with spade DB)

It is worth noting that this is a first iteration and not the complete thing at this moment. More specifically, in follow up iterations we need to:

validate commitmentProof aggregate-api should validate commD #805
validate cbor decoded offer with schema aggregate-api should have schema to validate decoded block #806
aggregateGet types are still unknown at this moment aggregate-api aggregate/get set types #807

New detailed issues for above will be created before this PR gets merged.

Part of storacha/w3filecoin-infra#19

Still to do:

change commp from text to CID

Usage:

aggregate-api - feat: add ucanto aggregation api service web3-storage/dealer#2
aggregate-client - feat: integrate aggregation client w3filecoin-infra#22

Closes #772 and #773

Upgrades ucanto to 8. There are no breaking changes that affect normal functionality as expected

vasco-santos · 2023-06-06T09:14:48Z

While writing this and getting to know more about commP, commD and so on, thinking perhaps we should do some renames from what we decided in spec:

aggregate/offer nb to rename commitmentProof to commD or dataSegementProof ?
CBOR block offer data structure to rename commitmentProof to pieceLink and size to pieceSize ?
- do we also need to store paddedSize ?
aggregate/get nb to rename commitmentProof to commD or dataSegementProof ?

vasco-santos · 2023-06-07T14:00:51Z

packages/access-api/package.json

@@ -95,6 +95,7 @@
    },
    "rules": {
      "unicorn/prefer-number-properties": "off",
+      "unicorn/expiring-todo-comments": "off",


lint fails with TODO: ... comments now ...

vasco-santos · 2023-06-07T14:14:46Z

A few follow ups will be worked on:

validate commitmentProof aggregate-api should validate commD #805
validate cbor decoded offer with schema aggregate-api should have schema to validate decoded block #806
aggregateGet types are still unknown at this moment aggregate-api aggregate/get set types #807
integrate spec changes from fix: align aggregation namings from commp learnings specs#62

alanshaw

LGTM, just a minor non blocking consistency thing on the failures...

alanshaw · 2023-06-07T14:19:17Z

packages/aggregate-api/src/aggregate/get.js

+
+class AggregateNotFound extends Server.Failure {
+  get name() {
+    return /** @type {const} */ ('AggregateGetFailure')


Suggested change

return /** @type {const} */ ('AggregateGetFailure')

return /** @type {const} */ ('AggregateNotFoundFailure')

alanshaw · 2023-06-07T14:28:05Z

packages/aggregate-api/src/aggregate/offer.js

+  get name() {
+    return /** @type {const} */ ('AggregateOfferInvalidUrlFailure')
+  }
+}


As far as I can tell most of the time we don't suffix with Error or Failure, e.g.

https://github.com/web3-storage/w3up/blob/57182b1d193924a6c5eefa3e01ef56719303c42e/packages/capabilities/src/types.ts#L47-L49
https://github.com/web3-storage/w3up/blob/57182b1d193924a6c5eefa3e01ef56719303c42e/packages/upload-api/src/types.ts#L236C14-L238

Perhaps be consistent with this?

good catch!

🤖 I have created a release *beep* *boop* --- ## [6.0.0](capabilities-v5.0.1...capabilities-v6.0.0) (2023-06-07) ### ⚠ BREAKING CHANGES * merge `@web3-storage/access-api` into `@web3-storage/upload-api` ([#790](#790)) ### Features * merge `@web3-storage/access-api` into `@web3-storage/upload-api` ([#790](#790)) ([4f6ddb6](4f6ddb6)) * w3 aggregate protocol client and api implementation ([#787](#787)) ([b58069d](b58069d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## [14.0.0](access-v13.0.2...access-v14.0.0) (2023-06-07) ### ⚠ BREAKING CHANGES * merge `@web3-storage/access-api` into `@web3-storage/upload-api` ([#790](#790)) ### Features * merge `@web3-storage/access-api` into `@web3-storage/upload-api` ([#790](#790)) ([4f6ddb6](4f6ddb6)) * w3 aggregate protocol client and api implementation ([#787](#787)) ([b58069d](b58069d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

🤖 I have created a release *beep* *boop* --- ## [7.0.0](w3up-client-v6.0.1...w3up-client-v7.0.0) (2023-06-08) ### ⚠ BREAKING CHANGES * merge `@web3-storage/access-api` into `@web3-storage/upload-api` ([#790](#790)) ### Features * merge `@web3-storage/access-api` into `@web3-storage/upload-api` ([#790](#790)) ([4f6ddb6](4f6ddb6)) * w3 aggregate protocol client and api implementation ([#787](#787)) ([b58069d](b58069d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

Previous PR went out without workflows commited #787 🤦🏼‍♂️ --------- Co-authored-by: Travis Vachon <travis.vachon@protocol.ai> Co-authored-by: Travis Vachon <travis.vachon@gmail.com>

🤖 I have created a release *beep* *boop* --- ## [9.1.0](upload-client-v9.0.1...upload-client-v9.1.0) (2023-06-20) ### Features * w3 aggregate protocol client and api implementation ([#787](#787)) ([b58069d](b58069d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

Context storacha/w3up#787 (comment) --------- Co-authored-by: Irakli Gozalishvili <contact@gozala.io>

🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2023-07-11) ### ⚠ BREAKING CHANGES * aggregate capabilities now have different nb properties and aggregate client api was simplified ### Features * w3 aggregate protocol client and api implementation ([#787](#787)) ([b58069d](b58069d)) ### Bug Fixes * update aggregate spec in client and api ([#824](#824)) ([ebefd88](ebefd88)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

🤖 I have created a release *beep* *boop* --- ## 1.0.0 (2023-07-11) ### ⚠ BREAKING CHANGES * aggregate capabilities now have different nb properties and aggregate client api was simplified ### Features * w3 aggregate protocol client and api implementation ([#787](#787)) ([b58069d](b58069d)) ### Bug Fixes * aggregate api test link comparison type ([#816](#816)) ([81bdf1c](81bdf1c)) * update aggregate spec in client and api ([#824](#824)) ([ebefd88](ebefd88)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: Vasco Santos <santos.vasco10@gmail.com>

vasco-santos added this to the w3up phase 4 milestone May 3, 2023

vasco-santos force-pushed the feat/w3-aggregate-protocol branch from 6b78e8d to 319bf4b Compare May 22, 2023 16:08

fix: upgrade ucanto to 8 (#794)

262bee3

Upgrades ucanto to 8. There are no breaking changes that affect normal functionality as expected

vasco-santos force-pushed the feat/w3-aggregate-protocol branch 19 times, most recently from 269f9fc to 83b0ed3 Compare May 26, 2023 13:03

feat: w3 aggregate protocol

9953dbd

vasco-santos force-pushed the feat/w3-aggregate-protocol branch 2 times, most recently from 5bfcb09 to 372c2f3 Compare June 1, 2023 12:55

vasco-santos force-pushed the feat/w3-aggregate-protocol branch 3 times, most recently from b0dcbb1 to 30d6d60 Compare June 6, 2023 09:11

fix: self signed invocation for web3.storage resource

c27fe2e

vasco-santos added 4 commits June 7, 2023 14:45

chore: change error message and appease linter

f217afd

fix: make commitment proof a link

7b75082

fix: unfroze package lock

d0af2bc

fix: move validations from client to api

d9e26cc

This was referenced Jun 7, 2023

aggregate-api should have schema to validate decoded block #806

Open

aggregate-api aggregate/get set types #807

Closed

vasco-santos requested a review from alanshaw June 7, 2023 14:00

vasco-santos commented Jun 7, 2023

View reviewed changes

vasco-santos mentioned this pull request Jun 7, 2023

fix: align aggregation namings from commp learnings storacha/specs#62

Merged

alanshaw approved these changes Jun 7, 2023

View reviewed changes

fix: consistency on error names

4b68649

vasco-santos merged commit b58069d into main Jun 7, 2023

vasco-santos deleted the feat/w3-aggregate-protocol branch June 7, 2023 14:43

This was referenced Jun 7, 2023

chore(main): release w3up-client 7.0.0 #809

Merged

chore(main): release access 14.0.0 #808

Merged

chore(main): release upload-client 9.1.0 #810

Merged

chore(main): release access-api 7.1.0 #796

Closed

chore(main): release capabilities 6.0.0 #811

Merged

vasco-santos mentioned this pull request Jun 7, 2023

fix: aggregation workflows #812

Merged

This was referenced Jun 21, 2023

chore(main): release aggregate-api 1.0.0 #821

Merged

chore(main): release aggregate-client 1.0.0 #822

Merged

vasco-santos added a commit to storacha/specs that referenced this pull request Jun 22, 2023

fix: align aggregation namings from commp learnings (#62)

63846e5

Context storacha/w3up#787 (comment) --------- Co-authored-by: Irakli Gozalishvili <contact@gozala.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: w3 aggregate protocol client and api implementation #787

feat: w3 aggregate protocol client and api implementation #787

vasco-santos commented May 3, 2023 •

edited

Loading

vasco-santos commented Jun 6, 2023

vasco-santos Jun 7, 2023

vasco-santos commented Jun 7, 2023

alanshaw left a comment

alanshaw Jun 7, 2023

alanshaw Jun 7, 2023

vasco-santos Jun 7, 2023

	return /** @type {const} */ ('AggregateGetFailure')
	return /** @type {const} */ ('AggregateNotFoundFailure')

feat: w3 aggregate protocol client and api implementation #787

feat: w3 aggregate protocol client and api implementation #787

Conversation

vasco-santos commented May 3, 2023 • edited Loading

vasco-santos commented Jun 6, 2023

vasco-santos Jun 7, 2023

Choose a reason for hiding this comment

vasco-santos commented Jun 7, 2023

alanshaw left a comment

Choose a reason for hiding this comment

alanshaw Jun 7, 2023

Choose a reason for hiding this comment

alanshaw Jun 7, 2023

Choose a reason for hiding this comment

vasco-santos Jun 7, 2023

Choose a reason for hiding this comment

vasco-santos commented May 3, 2023 •

edited

Loading