feat: experimental queries and interfaces for subscription and consumer tables #211
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is a first step toward storacha-network/w3up#711 as documented in https://github.com/web3-storage/w3up/pull/744/files?short_path=4e012d9#diff-4e012d97ece45b455806fabbe1f93ce11af56213b9137463a427e1c5d4d8bda6 This is an implementation spike of storing delegations in DynamoDB rather than D1. Because we already use DynamoDB in the upload-api, it made sense to reuse the configuration and testing infrastructure we've already built there. I copied a number of interfaces from https://github.com/web3-storage/w3up/tree/main/packages/access-api to get this done, and will migrate back to them if that ends up making sense. As this implies, the path forward for this is somewhat unclear and I need feedback and consensus from the team to keep this moving. We have a couple options to move forward: 1) Port the code in this PR over to the `access-api` repository and run it inside a cloudflare worker. This will require us to port some of the configuration and testing utilities in this repository to that repository and will leave us with a situation where compute in Cloudflare is reading and writing to DynamoDB in AWS. It will also require us to change the S3 logic in this PR to operate against R2, but that may Just Work thanks to theoretical API compatibility. This will also result in AWS-related logic being split between this repository and w3up, and we will probably want to extract common utilities to a separate package at some point in the future. 2) Port the delegations-related service endpoints in `w3up/access-api` over to this repository and get them running in an AWS Lambda. In this scenario we could either use S3 as implemented in this PR or continue to store delegations in R2. This option would move us in the direction of abstracting all operational details out of the w3up repository, following the pattern already used by w3up's `upload-api` package, which is imported by `upload-api` in this package. In this option I think it probably makes sense to migrate existing delegations from R2 to S3, but could be wrong on that front. My conversations with @Bengo suggested that the team may be mostly aligned behind option (2), especially because it seems like it moves us in the direction of isolating operational details of our services to this repository and turning the w3ui `*-api` packages into abstract implementations of the service providers we implement. Thoughts?
implement a test that represents my current best understanding of how provisions should work for now.
- add necessary context to run new providers - add validate-email GET and POST handlers as a new Lambda
…mer tables (#200) Implement the subscription and consumer tables specified by @Gozala in storacha-network/w3up#746 and use them to implement a new version of ProvisionStorage This builds on a PR review from @Gozala that lays out some of the queries we will need to support with these new tables: #194 (review) This is configured to merge into the main "D1 to Dynamo" development branch - I'm in favor of going to production with this implementation of `ProvisionStorage` rather than the one in that branch now.
fully remove standalone provisions table
I'm anticipating needing these queries, so adding them to the unstable interface and writing some basic tests to make sure they work
Since we already store delegations inside invocations in the invocation log, we can simply store the invocation CID and delegation CID and then fetch and find the needed delegations when they are requested. I'm not sure the implementation is correct, as I'm not super familiar with the distinction between invocations, workflows and receipts, but the types all check out! Would love careful review on the new `findDelegationInInvocation` method I've added to `upload-api/tables/delegations.js`
get the reworked provision and delegations test suites working here again, and implement `getCustomer` to get the customer/get test passing
rework the indexes of the subscription and consumer tables per his guidance in storacha-network/w3up#746 (comment)
We realized that at the moment, not all delegations can be retrieved from invocations, so re-enable storing delegations. This has the nice side-effect of fixing the test I disabled, which was the original goal of this particular sidequest.
fix the build?!
seeing some weird module resolution errors in CI - let's see if bumping the node version fixes them
maybe fix the build?
need to interact with the postmark API and fetch isn't in the standard env now!
must remove this before deploying to prod!
use a token on the dev postmark server - I will delete after testing
Node and Lambda's HTTP Response types are sliiiightly different, nice.
Dynamo can't store Infinity
configured and paid for a month of MailSlurp so we can take it for a test drive
reviewed this PR and made a bunch of updates
they weren't being added because Lambda is a special snowflake
I'll reintroduce these in a separate branch
This reverts commit d29b944.
7 tasks
|
closing to free up AWS seed.run resources (IAMs were running low) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Per @vasco-santos's suggestion in #194 (comment) I've moved the experimental queries and interfaces to a separate PR to keep that PR cleaner.
Mostly here as a guide for posterity and future work - this can be closed if we don't use it within a few months.