Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filecoin/offer from client rollout plan #344

Closed
7 tasks
vasco-santos opened this issue Mar 18, 2024 · 1 comment
Closed
7 tasks

filecoin/offer from client rollout plan #344

vasco-santos opened this issue Mar 18, 2024 · 1 comment

Comments

@vasco-santos
Copy link
Contributor

vasco-santos commented Mar 18, 2024

We need a rollout plan where we can hook clients that write to anywhere (R2 bucket without write events for now) while keeping functionality for old clients until desired. Old client currently does not invoke filecoin/offer, which means we need to trigger this from bucket event in the old code path.

In the context of writes to anywhere, we flagged this to be a good timing to adopt new capabilities, such as /space/content/add/shard in order to ease migration. We can keep for some time store/add pointing to the S3 bucket with current behaviour.

By the above, also filecoin/offer needs to keep both ways possible, ideally without needing to deploy two storefront APIs. For the old bucket, we can keep a trigger from bucket write where we simulate the client behaviour: invoke filecoin/offer. This will actually mean that in this code path we will need to compute Filecoin Piece at that point, as well as on validation. But, looking at our AWS bill, we paid 200 USD in February for lambda execution of whole w3infra. This assumes then that we will not incur a lot of additional cost while making migration much simpler.

With the above solution, we can have new client to write to new bucket via /space/content/add/shard capability, where client will also be fully responsible for invoking filecoin/offer, while triggering it manually on bucket event (note that if client on old world also manually triggers it, we have deduping in place).

There is a need to separate this work in two iterations though, one where we decouple from bucket event while still expecting writes to happen to either S3 or R2 (without client upgrade), and a second iteration where we are fully data anywhere.

We can in short expect:

vasco-santos added a commit to storacha-network/w3up that referenced this issue Mar 21, 2024
In context of storacha-network/project-tracking#25
and storacha-network/w3infra#344 we set the client
to perform filecoin/offer already so that new users, as well as users
that upgrade in the meantime can already perform this, in order to
better prepare us to get rid of bucket event

---------

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>
@vasco-santos
Copy link
Contributor Author

This was done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant