Create and document the access to S3 storage for testnet and mainnet

[frol]

We should make sure we have a sane separation of concerns.

Let's start with private access to S3 storage, but make sure we can change it later to public read access later (I assume so, but I want to be explicit)

[khorolets]

Access key of any user who has access to our AWS S3 buckets is required.

Bucket name is near-lake-testnet for testnet. The same naming concept will be used for mainnet. Where do you suggest to document this?

[frol]

@khorolets Do you believe the README on this repo is a bad place to put this documentation or why do you ask?

[khorolets]

@khorolets Do you believe the README on this repo is a bad place to put this documentation or why do you ask?

I'm asking because the info about the required credentials is already in the README. Do you want me to add bucket names and regions to README?

[frol]

There should be a section that describes how to use the data ("raw-mode" and also provide a reference to near-lake-framework).

Given that we can setup S3 in a way that the users will pay for their read usage, let's make sure we provide clear instructions on how to get it working (ideally, let's have curl commands showing how to download some specific block with all the shards)

[khorolets]

Given that we can setup S3 in a way that the users will pay for their read usage, let's make sure we provide clear instructions on how to get it working (ideally, let's have curl commands showing how to download some specific block with all the shards)

We haven't done any test regarding the "user will pay for their read usage" yet. And I doubt we can do it before the 1.0 milestone.

Though I can add the section that describes how to use the data with NEAR Lake Framework.

How about we extract the quoted part of your comment to a separate issue for the 1.0 milestone?

[frol]

Though I can add the section that describes how to use the data with NEAR Lake Framework.

Let's just have a section with a link to the framework.

We haven't done any test regarding the "user will pay for their read usage" yet. And I doubt we can do it before the 1.0 milestone.
How about we extract the quoted part of your comment to a separate issue for the 1.0 milestone?

Ok, please, extract it into a separate issue