How to store engula's bucket/object to aws s3?

[zojw]

Recently, it seems valuable to do little discuss the title question during investigation #247

Background

For the previous implement(temp deleted by interface design fusion), it maps Engula's bucket/object to S3's bucket directly.

But it should have some questions:

• one AWS s3 account only supports max to 1000 buckets(default 100 and need submit service to increase to 1000)
• delete bucket operation need for-loop list_object + delete_objects platform: add aws s3 storage #99 (comment)

What if use one bucket to store all data?

It also has another option that stores all data in one S3 bucket and maintains "engula bucket" info by prefix inside an S3 bucket.

For AWS S3, there is no difference between a bucket per tenant and a prefix per tenant in one bucket after using IAM to restrict prefix access( S3's request limit is based on prefix level)

Pros:

• One AWS S3 account can support more than 1000 engula buckets
• It has better performance for "list own bucket" when hosting multiple tenants(clusters) in one AWS S3 account(can filter in s3 side)
• It opens the door for implementing "lazy delete" if we need nonblocking delete on bucket & object, it can mark delete + set lifecycle to lazy cleanup objects in a prefix(less API call and less money used)

Cons:

• It still needs multiple buckets when the user tries to deploy partitioned storage in multiple AWS regions(bucket binds to the region at creat time)
• Other cloud vendors maybe have some limitations on bucket level, we still need multiple buckets to overcome it, Support multi s3 bucket juicedata/juicefs#1221 (reply in thread)
• More complex in implement

Some known differences in engula's storage requirements

May be incomplete

• Engula never updates an existing object(SST compaction and sealed Journal offloader always create a new object instead of putting & updating existing object), does bucket also can be the same?
• It's uncommon to read a deleted object in Engula, and it seems no correctness problem event if it read a "deleted object"(reply an old journal segment or read an old SST file when upper layer need read old)
• One S3 account should host multiple tenants(clusters)' object and list-object for tenant(current cluster) should be periodically triggered by kernel poll, list-bucket + filter may be inefficient when to compare to list-object-with-prefix

Propose Mapping

For current, we are focused on one region multiple AZ deployment, one bucket + prefix should be valuable to take a try

(And it also naturally for different tablet-group pass different regional-bucket-name as root bucket when tablet-group works done)

• Save all objects in one S3 bucket
• tenant_id (or cluster name) as the first prefix
• Engula bucket name as the second prefix

So in one S3 bucket, we will store

/{tenant_id}/_engula_meta_/{engula_bucket} -> [bucket meta]
/{tenant_id}/{engula_bucket}/{object_key} -> [object content]

So for each operation:

init

Need to specify the root S3 bucket name and region(location), storage can create bucket if not exist.

create bucket

Put meta object in /{tenant_id}/_engula_meta_/{engula_bucket}

list bucket

list_object_v2 for /{tenant_id}/_engula_meta_/

delete bucket

Delete meta object in /{tenant_id}/_engula_meta_/{engula_bucket} and set lifecycle to clean /{tenant_id}/{engula_bucket}

(maybe or not need some background compensation task for the fail call)

create object

Put object in /{tenant_id}/{engula_bucket}/{object}

• Optional check the existence of the bucket? if the bucket has not been created, it should not be listed and the engine should not put an object in a not exist bucket?

delete object

Delete objects in /{tenant_id}/{engula_bucket}/{object}

list object in bucket

list_object_v2 for /{tenant_id}/{engula_bucket}

TODO

I will do a prototype to verify it, as part of #247

[w41ter]

Put meta object in /{tenant_id}/_engula_meta_/{engula_bucket}

How about add buckets before {engula_bucket}? like /{tenant_id}/engula_meta/buckets/{engula_bucket}`. The Journal also needs to use a similar mechanism to express a stream.

[zojw]

ohh~ yes, I forget the journal, we need to add it