Explain how to configure ClickHouse on S3, with some potential caveats.

[alexey-milovidov]

Set up TTL for incomplete multipart uploads.
Do not set up a lifecycle policy for object deletion.
Do not set up object versioning.
Do not set up bucket replication.
Do not use FUSE over S3.
Do not point multiple shards to the same prefix in S3.
Do not delete objects on S3 manually.
Use separate buckets for ClickHouse and for other workloads.
How to properly configure the S3 bucket policy to allow the needed operations.
How to properly configure the instance IAM profiles.
How KMS affects performance.
Multi-region vs. single region buckets in GCS.
Choosing the proper region for a bucket.
Caveats of using R2 and B2.
Caveats of using Minio.
The need for a VPC gateway for S3.

[alifirat]

I'll also to recommend to add a daily check about your bucket size and the amount of data referenced by ClickHouse.

On the AWS, you can use CloudWatch to monitor your bucket size (it's published once) and compare to it to the system.parts table.