docs: please improve BACKUP/RESTORE documentation

[gena01]

Company or project name

No response

Describe the issue

Current BACKUP/RESTORE documentation covers the basic use cases very well and will work well on a standalone Clickhouse installation.

My usage is using S3/minio for backups in relation to 1 shard with 3 replicas setup. I have just re-read https://clickhouse.com/docs/operations/backup#configuring-backuprestore-to-use-an-s3-endpoint page and some things could be expanded/explained more:

1. The backups specifically to S3 mention directory as the last piece in the S3 url. My confusion lies in the fact that we had this before and were also able to backup to .zip file. I would love to see more information or maybe clearance where the directory makes more sense and why it's different from trying to have a compressed file on S3.
2. If i run a backup on schedule with full backup on weekend and incremental on other days on a distributed scheduler. Should I be running the regular version (and specifically same replica for this week?) vs running ON CLUSTER.... version. The ON CLUSTER clause actually validates and complains that it needs a directory for the backup. Would love to see more details on when to do this and how this is different (and maybe recommended in some cases?). I don't see much documentation on the ON CLUSTER backup bit.
3. RESTORE command for *ReplicatedMergeTree tables seems to work except that it doesn't re-use the uuid in the backup, but creates new uuid on the re-created table. It seems to work and restore the replicas when run as RESTORE ... ON CLUSTER, however if I want to restore a replica how would I go about doing that? How would I add a brand new replica if I have a backup on S3?

Additional context

No response

[Blargian]

Adding to this:

I might add, reading the documentation, I did not understand how does clickhouse handle concurrent inserts when a backup process is underway. We have tables in our production setup that are used by multiple services, and we can't afford to stop the insertion process into our tables. Does clickhouse support pre-where clauses to filter the data we want to backup (so older data is backed up, not newer 'to-be-merged' data?)