Hashing takes a long time

[ruslanen]

@Slach hi,

We use ClickHouse (different versions) + clickhouse-backup in production, on setups with a lot of
tables (for example: ~50K MergeTree + ~500K Join, ~954 GiB).

I noticed that hashing takes a really long time during backup.
Looks like fetching hash_of_all_files from system.parts runs together with FREEZE and they contend on some mutex inside ClickHouse - FREEZE jumps from ~100ms to ~5000ms per table because of it.

I tried just skipping that lookup and the difference is huge:

• with hashing: FREEZE ~5029ms, create ~9.5h
• without: FREEZE ~116ms, create ~26min

I know hash_of_all_files isn't useless - incremental create --diff-from-remote uses it to
dedup parts against the previous full backup, so skipping it on full backups would quietly hurt
future increments.

That's why I only did it behind an opt-in flag, not automatically.
Concretely, what I did:

• added a config field clickhouse.skip_hash_of_all_files (env CLICKHOUSE_SKIP_HASH_OF_ALL_FILES);
• added a --skip-hash-of-all-files CLI flag to create / create_remote / watch;
• gated the existing lookup on it in AddTableToLocalBackup - one extra condition, so the default
  behavior is unchanged:

// before
useHashOfAllFiles := version >= 19011000
// after
useHashOfAllFiles := version >= 19011000 && !b.cfg.ClickHouse.SkipHashOfAllFiles

When the flag is off (default) everything works exactly as today; when it's on, the per-part
hash_of_all_files read from system.parts is skipped and the FREEZE contention goes away.

My two main questions:

1. Would you accept a PR with this config/flag as-is? If yes, which shape do you prefer:
   • just keep the opt-in flag and document the diff-from-remote caveat,
   • or keep the flag but warn/block when it's used with --diff-from-remote?
2. Or is there a better way to speed up the hashing itself - e.g. batch/defer the system.parts
   lookup so it doesn't fight with FREEZE - so no flag would be needed at all?

[Slach]

feel free to make PR, i created issue #1465

[Slach]

sorry for late reply