Replies: 2 comments
-
parts merging is very complicated , it's an heuristic algorithm has relation with parts size, part create time, free disks , free mergeing thread and so on, if you really want to understand this, please look at the code base. the main logical is around MergeTreeDataMergerMutator::selectPartsToMerge if you are on a code base before and include 22.3-lts |
Beta Was this translation helpful? Give feedback.
-
Some more (detailed) information on your questions; In general as first point, it's good to understand the relation between parts and the partitioning key (https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/custom-partitioning-key)
For more details on merges, e.g. how they are picked and the parameters, see https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/SimpleMergeSelector.h#L6-L81 To configure multiple disks with S3, you could use config for example as: storage_configuration:
disks:
s3disk:
endpoint: ...
path: ...
region: ...
type: s3
s3diskWithCache:
cache_on_write_operations: 1
disk: s3disk
do_not_evict_index_and_mark_files: 0
max_size: ...
path: ...
type: cache
default:
keep_free_space_bytes: 536870912
policies:
default:
move_factor: 0.5
volumes:
01_hot:
disk: default
max_data_part_size_bytes: 4294967296
02_cold:
disk: s3diskWithCache
s3:
volumes:
main:
disk: s3diskWithCache There are various resources in our docs, e.g. https://clickhouse.com/docs/en/operations/storing-data/#using-local-cache We would recommend not using |
Beta Was this translation helpful? Give feedback.
-
Hello,
I wanted to understand background merges better. I understand that background merges keep happening until it reaches 150GB as defined by
max_bytes_to_merge_at_max_space_in_pool
.I would like to understand following.
ORDER BY
? So for e.g. ifORDER BY
is bydate
and if new inserts have older date it will eventually get merged with corresponding older part that has matching date (even if this older part is of 150GB and needs no merging) or is ordering not maintained across parts so this new insert can be merged in a new 150 GB part?ReplacingMergeTree
if duplicate entry to be removed are part of large parts having 150GB of size, do all of them gets reprocessed (as this deduplication only occurs during merge and there is no need to merge parts that have reached 150GB)?I am trying to understand all of this as I would like ClickHouse to make no (or minimal) changes to parts once they reach size defined by
max_bytes_to_merge_at_max_space_in_pool
. Reason for this is because as our data would grow we would want to use slower-larger disks and maybe S3 for older data. I understand I can also create multiple volumes, have ClickHouse move parts to this volume once it reaches 150GB usingmax_data_part_size_bytes
and setprefer_not_to_merge
on this volume. What would be disadvantages of doing this?Beta Was this translation helpful? Give feedback.
All reactions