Repository navigation
Liquid-clustering incremental clustering and autoCompact. #7794
|
Hi folks, I have a doubt about liquid clustering and autoOptimize/autoCompact. Does the autoCompact feature on a dataset has to be active for incremental clustering to happen? Or incremental clustering will happen just with liquid-clustering being active? I'm using Delta OSS 3.2.0. |
Replies: 1 comment
|
Auto compaction can stay off for this. In Delta OSS 3.2.0, |
Auto compaction can stay off for this. In Delta OSS 3.2.0,
CLUSTER BYsets up the liquid-clustering columns, andOPTIMIZE table_nameis what triggers incremental clustering of existing data;CLUSTER BYalone does not rewrite existing files. I randelta-spark==3.2.0withspark.databricks.delta.autoCompact.enabled=false, created a table clustered by(id, value), andOPTIMIZEstill wrote a liquid-clustered file. Auto compaction is a separate synchronous, post-write pass for small files.