Skip to content
Discussion options

You must be logged in to vote

Auto compaction can stay off for this. In Delta OSS 3.2.0, CLUSTER BY sets up the liquid-clustering columns, and OPTIMIZE table_name is what triggers incremental clustering of existing data; CLUSTER BY alone does not rewrite existing files. I ran delta-spark==3.2.0 with spark.databricks.delta.autoCompact.enabled=false, created a table clustered by (id, value), and OPTIMIZE still wrote a liquid-clustered file. Auto compaction is a separate synchronous, post-write pass for small files.

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by felipebart10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants