[Spark] Implement Hilbert clustering

This PR is part of #1874. This PR implements a new data clustering algorithm based on Hilbert curve. No code uses this new implementation yet. Will implement incremental clustering using ZCube in follow-up PRs. Design can be found at: https://docs.google.com/document/d/1FWR3odjOw4v4-hjFy_hVaNdxHVs4WuK1asfB6M6XEMw/edit#heading=h.uubbjjd24plb. Closes #2314 GitOrigin-RevId: abafaa717ba8f7d8809114858c0fd2a25861fcb8
delta-io · Dec 20, 2023 · 2940429 · 2940429
1 parent feb1258
commit 2940429
Show file tree

Hide file tree

Showing 10 changed files with 1,148 additions and 47 deletions.
diff --git a/spark/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala b/spark/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableCommand.scala
@@ -391,7 +391,8 @@ class OptimizeExecutor(
       MultiDimClustering.cluster(
         input,
         approxNumFiles,
-        zOrderByColumns)
+        zOrderByColumns,
+        "zorder")
     } else {
       val useRepartition = sparkSession.sessionState.conf.getConf(
         DeltaSQLConf.DELTA_OPTIMIZE_REPARTITION_ENABLED)