|
Does lakehouse.run_table_maintenance support Fast Optimize? I think the answer is yes, but I don't know for sure based on the doc and api docs.
|
Replies: 1 comment 2 replies
|
@mwc360 may know better. |
|
@mwc360 @m-kovalsky - wondering if you had any information on this? |
|
this method calls the table maintenance API which submits a spark job and waits for completion. Fast optimize is intentionally not enabled by the table maintenance API. The reason is that fast optimize skips compacting small files when it's not estimated to produce a large enough file. For many workloads it's more efficient to minimize write amplification and wait till either there's too many small files (we default it to 50) or wait till the sum of small files is estimated to produce a large enough small file (>64MB if Adaptive Target File Size is enabled). With the table maintenance API, since it is starting a dedicated session to do compaction, it would be a waste to short circuit the operation and do nothing. There's overhead to starting a session and initializing spark/delta internals, since nothing else is running in that session it makes more sense to allow it to proceed with doing any compaction work. Fast Optimize is designed for workloads where you run a bunch of OPTIMIZE jobs or interleave ELT code with maintenance code. FYI - IF you are calling this sempy method from a Spark job, you may want to consider just enabling Auto Compaction instead. The reason is that this method is synchronous, it will wait for the other spark session to start, complete the job, and report status. While this is taking place, the calling compute is probably doing nothing else (unless you are running a bunch of fancy multithreading). Instead, enable Auto Compaction and when you write to tables it will automatically eval if there's too many small files (> 50) and if so will do a lightweight compaction job on the same compute. Hope this helps! |
this method calls the table maintenance API which submits a spark job and waits for completion. Fast optimize is intentionally not enabled by the table maintenance API. The reason is that fast optimize skips compacting small files when it's not estimated to produce a large enough file. For many workloads it's more efficient to minimize write amplification and wait till either there's too many small files (we default it to 50) or wait till the sum of small files is estimated to produce a large enough small file (>64MB if Adaptive Target File Size is enabled). With the table maintenance API, since it is starting a dedicated session to do compaction, it would be a waste to short circuit the …