Add support for manually optimizing a table

[CGodiksen]

This PR closes #239 by adding a new method to DataFolder, adding OPTIMIZE [CLUSTER] [table_name[, table_name]+] [TARGET num_bytes] to the SQL parser, adding a new method to Cluster, adding the structure in Context and the Apache Arrow Flight server to call optimize, and adding optimize to modelardb_embedded and the Python library.

Unit testing have also been added in the relevant places and three new integration tests have been added to ensure the Apache Arrow Flight endpoint works. Note that the implementation matches vacuum 1-to-1.

[Copilot]

Pull request overview

Adds end-to-end support for manually optimizing (compacting) Delta Lake tables via a new OPTIMIZE [CLUSTER] ... [TARGET ...] SQL statement, exposed through the Arrow Flight server, embedded client, C API, and Python bindings.

Changes:

• Extend the SQL parser/dialect to recognize OPTIMIZE (including CLUSTER and TARGET).
• Add storage/server/cluster execution paths to run optimize locally or across a cluster via Flight do_get().
• Add embedded/C/Python APIs and unit/integration tests covering optimize behavior and error cases.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
crates/modelardb_storage/src/parser.rs	Adds OPTIMIZE statement parsing and statement mapping into ModelarDbStatement.
crates/modelardb_storage/src/data_folder/mod.rs	Introduces DataFolder::optimize_table() and unit tests validating file compaction behavior.
crates/modelardb_server/tests/integration_test.rs	Adds Flight-level integration tests for optimizing tables and missing-table errors.
crates/modelardb_server/src/remote.rs	Wires OPTIMIZE into Flight do_get() and adds optimize_tables() handler logic.
crates/modelardb_server/src/context.rs	Adds Context::optimize_table() plus related unit tests.
crates/modelardb_server/src/cluster.rs	Adds cluster-wide optimize support (optimize_cluster_tables).
crates/modelardb_embedded/src/operations/mod.rs	Extends embedded Operations trait with optimize().
crates/modelardb_embedded/src/operations/data_folder.rs	Implements optimize() for embedded DataFolder and adds unit tests.
crates/modelardb_embedded/src/operations/client.rs	Implements optimize() for embedded Flight Client.
crates/modelardb_embedded/src/capi.rs	Adds C ABI entrypoint modelardb_embedded_optimize.
crates/modelardb_embedded/bindings/python/tests/test_operations.py	Adds Python binding tests for optimize success/error.
crates/modelardb_embedded/bindings/python/modelardb/operations.py	Exposes optimize() in Python API via the C ABI.
crates/modelardb_embedded/bindings/c/modelardb_embedded.h	Declares the new C API function for optimize.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.