Skip to content

Add support for using Delta Lake vacuum() and Delta Lake optimize() through Arrow Flight #239

Description

@CGodiksen

Delta Lake has functionality for removing stale files and optimizing files, however since we provide a layer over Delta Lake, this functionality is not available to the user of ModelarDB. Actions should be added to Arrow Flight that makes it possible to use this functionality.

When vacuuming or optimizing a table, the metadata Delta Lake should also be vacuumed or optimized since this is cheap and we do not want the user to have to manually vacuum or optimize the metadata Delta Lake.

  • Add new endpoint to server Apache Arrow Flight server to vacuum a table.
  • Add new endpoint to server Apache Arrow Flight server to optimize a table.
  • Consider how to handle passing extra arguments such as retention period to endpoint.
  • Add new endpoint to manager Apache Arrow Flight server to vacuum a table in the entire cluster.
  • Add new endpoint to manager Apache Arrow Flight server to optimize a table in the entire cluster.
  • Consider how to handle passing extra arguments to optimize endpoint.
  • Consider if we should have endpoints to vacuum or optimize all tables.
  • Add integration tests.
  • Add unit tests.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions