[Feature][Spark]: Support drop partition

[hellojinsilei]

Description

Now Arctic Spark can not drop partitions, drop partitions is a necessary feature
Hope can support SQL like this:
ALTER TABLE table_identifier DROP [ IF EXISTS ] partition_spec [PURGE]

Use case/motivation

No response

Describe the solution

No response

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hellojinsilei]

What are the semantics of arctic drop partition?

In Iceberg, It only has the operation to drop the partition field, and dropping a partition field is a metadata operation and does not change any of the existing table data.
In Hive, dropping partition will delete data files and the table structure information.

Do we use hive tables to refer to the implementation of Hive, while non-hive tables refer to the implementation method of Iceberg? But how do we ensure that the table structure is consistent in this way? Or is there a better way？
@zhoujinsong @baiyangtx cc

[baiyangtx]

What are the semantics of arctic drop partition?

In Iceberg, It only has the operation to drop the partition field, and dropping a partition field is a metadata operation and does not change any of the existing table data. In Hive, dropping partition will delete data files and the table structure information.

Do we use hive tables to refer to the implementation of Hive, while non-hive tables refer to the implementation method of Iceberg? But how do we ensure that the table structure is consistent in this way? Or is there a better way？ @zhoujinsong @baiyangtx cc

I think the drop partition operation in arctic should convert to delete from table where pt=xxx, just simply remove files reference in the snapshot, and remove partition metadata in HMS, considering this is a DDL statement, so arctic should not write the EQ_DELETE_FILE in change store.

The Iceberg optimize rule RewriteDelete handle the delete statement, if it is a metadata delete, iceberg will not write any file.