Spark table purges and some procedures fail with encrypted tables due to missing metadata table support

[smaheshwar-pltr]

Feature Request / Improvement

#14763 shows a failing test.

The Spark purge codepath and other maintenance procedures require finding reachable files, using metadata tables. This doesn't work for encrypted tables currently.

Concretely, there are a couple of issues:

1. The AllManifestsTableScan dispatches manifest lists reads to executors, however the StandardEncryptionManager's fields required for that are not serializable
2. Finding content files requires reading manifests obtained from the manifest metadata tables here, but the ManifestFileBean does not include encryption key metadata to permit decryption during the read

Query engine

None

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[smaheshwar-pltr]

We could include key metadata in the the manifest table for (2) and make the encryption manager's state serializable to allow manifest list reads to be dispatched for (1), @tom-s-powell has #14751 and #14750 🙌

If we insist on KMS interaction occurring on driver, we can alternatively eagerly decrypt manifest lists on the driver with something like https://github.com/apache/iceberg/compare/main...smaheshwar-pltr:iceberg:sm/spark-purge-encryption?expand=1 also but maybe feels unnecessary.

cc @ggershinsky @huaxingao @szlta

[ggershinsky]

Makes sense, the security model of Iceberg table encryption does not preclude serialization of the encryption manager and the kms client.