fix: filter EXTERNAL property in SparkCatalogMetaStoreClient.toCatalogTable

[prashantwason]

Describe the issue this Pull Request addresses

Hudi's HMSDDLExecutor.createTable (HMSDDLExecutor.java#L128-L131) sets BOTH of the following on the Hive Table object when the table is external:

if (!syncConfig.getBoolean(HIVE_CREATE_MANAGED_TABLE)) {
  newTb.putToParameters("EXTERNAL", "TRUE");          // ← duplicate encoding
  newTb.setTableType(TableType.EXTERNAL_TABLE.toString());
}

When that Table flows through SparkCatalogMetaStoreClient.createTable → toCatalogTable → HiveExternalCatalog.createTable, HiveExternalCatalog.verifyTableProperties throws:

org.apache.spark.sql.AnalysisException: Cannot set or change the preserved property key: 'EXTERNAL'
  at org.apache.spark.sql.hive.HiveExternalCatalog.verifyTableProperties(HiveExternalCatalog.scala:147)
  ...
  at org.apache.spark.sql.hive.SparkCatalogMetaStoreClient.createTable(SparkCatalogMetaStoreClient.scala:59)
  at org.apache.hudi.hive.ddl.HMSDDLExecutor.createTable(HMSDDLExecutor.java:136)

Spark uses CatalogTableType.EXTERNAL on the CatalogTable itself to encode external-ness, and rejects EXTERNAL=... as a duplicate (and forbidden) encoding. So Hudi's normal HMSDDLExecutor-shaped Table cannot be created at all when SparkCatalogMetaStoreClient is in use (i.e. hoodie.datasource.hive_sync.use_spark_catalog=true), unless the user explicitly sets HIVE_CREATE_MANAGED_TABLE=true.

Summary and Changelog

• SparkCatalogMetaStoreClient.toCatalogTable: extend the existing filter that strips spark.sql.* keys to also strip EXTERNAL. The tableType field below already maps EXTERNAL_TABLE → CatalogTableType.EXTERNAL, so dropping the redundant property is safe and avoids the verifyTableProperties rejection.
• TestSparkCatalogMetaStoreClient: add a regression test mirroring the real HMSDDLExecutor shape (tableType=EXTERNAL_TABLE AND parameters[EXTERNAL]=TRUE).

Same family as #18654 (filter spark.sql.*).

Impact

Restores the ability to use SparkCatalogMetaStoreClient with the default Hive-sync codepath (external tables). No previously-successful path changes behavior — this only converts a previously-thrown exception into a successful create on the same exact input shape.

Risk Level

low — one-line filter extension mirroring existing logic, covered by a new unit test.

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable
• CI passed

🤖 Generated with Claude Code

[hudi-agent]

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for the contribution! This PR strips the redundant EXTERNAL parameter in SparkCatalogMetaStoreClient.toCatalogTable so tables created via HMSDDLExecutor flow through Spark's HiveExternalCatalog.verifyTableProperties without tripping the reserved-key check, with a regression test mirroring the real call shape. No correctness issues found. A few style/readability suggestions in the inline comments. Please take a look, and this should be ready for a Hudi committer or PMC member to take it from here. One suggestion below: the new test could be strengthened with an explicit assertion that the EXTERNAL property was actually stripped from the stored table parameters.

cc @yihua

[hudi-agent]

🤖 nit: the test verifies createTable doesn't throw and that comment survives, but it never explicitly asserts that EXTERNAL was removed. Could you add something like assertFalse(client.getTable(databaseName, tableName).getParameters.containsKey("EXTERNAL")) to directly validate the actual fix?

_{- AI-generated; verify before applying. React 👍/👎 to flag quality.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 68.08%. Comparing base (38db5ed) to head (f8f8716).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
...e/spark/sql/hive/SparkCatalogMetaStoreClient.scala	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18672      +/-   ##
============================================
+ Coverage     68.06%   68.08%   +0.01%     
+ Complexity    28922    28919       -3     
============================================
  Files          2518     2519       +1     
  Lines        140574   140611      +37     
  Branches      17419    17423       +4     
============================================
+ Hits          95682    95731      +49     
+ Misses        37036    37022      -14     
- Partials       7856     7858       +2     

Flag	Coverage Δ
common-and-other-modules	44.35% <0.00%> (-0.02%)	⬇️
hadoop-mr-java-client	44.95% <ø> (+0.06%)	⬆️
spark-client-hadoop-common	48.44% <ø> (+0.01%)	⬆️
spark-java-tests	48.61% <0.00%> (-0.02%)	⬇️
spark-scala-tests	44.75% <0.00%> (+0.05%)	⬆️
utilities	37.67% <0.00%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...e/spark/sql/hive/SparkCatalogMetaStoreClient.scala	30.07% <0.00%> (ø)

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hudi-bot]

CI report:

• f8f8716 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build