[spark] Support partition statistics in SHOW TABLE EXTENDED PARTITION command

[kerwin-zk]

Purpose

Examples

CREATE TABLE db.orders (
  id INT,
  amount DOUBLE,
  dt STRING
) PARTITIONED BY (dt);

INSERT INTO db.orders VALUES 
  (1, 10.5, '2025-01-01'),
  (2, 20.0, '2025-01-01'),
  (3, 15.0, '2025-01-02');

Before

SHOW TABLE EXTENDED IN db LIKE 'orders' PARTITION(dt = '2025-01-01');

Partition Values: [dt=2025-01-01]

Only partition values were displayed. The TODO "Partition Parameters", "Created Time", "Last Access", "Partition Statistics" was left unimplemented, and loadPartitionMetadata always returned an empty map.

After

SHOW TABLE EXTENDED IN db LIKE 'orders' PARTITION(dt = '2025-01-01');

Partition Values: [dt=2025-01-01]
Partition Parameters: {recordCount=2, fileSizeInBytes=741, fileCount=1, lastFileCreationTime=1744105200000}
Partition Statistics: 2 rows, 741 bytes

SHOW TABLE EXTENDED IN db LIKE 'orders' PARTITION(dt = '2025-01-02');

Partition Values: [dt=2025-01-02]
Partition Parameters: {recordCount=1, fileSizeInBytes=538, fileCount=1, lastFileCreationTime=1744105200000}
Partition Statistics: 1 rows, 538 bytes

Tests

CI

[Copilot]

Pull request overview

Adds partition-level statistics to Spark’s SHOW TABLE EXTENDED ... PARTITION(...) output for Paimon tables by wiring SupportsPartitionManagement.loadPartitionMetadata to real partition stats and surfacing them in the Spark 3 command implementation.

Changes:

• Implement loadPartitionMetadata in PaimonPartitionManagement to return partition stats (record count, file size, file count, last file creation time) from snapshot partition entries.
• Update Spark 3 PaimonShowTablePartitionCommand to display “Partition Parameters” and a human-readable “Partition Statistics” line.
• Extend unit tests to assert that partition parameters (and recordCount values) appear in SHOW TABLE EXTENDED ... PARTITION(...) output.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
paimon-spark/paimon-spark3-common/src/main/scala/org/apache/paimon/spark/commands/PaimonShowTablePartitionCommand.scala	Formats and prints partition metadata and derived partition statistics in SHOW TABLE EXTENDED (Spark 3 path).
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonPartitionManagement.scala	Implements loadPartitionMetadata by reading partition entries from the snapshot reader and returning stats as metadata.
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/DescribeTableTestBase.scala	Adds assertions that SHOW TABLE EXTENDED PARTITION output includes partition parameter keys and expected recordCount values.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[YannByron]

+1