[SPARK-53839][SQL] Add V2 table support to DESCRIBE TABLE AS JSON

[vinodkc]

What changes were proposed in this pull request?

This PR extends DESCRIBE TABLE AS JSON support to DataSource V2 tables

Why are the changes needed?

Prior to this PR, DESCRIBE TABLE AS JSON only supported DataSource V1 tables. Users can now access metadata in json format for both V1 and V2 tables using the same SQL syntax

Does this PR introduce any user-facing change?

Yes. Users can now use DESCRIBE TABLE AS JSON with DataSource V2 tables

CREATE TABLE v2_table (id INT, name STRING) USING delta;

DESCRIBE EXTENDED v2_table AS JSON;
-- Returns:
{
  "table_name": "v2_table",
  "catalog_name": "spark_catalog",
  "namespace": ["default"],
  "schema_name": "default",
  "columns": [
    {"name": "id", "type": {"name": "int"}, "nullable": true},
    {"name": "name", "type": {"name": "string"}, "nullable": true}
  ],
  "type": "MANAGED",
  "provider": "delta",
  "location": "file:/tmp/warehouse/v2_table",
  "capabilities": ["BATCH_READ", "BATCH_WRITE", "STREAMING_WRITE"],
  "table_properties": {
    "delta.minReaderVersion": "3",
    "delta.minWriterVersion": "7"
  }
}

V2-Specific JSON Fields:
The JSON output for V2 tables includes additional fields not available for V1 tables:

• capabilities: Array of table capabilities (e.g., ["BATCH_READ", "BATCH_WRITE"])
• partitioning: Array of partition transforms with type and columns
• metadata_columns: Array of metadata columns with schema information
• constraints: Array of table constraints (primary key, check constraints)
• statistics: Object with size_in_bytes and num_rows when available

How was this patch tested?

Added Tests in DescribeTableSuite.scala

Was this patch authored or co-authored using generative AI tooling?

No.

[asl3]

Could you add tests which include the statistics and constraints output?