feat(csharp/src/Drivers/Databricks): Optimize GetColumnsExtendedAsync via DESC TABLE EXTENDED #2953

jackyhu-db · 2025-06-11T00:06:08Z

Optimize GetColumnsExtendedAsync in Databricks via `DESC TABLE EXTENDED`

Motivation

Currently, HiveServer2Statement.GetColumnsExtendedAsync will make 3 thrift calls GetColumns, GetPrimaryKeys and GetCrossReference to get all the information and then join them into one result set. Now Databricks introduces a SQL DESC TABLE EXTENDED <table> AS JSON that will return all of these information in one query, this can save 2 extra roundtrips and improve the performance.

Description

Override HiveServer2Statement.GetColumnsExtendedAsync in DatabricksStatement by executing single SQL DESC TABLE EXTENDED <table> AS JSON to get all the required column info for GetColumnsExtendedAsync then combine and join these info into the result. As this SQL DESC TABLE EXTENDED <table> AS JSON is only available in Databricks Runtime 16.2 or above, it will check the ServerProtocolVersion. if it does not meet the requirement, it will fallback the implementation of base class.

Change

Added DescTableExtendedResult that represents the result of DESC TABLE EXTENDED <table> AS JSON (see format here). It also
- Parses the string-based table_constraints to the structured primary key and foreign key constraints
- Adds calculated properties DataType, ColumnSize, FullTypeName which are calculated from the column type and type specific properties
Changed the access modifiers of some properties and methods in HiveServer2Statement to protected so that they can be used/overridden in subclass DatabricksStatement
- PrimaryKeyFields
- ForeignKeyFields
- PrimaryKeyPrefix
- ForeignKeyPrefix
- GetColumnsExtendedAsync
- CreateEmptyExtendedColumnsResult
Updated DatabricksStatement with the changes below
- Added GetColumnsExtendedAsync in DatabricksStatement, it executes query DESC TABLE EXTENDED <table> AS JSON to get all the required info and then json them into the QueryResult
- Moved the column metadata schema creation logic from GetColumnsAsync to a reusable method CreateColumnMetadataSchema
- Moved the column metadata data array creation logic from GetColumnsAsync to a reusable method CreateColumnMetadataEmptyArray
Added a Databricks connection Parameter adbc.databricks.use_desc_table_extended
Added an internal calculated property CanUseDescTableExtended in DatabricksConnection, DatabricksStatement will call it to decide if it will override GetColumnsExtendedAsync
Improved the Databricks driver integration test StatementTest:CanGetColumnsExtended
- Setup the required table resources during the test instead of depending on manual setup before the test
- Support running multiple test cases from the inputs
- Add a deep result check to make sure all the properties of all the columns are set correctly

Testing

Added Test class DescTableExtendedResultTest to cover all the deserialization and parsing cases from raw result of DESC TABLE EXTENDED <table> AS JSON
Tested all the ColumnsExtended relevant test cases in Databricks/StatementTest

…xtended to implement GetColumnsExtendedAsync

eric-wang-1990 · 2025-06-11T21:07:06Z

csharp/test/Drivers/Databricks/Result/DescTableExtendedResultTest.cs

+* limitations under the License.
+*/
+
+using Apache.Arrow.Adbc.Drivers.Databricks.Result;


Can we mocify/improve the existing GetColumnExtended tests to verify that:
With the flag on/off, they get same number of columns, same schema, and same data?

CurtHagenlocher

Thanks, looks good! I've just got a few nits about formatting and casing of method names.

CurtHagenlocher · 2025-06-19T14:17:11Z

csharp/src/Drivers/Apache/Hive2/HiveServer2Statement.cs

            }

+


nit: extra blank line

CurtHagenlocher · 2025-06-19T14:17:46Z

csharp/src/Drivers/Databricks/DatabricksConnection.cs

+        /// </summary>
+        internal bool CanUseDescTableExtended => _useDescTableExtended && ServerProtocolVersion != null && ServerProtocolVersion >= TProtocolVersion.SPARK_CLI_SERVICE_PROTOCOL_V7;
+
+


nit: extra blank line

CurtHagenlocher · 2025-06-19T14:19:04Z

csharp/src/Drivers/Databricks/DatabricksStatement.cs

+            return string.Join(".", parts);
+        }
+
+


nit: extra blank line

CurtHagenlocher · 2025-06-19T14:20:58Z

csharp/src/Drivers/Databricks/DatabricksStatement.cs

+            return CreateExtendedColumnsResult(columnMetadataSchema,result);
+        }
+
+


nit: extra blank line

CurtHagenlocher · 2025-06-19T14:22:40Z

csharp/src/Drivers/Databricks/DatabricksStatement.cs

+            ];
+        }
+
+


nit: extra blank line

CurtHagenlocher · 2025-06-19T14:25:18Z

csharp/src/Drivers/Databricks/Result/DescTableExtendedResult.cs

+using System.Text.RegularExpressions;
+using static Apache.Arrow.Adbc.Drivers.Apache.Hive2.HiveServer2Connection;
+
+


nit: extra blank lines here and at line 35

CurtHagenlocher · 2025-06-19T14:30:44Z

csharp/src/Drivers/Databricks/Result/DescTableExtendedResult.cs

+                }
+            }
+
+            private int getIntervalSize()


Suggested change

private int getIntervalSize()

private int GetIntervalSize()

CurtHagenlocher · 2025-06-19T14:31:39Z

csharp/src/Drivers/Databricks/Result/DescTableExtendedResult.cs

+            }
+        }
+
+        private void parseConstraints()


Suggested change

private void parseConstraints()

private void ParseConstraints()

CurtHagenlocher · 2025-06-19T14:33:34Z

csharp/src/Drivers/Databricks/Result/DescTableExtendedResult.cs

+            // Constraints string format example:
+            // "[ (pk_constraint, PRIMARY KEY (`col1`, `col2`)), (fk_constraint, FOREIGN KEY (`col3`) REFERENCES `catalog`.`schema`.`table` (`refcol1`, `refcol2`)) ]"
+
+            var constraintString = TableConstraints.Trim();


when not null, we're trimming twice. Consider instead doing

var constraintString = TableConstraints?.Trim(); if (constraintString == null || constraintString.Length == 0)

jackyhu-db added 13 commits June 8, 2025 19:11

Implement GetColumnsExtendedAsync via DESC TABLE EXTENDED

19f6903

Add DescTableExtendedResult

fc89b32

Change to use DescTableExtendedResult

3bcf3e2

Implement CreateExtendedColumnsResult

55ceff4

Fix executeQuery and get E2E work

4621acb

Add DataType mapping support

99142b6

Add IsNumber in DescTableExtendedResult to set numPrecRadix

80e9634

Support column size

aee8bb2

Support FullTypeName

28b7c4e

Fix autoInremental column

8be918a

Add ServerProtocolVersion property in HiveServer2Connection

c4b4199

Add a protocol version check and flag check before using desc table e…

8bf1d19

…xtended to implement GetColumnsExtendedAsync

Merge remote-tracking branch 'adbc/main' into desc-table-extended

4f04d8d

jackyhu-db requested a review from CurtHagenlocher as a code owner June 11, 2025 00:06

github-actions bot modified the milestone: ADBC Libraries 19 Jun 11, 2025

Fix format issues

7d83b75

jackyhu-db force-pushed the desc-table-extended branch from 81b326f to 7d83b75 Compare June 11, 2025 02:05

jackyhu-db added 2 commits June 11, 2025 12:36

Add fallback when query execution fails

c9fa98f

Add CHAR and VARCHAR length support

8c73c0b

jackyhu-db force-pushed the desc-table-extended branch from 49b3f6e to 8c73c0b Compare June 11, 2025 04:36

Fix some properties of some column types

a6980a1

eric-wang-1990 reviewed Jun 11, 2025

View reviewed changes

jackyhu-db added 2 commits June 12, 2025 09:17

Fix column size for decimal type

f2d5747

Add FK_NAME and KEQ_SEQ fields

d9fd6e8

jackyhu-db force-pushed the desc-table-extended branch 2 times, most recently from 4d4af63 to 4a0d05a Compare June 12, 2025 03:53

Improve Integration test Databricks CanGetColumnsExtended

bb93562

jackyhu-db force-pushed the desc-table-extended branch from 4a0d05a to bb93562 Compare June 12, 2025 04:19

Fix a hard-code ref table name

f7cf526

jackyhu-db added 4 commits June 13, 2025 11:03

Change Strng column size to Int.Max

241d75a

Set column size 0 if column type does not have column size

5b18938

Add space between key/value in Map type name

34dc30d

Fix column size for interval type

03220d2

CurtHagenlocher requested changes Jun 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(csharp/src/Drivers/Databricks): Optimize GetColumnsExtendedAsync via DESC TABLE EXTENDED #2953

feat(csharp/src/Drivers/Databricks): Optimize GetColumnsExtendedAsync via DESC TABLE EXTENDED #2953

jackyhu-db commented Jun 11, 2025 •

edited

Loading

Uh oh!

eric-wang-1990 Jun 11, 2025

Uh oh!

CurtHagenlocher left a comment

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

CurtHagenlocher Jun 19, 2025

Uh oh!

Uh oh!

		/// </summary>
		internal bool CanUseDescTableExtended => _useDescTableExtended && ServerProtocolVersion != null && ServerProtocolVersion >= TProtocolVersion.SPARK_CLI_SERVICE_PROTOCOL_V7;

		return CreateExtendedColumnsResult(columnMetadataSchema,result);
		}

		using System.Text.RegularExpressions;
		using static Apache.Arrow.Adbc.Drivers.Apache.Hive2.HiveServer2Connection;

	private void parseConstraints()
	private void ParseConstraints()

feat(csharp/src/Drivers/Databricks): Optimize GetColumnsExtendedAsync via DESC TABLE EXTENDED #2953

Are you sure you want to change the base?

feat(csharp/src/Drivers/Databricks): Optimize GetColumnsExtendedAsync via DESC TABLE EXTENDED #2953

Conversation

jackyhu-db commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Optimize GetColumnsExtendedAsync in Databricks via DESC TABLE EXTENDED

Motivation

Description

Change

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CurtHagenlocher left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jackyhu-db commented Jun 11, 2025 •

edited

Loading

Optimize GetColumnsExtendedAsync in Databricks via `DESC TABLE EXTENDED`