[SPARK-57831][SQL] Align Hive-metastore compatibility classification for nanosecond-precision timestamp types#57084
Closed
stevomitric wants to merge 2 commits into
Closed
Conversation
…for nanosecond-precision timestamp types Classify nanosecond-precision timestamp types (TimestampNTZNanosType and TimestampLTZNanosType, precision 7-9) as Hive-incompatible in HiveExternalCatalog.isHiveCompatibleDataType, matching the existing treatment of TimestampNTZType. These are Spark-specific types with no Hive metastore equivalent, so tables using them must be persisted in Spark SQL specific format (real schema in table properties, dummy array<string> FieldSchema) rather than attempted as Hive-native columns. Without this, creating such a table throws a HiveException on the non-standard type string. Adds coverage in MetastoreDataSourcesSuite. Co-authored-by: Isaac
…compatible path The isHiveCompatibleDataType change also routes nanosecond timestamp *partition* columns through the incompatible-partition placeholder path (HiveClientImpl INCOMPATIBLE_PARTITION_TYPE_PLACEHOLDER), since createDataSourceTable computes incompatibleTypes over the full schema (data + partition columns). The existing tests only exercised regular columns. Add a create/reload test for a table partitioned on a nanos timestamp column, asserting (a) the partition column drives the Hive-incompatible WARN, (b) the raw HMS schema is the dummy array<string> (partition placeholder filtered out on read-back), and (c) the reloaded logical schema round-trips to the real nanos partition type at the end of the schema. Co-authored-by: Isaac
stevomitric
changed the title
Contributor
Author
|
cc @uros-b , @MaxGekk , @cloud-fan PTAL. |
cloud-fan
approved these changes
Jul 8, 2026
cloud-fan
left a comment
cloud-fan
left a comment
Contributor
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 0 nits.
Clean, minimal change. case _: AnyTimestampNanoType => false is consistent with the adjacent TimestampNTZType arm and captures both nanos types via the abstract parent; the three MetastoreDataSourcesSuite tests mirror the existing SPARK-37283 pattern, and tests 1/2 are load-bearing. LGTM.
cloud-fan
pushed a commit
that referenced
this pull request
Jul 9, 2026
…for nanosecond-precision timestamp types ### What changes were proposed in this pull request? Classify the nanosecond-precision timestamp types as Hive-incompatible in `HiveExternalCatalog.isHiveCompatibleDataType`. ### Why are the changes needed? These are Spark-specific types with no Hive metastore equivalent. Previously they fell through to the `case _ => true` default, so a table using them could take the Hive-compatible metastore path and store a non-standard `timestamp_ntz(9)` / `timestamp_ltz(9)` type string in the HMS `FieldSchema`. ### Does this PR introduce _any_ user-facing change? No. This affects only how nanosecond-typed tables are persisted to the Hive metastore. ### How was this patch tested? Added coverage in `MetastoreDataSourcesSuite` exercising create/reload of tables with nanosecond timestamp columns. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 Closes #57084 from stevomitric/stevomitric/spark-57831-nanos. Authored-by: Stevo Mitric <stevomitric2000@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 10d5950) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Classify the nanosecond-precision timestamp types as Hive-incompatible in
HiveExternalCatalog.isHiveCompatibleDataType.Why are the changes needed?
These are Spark-specific types with no Hive metastore equivalent. Previously they fell through to the
case _ => truedefault, so a table using them could take the Hive-compatible metastore path and store a non-standardtimestamp_ntz(9)/timestamp_ltz(9)type string in the HMSFieldSchema.Does this PR introduce any user-facing change?
No. This affects only how nanosecond-typed tables are persisted to the Hive metastore.
How was this patch tested?
Added coverage in
MetastoreDataSourcesSuiteexercising create/reload of tables with nanosecond timestamp columns.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8