Skip to content

[SPARK-53095][SQL] Support HMS v4.1 #51809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

[SPARK-53095][SQL] Support HMS v4.1 #51809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0222e1d
Support HMS v4.1
yaooqinn Aug 4, 2025
125825d
Support HMS v4.1
yaooqinn Aug 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/sql-data-sources-hive-tables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ The following options can be used to configure the version of Hive that is used
<td><code>2.3.10</code></td>
<td>
Version of the Hive metastore. Available
options are <code>2.0.0</code> through <code>2.3.10</code>, <code>3.0.0</code> through <code>3.1.3</code>, and <code>4.0.0</code> through <code>4.0.1</code>.
options are <code>2.0.0</code> through <code>2.3.10</code>, <code>3.0.0</code> through <code>3.1.3</code>, and <code>4.0.0</code> through <code>4.1.0</code>.
</td>
<td>1.4.0</td>
</tr>
Expand Down
2 changes: 1 addition & 1 deletion docs/sql-migration-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1059,7 +1059,7 @@ Python UDF registration is unchanged.
Spark SQL is designed to be compatible with the Hive Metastore, SerDes and UDFs.
Currently, Hive SerDes and UDFs are based on built-in Hive,
and Spark SQL can be connected to different versions of Hive Metastore
(from 2.0.0 to 2.3.10 and 3.0.0 to 3.1.3. Also see [Interacting with Different Versions of Hive Metastore](sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore)).
(from 2.0.0 to 2.3.10 and 3.0.0 to 4.1.0. Also see [Interacting with Different Versions of Hive Metastore](sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore).

#### Deploying in Existing Hive Warehouses
{:.no_toc}
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ private[spark] object HiveUtils extends Logging {
.doc("Version of the Hive metastore. Available options are " +
"<code>2.0.0</code> through <code>2.3.10</code>, " +
"<code>3.0.0</code> through <code>3.1.3</code> and " +
"<code>4.0.0</code> through <code>4.0.1</code>.")
"<code>4.0.0</code> through <code>4.1.0</code>.")
.version("1.4.0")
.stringConf
.checkValue(isCompatibleHiveVersion, "Unsupported Hive Metastore version")
Expand Down
1 change: 1 addition & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ private[hive] class HiveClientImpl(
case hive.v3_0 => new Shim_v3_0()
case hive.v3_1 => new Shim_v3_1()
case hive.v4_0 => new Shim_v4_0()
case hive.v4_1 => new Shim_v4_1()
}

// Create an internal session state for this HiveClientImpl.
Expand Down
2 changes: 2 additions & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1543,3 +1543,5 @@ private[client] class Shim_v4_0 extends Shim_v3_1 {
renamePartitionMethod.invoke(hive, table, oldPartSpec, newPart, writeIdInLoadTableOrPartition)
}
}

private[client] class Shim_v4_1 extends Shim_v4_0
1 change: 1 addition & 0 deletions sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ private[hive] object IsolatedClientLoader extends Logging {
case (3, 0, _) => Some(hive.v3_0)
case (3, 1, _) => Some(hive.v3_1)
case (4, 0, _) => Some(hive.v4_0)
case (4, 1, _) => Some(hive.v4_1)
case _ => None
}.getOrElse {
throw QueryExecutionErrors.unsupportedHiveMetastoreVersionError(
Expand Down
19 changes: 18 additions & 1 deletion sql/hive/src/main/scala/org/apache/spark/sql/hive/client/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,25 @@ package object client {
"org.pentaho:pentaho-aggdesigner-algorithm",
"org.apache.hive:hive-vector-code-gen"))

case object v4_1 extends HiveVersion("4.1.0",
extraDeps =
"org.antlr:antlr4-runtime:4.9.3" ::
"org.apache.derby:derby:10.14.1.0" ::
"org.apache.hadoop:hadoop-hdfs:3.4.1" ::
"org.datanucleus:datanucleus-api-jdo:6.0.5" ::
"org.datanucleus:datanucleus-core:6.0.11" ::
"org.datanucleus:datanucleus-rdbms:6.0.10" ::
"org.datanucleus:javax.jdo:3.2.1" ::
"org.springframework:spring-core:5.3.39" ::
"org.springframework:spring-jdbc:5.3.39" :: Nil,
exclusions =
"org.apache.curator:*" ::
"org.apache.hive:hive-service-rpc" ::
"org.apache.tez:tez-api" ::
"org.apache.zookeeper:zookeeper" :: Nil)

val allSupportedHiveVersions: Set[HiveVersion] =
Set(v2_0, v2_1, v2_2, v2_3, v3_0, v3_1, v4_0)
Set(v2_0, v2_1, v2_2, v2_3, v3_0, v3_1, v4_0, v4_1)
}
// scalastyle:on

Expand Down
8 changes: 5 additions & 3 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ class HiveClientSuite(version: String) extends HiveVersionSuite(version) {

private val emptyDir = Utils.createTempDir().getCanonicalPath

private val ver = IsolatedClientLoader.hiveVersion(version)

/**
* Drops table `tableName` after calling `f`.
*/
Expand Down Expand Up @@ -166,7 +168,7 @@ class HiveClientSuite(version: String) extends HiveVersionSuite(version) {
// test alter database location
val tempDatabasePath2 = Utils.createTempDir().toURI
// Hive support altering database location since HIVE-8472.
if (version == "3.0" || version == "3.1" || version == "4.0") {
if (ver.compare(hive.v3_0) >= 0) {
client.alterDatabase(database.copy(locationUri = tempDatabasePath2))
val uriInCatalog = client.getDatabase("temporary").locationUri
assert("file" === uriInCatalog.getScheme)
Expand Down Expand Up @@ -336,7 +338,7 @@ class HiveClientSuite(version: String) extends HiveVersionSuite(version) {
}

test("listTables(database)") {
assert(client.listTables("default") === Seq("src", "temporary", "view1"))
assert((client.listTables("default") diff Seq("src", "temporary", "view1")) === Nil)
Copy link
Contributor

@LuciferYang LuciferYang Aug 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is a change needed here?

Copy link
Member Author

@yaooqinn yaooqinn Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return order changes(haven't known the cause)

}

test("listTables(database, pattern)") {
Expand Down Expand Up @@ -579,7 +581,7 @@ class HiveClientSuite(version: String) extends HiveVersionSuite(version) {

test("sql create index and reset") {
// HIVE-18448 Since Hive 3.0, INDEX is not supported.
if (version != "3.0" && version != "3.1" && version != "4.0") {
if (ver.compare(hive.v3_0) < 0) {
client.runSqlHive("CREATE TABLE indexed_table (key INT)")
client.runSqlHive("CREATE INDEX index_1 ON TABLE indexed_table(key) " +
"as 'COMPACT' WITH DEFERRED REBUILD")
Expand Down
2 changes: 1 addition & 1 deletion sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientVersions.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ private[client] trait HiveClientVersions {
protected val versions = if (testVersions.nonEmpty) {
testVersions.get.split(",").map(_.trim).filter(_.nonEmpty).toIndexedSeq
} else {
IndexedSeq("2.0", "2.1", "2.2", "2.3", "3.0", "3.1", "4.0")
IndexedSeq("2.0", "2.1", "2.2", "2.3", "3.0", "3.1", "4.0", "4.1")
}
}