Skip to content

Commit

Permalink
[SPARK-29260][SQL] Support ALTER DATABASE SET LOCATION if HMS suppo…
Browse files Browse the repository at this point in the history
…rts (apache#1436)

Currently for `ALTER DATABASE SET LOCATION` command, Spark will throw exception when Hive version (e.g., specified via `spark.sql.hive.metastore.version`) is not 3.0/3.1. This PR removes the check so that the command works as long as the Hive version used by the Hive metastore (which could be different from the version used by Spark) supports the alter database location feature added via [HIVE-8472](https://issues.apache.org/jira/browse/HIVE-8472). If it does not support it, the same exception will still be thrown from Spark side.

For the command `ALTER DATABASE SET LOCATION` command, Spark currently throws exception like the following:
```
AnalysisException: Hive 2.3.9 does not support altering database location
```

This is not accurate since it only considers the client version, while the feature support is on the Hive metastore server side. Therefore, the command should succeed if Spark is using Hive 2.3 while the remote Hive megastore is using Hive 3.1. On the other hand, the command will not succeed if Spark is using Hive 3.1 (thus no exception) but the remote Hive metastore is using 2.3.

Yes, previously Spark users using Hive client with version other than 3.0/3.1 won't be able to run `ALTER DATABASE SET LOCATION` command against Hive metastore 3.x. After this PR it should work.

Modified the existing test case.

Closes apache#36750 from sunchao/SPARK-29260.

Lead-authored-by: Chao Sun <sunchao@apple.com>
Co-authored-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Yuming Wang <yumwang@ebay.com>

Co-authored-by: Chao Sun <sunchao@apache.org>
  • Loading branch information
2 people authored and GitHub Enterprise committed Jun 3, 2022
1 parent 9abcd01 commit c00cc39
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -1615,8 +1615,8 @@ object QueryCompilationErrors {
new AnalysisException(s"$tableIdentifier should be converted to HadoopFsRelation.")
}

def alterDatabaseLocationUnsupportedError(version: String): Throwable = {
new AnalysisException(s"Hive $version does not support altering database location")
def alterDatabaseLocationUnsupportedError(): Throwable = {
new AnalysisException("Hive metastore does not support altering database location")
}

def hiveTableTypeUnsupportedError(tableType: String): Throwable = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -355,14 +355,17 @@ private[hive] class HiveClientImpl(
}

override def alterDatabase(database: CatalogDatabase): Unit = withHiveState {
if (!getDatabase(database.name).locationUri.equals(database.locationUri)) {
// SPARK-29260: Enable supported versions once it support altering database location.
if (!(version.equals(hive.v3_0) || version.equals(hive.v3_1))) {
throw QueryCompilationErrors.alterDatabaseLocationUnsupportedError(version.fullVersion)
}
}
val loc = getDatabase(database.name).locationUri
val changeLoc = !database.locationUri.equals(loc)

val hiveDb = toHiveDatabase(database)
client.alterDatabase(database.name, hiveDb)

if (changeLoc && getDatabase(database.name).locationUri.equals(loc)) {
// Some Hive versions don't support changing database location, so we check here to see if
// the location is actually changed, and throw an error if not.
throw QueryCompilationErrors.alterDatabaseLocationUnsupportedError()
}
}

private def toHiveDatabase(
Expand Down

0 comments on commit c00cc39

Please sign in to comment.