Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix getting views for Hive 2.3+ metastore #833

Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -117,6 +117,9 @@
private final Duration maxRetryTime;
private final int maxRetries;

private volatile boolean metastoreKnownToSupportTableParamEqualsPredicate;

This comment has been minimized.

Copy link
@electrum

electrum May 31, 2019

Member

Maybe shorten KnownToSupport to Supports

This comment has been minimized.

Copy link
@findepi

findepi May 31, 2019

Author Member

Then i wouldn't be able to use false as an initial value (because initially i don't know yet).

private volatile boolean metastoreKnownToSupportTableParamLikePredicate;

@Inject
public ThriftHiveMetastore(MetastoreLocator metastoreLocator, ThriftHiveMetastoreConfig thriftConfig)
{
@@ -722,10 +725,7 @@ private void revokeRole(String role, String granteeName, PrincipalType granteeTy
.stopOn(UnknownDBException.class)
.stopOnIllegalExceptions()
.run("getAllViews", stats.getGetAllViews().wrap(() -> {
try (ThriftMetastoreClient client = clientProvider.createMetastoreClient()) {
String filter = HIVE_FILTER_FIELD_PARAMS + PRESTO_VIEW_FLAG + " = \"true\"";
return Optional.of(client.getTableNamesByFilter(databaseName, filter));
}
return Optional.of(getPrestoViews(databaseName));
}));
}
catch (UnknownDBException e) {
@@ -739,6 +739,48 @@ private void revokeRole(String role, String granteeName, PrincipalType granteeTy
}
}

private List<String> getPrestoViews(String databaseName)
throws TException
{
/*
* Thrift call `get_table_names_by_filter` may be translated by Metastore to a SQL query against Metastore database.
* Hive 2.3 on some databases uses CLOB for table parameter value column and some databases disallow `=` predicate over
* CLOB values. At the same time, they allow `LIKE` predicates over them.
*/
String filterWithEquals = HIVE_FILTER_FIELD_PARAMS + PRESTO_VIEW_FLAG + " = \"true\"";
String filterWithLike = HIVE_FILTER_FIELD_PARAMS + PRESTO_VIEW_FLAG + " LIKE \"true\"";

if (metastoreKnownToSupportTableParamEqualsPredicate) {
try (ThriftMetastoreClient client = clientProvider.createMetastoreClient()) {
return client.getTableNamesByFilter(databaseName, filterWithEquals);
}
}
if (metastoreKnownToSupportTableParamLikePredicate) {
try (ThriftMetastoreClient client = clientProvider.createMetastoreClient()) {
return client.getTableNamesByFilter(databaseName, filterWithLike);
}
}

try (ThriftMetastoreClient client = clientProvider.createMetastoreClient()) {
List<String> views = client.getTableNamesByFilter(databaseName, filterWithEquals);
metastoreKnownToSupportTableParamEqualsPredicate = true;
return views;
}
catch (TException | RuntimeException firstException) {
try (ThriftMetastoreClient client = clientProvider.createMetastoreClient()) {
List<String> views = client.getTableNamesByFilter(databaseName, filterWithLike);
metastoreKnownToSupportTableParamLikePredicate = true;
return views;
}
catch (TException | RuntimeException secondException) {
if (firstException != secondException) {

This comment has been minimized.

Copy link
@martint

martint May 31, 2019

Member

Why would firstException == secondException?

This comment has been minimized.

Copy link
@findepi

findepi May 31, 2019

Author Member

Technically you can keep an exception and rethrow it multiple times. I see no particular reason why this could happen here.
Apparently we often guard calls to java.lang.Throwable#addSuppressed like this (for apparently no particular reason), so I am blindly following an established convention 😉

This comment has been minimized.

Copy link
@electrum

electrum May 31, 2019

Member

It seems unlikely to happen in practice but is a good safety measure.

firstException.addSuppressed(secondException);
}
}
throw firstException;
}
}

@Override
public void createDatabase(Database database)
{
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.