[Bug]: Failed to list tables from Hive 4 Metastore in Hadoop2 Build

[owenmonn]

What happened?

When building Amoro with the Hadoop2 profile, an error occurs when attempting to list tables from a Hive 4 Metastore catalog. This is due to changes in the Hive Metastore Client API specifications in the currently used Hive dependency (2.3.9) for the Hadoop2 build. This is related to the deprecation of older Hive Metastore APIs (HIVE-26537).
I will submit a PR to change the Hive dependency for the Hadoop2 build to 2.3.8.

Affects Versions

master

What table formats are you seeing the problem on?

No response

What engines are you seeing the problem on?

AMS

How to reproduce

1. Build Amoro with the Hadoop2 profile.
2. In the AMS web interface, register a Hive 4 catalog (Type: External, Metastore: Hive Metastore).
3. In the AMS web interface, navigate to the Tables tab and select the registered catalog (to list tables).

Relevant log output

2025-04-03 04:15:11,896 ERROR [JettyServerThreadPool-56909] [org.apache.amoro.server.dashboard.DashboardServer] [] - An error occurred while processing the url:http://myhost/ams/v1/catalogs/bigdata_dev_iceberg/databases/default/tables
java.lang.RuntimeException: Failed to list all tables under namespace default
    at org.apache.iceberg.hive.HiveCatalog.listTables(HiveCatalog.java:158) ~[iceberg-hive-metastore-1.4.3.jar:?]
    at org.apache.iceberg.CachingCatalog.listTables(CachingCatalog.java:135) ~[iceberg-core-1.4.3.jar:?]
    at org.apache.amoro.formats.iceberg.IcebergCatalog.lambda$listTables$5(IcebergCatalog.java:94) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:256) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:231) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_422]
    at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_422]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1918) ~[hadoop-common-2.10.2.jar:?]
    at org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:231) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.formats.iceberg.IcebergCatalog.listTables(IcebergCatalog.java:92) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.CommonUnifiedCatalog.listTables(CommonUnifiedCatalog.java:146) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.server.catalog.ExternalCatalog.lambda$listTables$10(ExternalCatalog.java:122) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:256) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.table.TableMetaStore.lambda$doAs$0(TableMetaStore.java:231) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_422]
    at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_422]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1918) ~[hadoop-common-2.10.2.jar:?]
    at org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:231) ~[amoro-core-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.server.catalog.ExternalCatalog.doAs(ExternalCatalog.java:161) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.server.catalog.ExternalCatalog.listTables(ExternalCatalog.java:119) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
    at org.apache.amoro.server.dashboard.controller.TableController.getTableList(TableController.java:509) ~[amoro-ams-server-0.7.0-incubating.jar:0.7.0-incubating]
    at io.javalin.core.security.SecurityUtil.noopAccessManager(SecurityUtil.kt:20) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServlet.addHandler$lambda-0(JavalinServlet.kt:96) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServlet$lifecycle$2$1$1.invoke(JavalinServlet.kt:43) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:99) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.8.jar:4.6.8]
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) ~[?:1.8.0_422]
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-11$lambda-10(JavalinServletHandler.kt:119) ~[javalin-4.6.8.jar:4.6.8]
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996) ~[?:1.8.0_422]
    at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:119) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.8.jar:4.6.8]
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) ~[?:1.8.0_422]
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServletHandler.executeNextTask$lambda-11$lambda-10(JavalinServletHandler.kt:119) ~[javalin-4.6.8.jar:4.6.8]
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996) ~[?:1.8.0_422]
    at io.javalin.http.JavalinServletHandler.executeNextTask(JavalinServletHandler.kt:119) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$lambda-1(JavalinServletHandler.kt:85) ~[javalin-4.6.8.jar:4.6.8]
    at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995) ~[?:1.8.0_422]
    at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137) ~[?:1.8.0_422]
    at io.javalin.http.JavalinServletHandler.queueNextTaskOrFinish$javalin(JavalinServletHandler.kt:85) ~[javalin-4.6.8.jar:4.6.8]
    at io.javalin.http.JavalinServlet.service(JavalinServlet.kt:89) ~[javalin-4.6.8.jar:4.6.8]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
    at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58) ~[javalin-4.6.8.jar:4.6.8]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) ~[jakarta.servlet-api-4.0.3.jar:4.0.3]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52) ~[javalin-4.6.8.jar:4.6.8]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at java.lang.Thread.run(Thread.java:750) [?:1.8.0_422]
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_table_objects_by_name'
    at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[libthrift-0.9.3.jar:0.9.3]
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.9.3.jar:0.9.3]
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name(ThriftHiveMetastore.java:1544) ~[hive-metastore-2.3.9.jar:2.3.9]
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name(ThriftHiveMetastore.java:1530) ~[hive-metastore-2.3.9.jar:2.3.9]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:1363) ~[hive-metastore-2.3.9.jar:2.3.9]
    at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_422]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_422]
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) ~[hive-metastore-2.3.9.jar:2.3.9]
    at com.sun.proxy.$Proxy51.getTableObjectsByName(Unknown Source) ~[?:?]
    at org.apache.iceberg.hive.HiveCatalog.lambda$listTables$2(HiveCatalog.java:133) ~[iceberg-hive-metastore-1.4.3.jar:?]
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58) ~[iceberg-core-1.4.3.jar:?]
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51) ~[iceberg-core-1.4.3.jar:?]
    at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:122) ~[iceberg-hive-metastore-1.4.3.jar:?]
    at org.apache.iceberg.hive.HiveCatalog.listTables(HiveCatalog.java:133) ~[iceberg-hive-metastore-1.4.3.jar:?]
    ... 81 more

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zhoujinsong]

@owenmonn Thanks for reporting this issue.

I am not sure why you need to build with hadoop2 profile while you are using Hive 4.

[owenmonn]

@zhoujinsong
I am using the Hadoop2 profile to register a Hadoop2 + Hive2 Metastore catalog in Amoro. At the same time, I also need to register a Hadoop3 + Hive4 Metastore catalog in the same Amoro instance.

[zhoujinsong]

It makes sense to me. Thanks for the information!