Iceberg fails to create the Hive table

[Niliria]

So I'm just following the Getting Started
Apache-Hive-2.1.0
Apache-Hadoop-2.7.3
iceberg-hive-runtime-0.12.1.jar
Metastore of Hive is deployed on the local Mysql server

When I perform

SET iceberg.engine.hive.enabled=true;
CREATE DATABASE IF NOT EXISTS iceberg_db;
ADD JAR /opt/software/hive/lib/iceberg-hive-runtime-0.12.1.jar;
CREATE TABLE iceberg_db.test_3 (
  id bigint,
  name string
)
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'

Job exception：

2022-05-10T16:30:54,595 INFO  [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - Executing command(queryId=root_20220510163054_4db7c534-2f5d-47db-b515-618f64aa0ecc): CREATE TABLE iceberg_db.test_3 (
  id bigint,
  name string
)
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
2022-05-10T16:30:54,595 INFO  [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - Starting task [Stage-0:DDL] in serial mode
2022-05-10T16:30:54,595 INFO  [HiveServer2-Background-Pool: Thread-15666]: plan.CreateTableDesc (:()) - Use StorageHandler-supplied org.apache.iceberg.mr.hive.HiveIcebergSerDe for table test_3
2022-05-10T16:30:54,596 INFO  [HiveServer2-Background-Pool: Thread-15666]: exec.DDLTask (:()) - creating table iceberg_db.test_3 on null
2022-05-10T16:30:54,641 WARN  [HiveServer2-Background-Pool: Thread-15666]: conf.HiveConf (HiveConf.java:initialize(3683)) - HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2022-05-10T16:30:54,641 WARN  [HiveServer2-Background-Pool: Thread-15666]: conf.HiveConf (HiveConf.java:initialize(3683)) - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
2022-05-10T16:30:54,649 INFO  [ForkJoinPool.commonPool-worker-0]: hive.metastore (:()) - Closed a connection to metastore, current connections: 8
2022-05-10T16:30:54,658 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://master:9083
2022-05-10T16:30:54,659 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Opened a connection to metastore, current connections: 9
2022-05-10T16:30:54,661 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Connected to metastore.
2022-05-10T16:30:54,666 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.HiveIcebergSerDe (:()) - Using hive schema {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"id","required":false,"type":"long"},{"id":1,"name":"name","required":false,"type":"string"}]}
2022-05-10T16:30:54,667 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.HiveIcebergSerDe (:()) - Using hive schema {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"id","required":false,"type":"long"},{"id":1,"name":"name","required":false,"type":"string"}]}
2022-05-10T16:30:54,819 WARN  [HiveServer2-Background-Pool: Thread-15666]: metastore.RetryingMetaStoreClient (:()) - MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
        at com.sun.proxy.$Proxy22.lock(Unknown Source)
        at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:430)
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
        at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:430)
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226)
        at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127)
        at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:165)
        at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:78)
        at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165)
        at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724)
        at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
        at com.sun.proxy.$Proxy22.createTable(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254)
        at com.sun.proxy.$Proxy22.createTable(Unknown Source)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
        at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-05-10T16:30:55,819 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Closed a connection to metastore, current connections: 8
2022-05-10T16:30:55,819 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://master:9083
2022-05-10T16:30:55,819 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Opened a connection to metastore, current connections: 9
2022-05-10T16:30:55,820 INFO  [HiveServer2-Background-Pool: Thread-15666]: hive.metastore (:()) - Connected to metastore.
2022-05-10T16:30:55,825 ERROR [HiveServer2-Background-Pool: Thread-15666]: exec.DDLTask (:()) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:840)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
        at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:297)
        at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127)
        at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:165)
        at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:78)
        at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165)
        at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724)
        at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
        at com.sun.proxy.$Proxy22.createTable(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254)
        at com.sun.proxy.$Proxy22.createTable(Unknown Source)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830)
        ... 22 more
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
        at com.sun.proxy.$Proxy22.lock(Unknown Source)
        at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:430)
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
        at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:430)
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226)
        ... 40 more

2022-05-10T16:30:55,825 ERROR [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3
2022-05-10T16:30:55,825 INFO  [HiveServer2-Background-Pool: Thread-15666]: ql.Driver (:()) - Completed executing command(queryId=root_20220510163054_4db7c534-2f5d-47db-b515-618f64aa0ecc); Time taken: 1.23 seconds
2022-05-10T16:30:55,825 ERROR [HiveServer2-Background-Pool: Thread-15666]: operation.Operation (:()) - Error running hive query: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3
        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
        at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
        at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
        at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:840)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979)
        at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
        ... 11 more
Caused by: java.lang.RuntimeException: Metastore operation failed for iceberg_db.test_3
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:297)
        at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:127)
        at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:165)
        at org.apache.iceberg.catalog.Catalog.createTable(Catalog.java:78)
        at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165)
        at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724)
        at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
        at com.sun.proxy.$Proxy22.createTable(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254)
        at com.sun.proxy.$Proxy22.createTable(Unknown Source)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830)
        ... 22 more
Caused by: org.apache.thrift.transport.TTransportException
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:150)
        at com.sun.proxy.$Proxy22.lock(Unknown Source)
        at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:430)
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
        at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
        at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
        at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:430)
        at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:226)
        ... 40 more

The metadata folder is created on HDFS；

[myeh3]

Met the same issue today!!!

[pvary]

Is there anything in the HMS logs for this timeframe?

Also Hive 2.3.9 would be recommended. I think I have seen several issues with Hive 2.1.0 already (but not specifically this one).

Also if you feel brave enough, you can try Hive 4.0.0-alpha-1, as it has more comprehensive Iceberg support,

[Niliria]

Is there anything in the HMS logs for this timeframe?

Also Hive 2.3.9 would be recommended. I think I have seen several issues with Hive 2.1.0 already (but not specifically this one).

Also if you feel brave enough, you can try Hive 4.0.0-alpha-1, as it has more comprehensive Iceberg support,

Thank you very much for your reply!
Today I tried again in the current Hive version, but the result still failed;
I can't find any more error messages. According to the HMS log information, all I can know is about metaStore connection failure, but if I directly create a hive table, it is successful, so I don't know what the problem is
The complete log is as follows:

2022-05-12T20:45:31,510 INFO  [HiveServer2-Handler-Pool: Thread-2423]: ql.Driver (:()) - Compiling command(queryId=root_20220512204531_9bce01ff-5a63-44fd-bc37-df442fed368f): CREATE TABLE test_5(
    id INT
    ,name STRING
)
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
2022-05-12T20:45:31,510 INFO  [HiveServer2-Handler-Pool: Thread-2423]: parse.CalcitePlanner (:()) - Starting Semantic Analysis
2022-05-12T20:45:31,510 INFO  [HiveServer2-Handler-Pool: Thread-2423]: parse.CalcitePlanner (:()) - Creating table iceberg_test.test_5 position=13
2022-05-12T20:45:31,512 INFO  [HiveServer2-Handler-Pool: Thread-2423]: sqlstd.SQLStdHiveAccessController (:()) - Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=d7695262-4414-47af-8c29-58f99da236f3, clientType=HIVESERVER2]
2022-05-12T20:45:31,512 INFO  [HiveServer2-Handler-Pool: Thread-2423]: ql.Driver (:()) - Semantic Analysis Completed
2022-05-12T20:45:31,512 INFO  [HiveServer2-Handler-Pool: Thread-2423]: ql.Driver (:()) - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
2022-05-12T20:45:31,512 INFO  [HiveServer2-Handler-Pool: Thread-2423]: ql.Driver (:()) - Completed compiling command(queryId=root_20220512204531_9bce01ff-5a63-44fd-bc37-df442fed368f); Time taken: 0.002 seconds
2022-05-12T20:45:31,516 INFO  [d7695262-4414-47af-8c29-58f99da236f3 HiveServer2-Handler-Pool: Thread-2423]: conf.HiveConf (HiveConf.java:getLogIdVar(3511)) - Using the default value passed in for log id: d7695262-4414-47af-8c29-58f99da236f3
2022-05-12T20:45:31,516 INFO  [HiveServer2-Handler-Pool: Thread-2423]: session.SessionState (:()) - Resetting thread name to  HiveServer2-Handler-Pool: Thread-2423
2022-05-12T20:45:31,522 INFO  [HiveServer2-Background-Pool: Thread-7328]: ql.Driver (:()) - Concurrency mode is disabled, not creating a lock manager
2022-05-12T20:45:31,522 INFO  [HiveServer2-Background-Pool: Thread-7328]: ql.Driver (:()) - Executing command(queryId=root_20220512204531_9bce01ff-5a63-44fd-bc37-df442fed368f): CREATE TABLE test_5(
    id INT
    ,name STRING
)
STORED BY 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler'
2022-05-12T20:45:31,522 INFO  [HiveServer2-Background-Pool: Thread-7328]: ql.Driver (:()) - Starting task [Stage-0:DDL] in serial mode
2022-05-12T20:45:31,522 INFO  [HiveServer2-Background-Pool: Thread-7328]: plan.CreateTableDesc (:()) - Use StorageHandler-supplied org.apache.iceberg.mr.hive.HiveIcebergSerDe for table test_5
2022-05-12T20:45:31,523 INFO  [HiveServer2-Background-Pool: Thread-7328]: exec.DDLTask (:()) - creating table iceberg_test.test_5 on null
2022-05-12T20:45:31,637 WARN  [HiveServer2-Background-Pool: Thread-7328]: conf.HiveConf (HiveConf.java:initialize(3683)) - HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
2022-05-12T20:45:31,637 WARN  [HiveServer2-Background-Pool: Thread-7328]: conf.HiveConf (HiveConf.java:initialize(3683)) - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
2022-05-12T20:45:31,653 INFO  [ForkJoinPool.commonPool-worker-1]: hive.metastore (:()) - Closed a connection to metastore, current connections: 4
2022-05-12T20:45:31,679 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://master:9083
2022-05-12T20:45:31,679 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Opened a connection to metastore, current connections: 5
2022-05-12T20:45:31,686 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Connected to metastore.
2022-05-12T20:45:31,691 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.HiveIcebergSerDe (:()) - Using hive schema {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"id","required":false,"type":"int"},{"id":1,"name":"name","required":false,"type":"string"}]}
2022-05-12T20:45:31,700 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.HiveIcebergSerDe (:()) - Using hive schema {"type":"struct","schema-id":0,"fields":[{"id":0,"name":"id","required":false,"type":"int"},{"id":1,"name":"name","required":false,"type":"string"}]}
2022-05-12T20:45:31,795 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Closed a connection to metastore, current connections: 4
2022-05-12T20:45:31,795 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Trying to connect to metastore with URI thrift://master:9083
2022-05-12T20:45:31,795 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Opened a connection to metastore, current connections: 5
2022-05-12T20:45:31,804 INFO  [HiveServer2-Background-Pool: Thread-7328]: hive.metastore (:()) - Connected to metastore.
2022-05-12T20:45:31,807 ERROR [HiveServer2-Background-Pool: Thread-7328]: exec.DDLTask (:()) - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Metastore operation failed for iceberg_test.test_5
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:840)
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845)
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
    at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
    at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: Metastore operation failed for iceberg_test.test_5
    at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:284)
    at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)
    at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:216)
    at org.apache.iceberg.BaseMetastoreCatalog.createTable(BaseMetastoreCatalog.java:49)
    at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165)
    at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
    at com.sun.proxy.$Proxy22.createTable(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254)
    at com.sun.proxy.$Proxy22.createTable(Unknown Source)
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830)
    ... 22 more
Caused by: org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104)
    at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:413)
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:62)
    at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
    at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:413)
    at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:217)
    ... 42 more

2022-05-12T20:45:31,807 ERROR [HiveServer2-Background-Pool: Thread-7328]: ql.Driver (:()) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Metastore operation failed for iceberg_test.test_5
2022-05-12T20:45:31,807 INFO  [HiveServer2-Background-Pool: Thread-7328]: ql.Driver (:()) - Completed executing command(queryId=root_20220512204531_9bce01ff-5a63-44fd-bc37-df442fed368f); Time taken: 0.285 seconds
2022-05-12T20:45:31,808 ERROR [HiveServer2-Background-Pool: Thread-7328]: operation.Operation (:()) - Error running hive query: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: Metastore operation failed for iceberg_test.test_5
    at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:387)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
    at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:90)
    at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:299)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:312)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Metastore operation failed for iceberg_test.test_5
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:840)
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:845)
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:333)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1858)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1562)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1313)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1084)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1077)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
    ... 11 more
Caused by: java.lang.RuntimeException: Metastore operation failed for iceberg_test.test_5
    at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:284)
    at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)
    at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:216)
    at org.apache.iceberg.BaseMetastoreCatalog.createTable(BaseMetastoreCatalog.java:49)
    at org.apache.iceberg.mr.Catalogs.createTable(Catalogs.java:165)
    at org.apache.iceberg.mr.hive.HiveIcebergMetaHook.commitCreateTable(HiveIcebergMetaHook.java:147)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:738)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:724)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
    at com.sun.proxy.$Proxy22.createTable(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2254)
    at com.sun.proxy.$Proxy22.createTable(Unknown Source)
    at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:830)
    ... 22 more
Caused by: org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104)
    at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:413)
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:62)
    at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
    at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:413)
    at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:217)
    ... 42 more

[Niliria]

Met the same issue today!!!

This is a bad problem, and I don't want to fix it by changing the Hive version

[pvary]

@Niliria: The log you have provided is HiveServer2 logs. This part of the Exception shows, that there is an issue with communicating with the HiveMetastore. Specifically when calling the lock method in HMS. The TransportException shows that the issue is on transport level (or unexpected error on server side). You might get more info in the HMS logs.

Caused by: org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:4549)
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:4536)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:2104)
    at org.apache.iceberg.hive.HiveTableOperations.lambda$acquireLock$7(HiveTableOperations.java:413)
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:62)
    at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
    at org.apache.iceberg.hive.HiveTableOperations.acquireLock(HiveTableOperations.java:413)
    at org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:217)
    ... 42 more

[pvary]

@Niliria: Were you able to solve the issue?