Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing trino-iceberg-minio setup #45

Closed
tschaub opened this issue Apr 4, 2024 · 4 comments
Closed

Failing trino-iceberg-minio setup #45

tschaub opened this issue Apr 4, 2024 · 4 comments

Comments

@tschaub
Copy link
Contributor

tschaub commented Apr 4, 2024 •
edited

Running docker compose up with the trino-iceberg-minio example fails with things like this in the logs:

hive-metastore-1     | MetaException(message:Error creating transactional connection factory)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
hive-metastore-1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hive-metastore-1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hive-metastore-1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore-1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore-1     | 	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
hive-metastore-1     | 	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
hive-metastore-1     | Caused by: MetaException(message:Error creating transactional connection factory)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
hive-metastore-1     | 	... 11 more

In trying to work around this, I have tried to modify the setup to use a JDBC catalog. See https://github.com/tschaub/trino-example.
@edkry
Copy link

edkry commented Apr 7, 2024

Having the same error on Delta Lake setup

@iuliu-b
Copy link

iuliu-b commented Apr 16, 2024 •
edited

Same error on hive/trino-minio, full stacktrace:

hive-metastore_1     | MetaException(message:Error creating transactional connection factory)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hive-metastore_1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore_1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore_1     | 	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
hive-metastore_1     | 	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
hive-metastore_1     | Caused by: MetaException(message:Error creating transactional connection factory)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
hive-metastore_1     | 	... 11 more
hive-metastore_1     | Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
hive-metastore_1     | NestedThrowables:
hive-metastore_1     | java.lang.reflect.InvocationTargetException
hive-metastore_1     | 	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
hive-metastore_1     | 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
hive-metastore_1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore_1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore_1     | 	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1975)
hive-metastore_1     | 	at java.security.AccessController.doPrivileged(Native Method)
hive-metastore_1     | 	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1970)
hive-metastore_1     | 	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1177)
hive-metastore_1     | 	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:814)
hive-metastore_1     | 	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:702)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:647)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:690)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:480)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:417)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:372)
hive-metastore_1     | 	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
hive-metastore_1     | 	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:705)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:683)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:677)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:760)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:525)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hive-metastore_1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore_1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
hive-metastore_1     | 	... 13 more
hive-metastore_1     | Caused by: java.lang.reflect.InvocationTargetException
hive-metastore_1     | 	at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
hive-metastore_1     | 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
hive-metastore_1     | 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
hive-metastore_1     | 	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
hive-metastore_1     | 	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
hive-metastore_1     | 	at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
hive-metastore_1     | 	at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
hive-metastore_1     | 	at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)
hive-metastore_1     | 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
hive-metastore_1     | 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
hive-metastore_1     | 	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
hive-metastore_1     | 	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
hive-metastore_1     | 	at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
hive-metastore_1     | 	at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
hive-metastore_1     | 	... 43 more
hive-metastore_1     | Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Too many connections
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
hive-metastore_1     | 	... 59 more
hive-metastore_1     | Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Too many connections
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
hive-metastore_1     | 	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
hive-metastore_1     | 	... 61 more
hive-metastore_1     | Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
hive-metastore_1     | 	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
hive-metastore_1     | 	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
hive-metastore_1     | 	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
hive-metastore_1     | 	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
hive-metastore_1     | 	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
hive-metastore_1     | 	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
hive-metastore_1     | 	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
hive-metastore_1     | 	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
hive-metastore_1     | 	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
hive-metastore_1     | 	... 65 more

@iuliu-b iuliu-b mentioned this issue Apr 17, 2024
Open
@iuliu-b
Copy link

iuliu-b commented Apr 17, 2024

This has not fixed trino-iceberg-minio for me (I realize that it should have no impact on trino-minio)

@tschaub
Copy link
Contributor Author

tschaub commented Apr 17, 2024

@bitsondatadev - I should have been clearer, the stack trace above came after I had already removed the -XX:-UseBiasedLocking config (as you did in 5414225).

If others are looking for a Docker compose setup with Trino, Iceberg, and MinIO, here is what worked for me: https://github.com/tschaub/trino-example/blob/main/docker-compose.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tschaub @iuliu-b @edkry