api pod fails to start on 2.20.0 upgrade

[jderieg]

Bug description 🐞

I'm attempting to upgrade Terrakube from 2.19.2 to 2.20.0 in EKS. I'm installing with the helm chart, so I'm also going from chart version 3.14.2 to 3.16.0. I get the following error on the api pod as it starts up. We're using an RDS cluster running PostgreSQL version 16.1 for the terrakube db.

2024-03-26 17:38:42.583  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: The connection attempt failed.
2024-03-26 17:38:42.587  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2024-03-26 17:38:42.620  INFO 1 --- [           main] ConditionEvaluationReportLoggingListener : 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2024-03-26 17:38:42.660 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: The connection attempt failed.
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1804) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1168) ~[spring-context-5.3.31.jar:5.3.31]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:919) ~[spring-context-5.3.31.jar:5.3.31]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591) ~[spring-context-5.3.31.jar:5.3.31]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.18.jar:2.7.18]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732) ~[spring-boot-2.7.18.jar:2.7.18]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:409) ~[spring-boot-2.7.18.jar:2.7.18]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.18.jar:2.7.18]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1300) ~[spring-boot-2.7.18.jar:2.7.18]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1289) ~[spring-boot-2.7.18.jar:2.7.18]
	at org.terrakube.api.ServerApplication.main(ServerApplication.java:18) ~[classes/:2.20.0]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[workspace/:na]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[workspace/:na]
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[workspace/:na]
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[workspace/:na]
Caused by: liquibase.exception.DatabaseException: org.postgresql.util.PSQLException: The connection attempt failed.
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:271) ~[liquibase-core-4.8.0.jar:na]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1863) ~[spring-beans-5.3.31.jar:5.3.31]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1800) ~[spring-beans-5.3.31.jar:5.3.31]
	... 26 common frames omitted
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:358) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:273) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.Driver.makeConnection(Driver.java:446) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.Driver.connect(Driver.java:298) ~[postgresql-42.7.2.jar:42.7.2]
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source) ~[java.sql:na]
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source) ~[java.sql:na]
	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:104) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:88) ~[postgresql-42.7.2.jar:42.7.2]
	at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:266) ~[liquibase-core-4.8.0.jar:na]
	... 28 common frames omitted
Caused by: java.net.SocketTimeoutException: Connect timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(Unknown Source) ~[na:na]
	at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source) ~[na:na]
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source) ~[na:na]
	at java.base/java.net.Socket.connect(Unknown Source) ~[na:na]
	at org.postgresql.core.PGStream.createSocket(PGStream.java:243) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.PGStream.<init>(PGStream.java:98) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:136) ~[postgresql-42.7.2.jar:42.7.2]
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:262) ~[postgresql-42.7.2.jar:42.7.2]
	... 37 common frames omitted

Steps to reproduce

Upgrade from 2.19.2 to 2.20.0 with the aforementioned configuration.

Expected behavior

New pods should come up with no errors.

Example repository

No response

Anything else?

No response

[alfespa17]

Hello @jderieg I few days ago there were 2 pull request that added some new customization options for the posgresql connection, maybe that is the issue because your connection looks like is failing, the pull request added the following env variables and default values:

• DatasourcePort
  • Default value: 5432
• DatasourceSchema:
  • Default value: public

Not sure what is the default schema in RDS because right now by default is using "public"

[jderieg]

yeah, it's using the 'public' schema, and the default port 5432, so that PR shouldn't have affected anything for us by adding those 2 vars and default values.

[jderieg]

OK, I figured it out. The Helm chart specifies a default value in the API block as databasePort: "3306". I set that in our helm values file to be 5432, and the upgrade worked. I'm kind of confused how that hasn't caused problems previously though. The version 3.14.2 also had that value defaulted to 3306, so I'm not sure how it worked before.

[alfespa17]

OK, I figured it out. The Helm chart specifies a default value in the API block as databasePort: "3306". I set that in our helm values file to be 5432, and the upgrade worked. I'm kind of confused how that hasn't caused problems previously though. The version 3.14.2 also had that value defaulted to 3306, so I'm not sure how it worked before.

This was because port customization was only available for MYSQL (port 3306) and the value was ignored for PostgreSql

[jderieg]

OK, makes sense. That might be good for people to know that that is a breaking change. If they're using PostgreSQL, but not specifying that port in the helm chart for the API, the upgrade will break.

[alfespa17]

I think we can add that comment when we create the release discussion.