[helm] Cannot run more than two worker

[firehist]

Helm Chart Version

0.101.2

What step the error happened?

Other

Relevant information

We are running Airbyte, thanks to the official helm chart in version 0.101.2.

We are using external Google Secret Key Manager, Google Cloud Storage and our PostgreSQL on Google Cloud SQL as external in our values.yaml.

Everything is working well, except when I have the great idea to scale up worker > 1.

Other workers will fail with the attached log ouput.

It seems to receive a string //airbyte-airbyte-server-svc" which should be the internal API HOST or something like that, but the parsePort() fails, as there is no port in this string...
The second worker POD definition is exactly the same than for the first one, as it is controlled by a Deployment.

Relevant log output

at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2732) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.findBean(DefaultBeanContext.java:1794) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.AbstractBeanResolutionContext.findBean(AbstractBeanResolutionContext.java:112) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveOptionalBean(AbstractInitializableBeanDefinition.java:2305) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.AbstractInitializableBeanDefinition.findBeanForField(AbstractInitializableBeanDefinition.java:1904) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.airbyte.workers.$ApplicationInitializer$Definition.inject(Unknown Source) ~[io.airbyte-airbyte-workers-0.61.0.jar:?]
	at io.airbyte.workers.$ApplicationInitializer$Definition.instantiate(Unknown Source) ~[io.airbyte-airbyte-workers-0.61.0.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2311) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2281) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2293) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3095) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2997) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2958) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2932) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.addCandidateToList(DefaultBeanContext.java:3521) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistrations(DefaultBeanContext.java:3476) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.getBeanRegistrations(DefaultBeanContext.java:3450) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:1432) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:878) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext.getBeansOfType(DefaultBeanContext.java:868) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.lambda$$1(ApplicationEventPublisherFactory.java:215) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.core.util.SupplierUtil$1.get(SupplierUtil.java:47) ~[micronaut-core-4.4.8.jar:4.4.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery.onStart(NettyServiceDiscovery.java:54) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.netty.discovery.$NettyServiceDiscovery$Definition$Exec.dispatch(Unknown Source) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:3856) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.aop.chain.AdapterIntroduction.intercept(AdapterIntroduction.java:84) ~[micronaut-aop-4.4.8.jar:4.4.8]
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:138) ~[micronaut-aop-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery$ApplicationEventListener$onStart1$Intercepted.onApplicationEvent(Unknown Source) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:266) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226) ~[micronaut-inject-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:679) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:317) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:110) ~[micronaut-http-server-netty-4.4.8.jar:4.4.8]
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:80) [micronaut-context-4.4.8.jar:4.4.8]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:328) [micronaut-context-4.4.8.jar:4.4.8]
	at io.micronaut.runtime.Micronaut.run(Micronaut.java:314) [micronaut-context-4.4.8.jar:4.4.8]
	at io.airbyte.workers.Application.main(Application.java:15) [io.airbyte-airbyte-workers-0.61.0.jar:?]
Caused by: java.lang.NumberFormatException: For input string: "//airbyte-airbyte-server-svc"
	at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?]
	at java.base/java.lang.Integer.parseInt(Integer.java:648) ~[?:?]
	at java.base/java.lang.Integer.parseInt(Integer.java:778) ~[?:?]
	at io.airbyte.workers.config.ApiClientBeanFactory.parsePort(ApiClientBeanFactory.java:160) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
	at io.airbyte.workers.config.ApiClientBeanFactory.apiClient(ApiClientBeanFactory.java:53) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
	at io.airbyte.workers.config.$ApiClientBeanFactory$ApiClient0$Definition.instantiate(Unknown Source) ~[io.airbyte-airbyte-commons-worker-0.61.0.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2311) ~[micronaut-inject-4.4.8.jar:4.4.8]
	... 85 more

[firehist]

Waouh, problem fixed by removing the http:// prefix in the INTERNAL_API_HOST environment variable (coming from airbyte-airbyte-env config map).

It can be related to the latest chart release ?
https://artifacthub.io/packages/helm/airbyte/airbyte/0.101.2?modal=template&template=airbyte-db.yaml&compare-to=0.94.1

[marcosmarxm]

Thanks for reporting the issue @firehist I added this to deployment team backlog for further investigation