Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Vault Secrets Manager - SSL Exception when connecting to vault #22065

Closed
seanglynn-thrive opened this issue Jan 30, 2023 · 4 comments
Closed

Vault Secrets Manager - SSL Exception when connecting to vault #22065

seanglynn-thrive opened this issue Jan 30, 2023 · 4 comments
Assignees
@octavia-squidington-iii
Labels
area/platform issues related to the platform community Stale team/prod-eng type/bug Something isn't working

Comments

@seanglynn-thrive
Copy link

seanglynn-thrive commented Jan 30, 2023 •
edited
Loading

Related to #10519

SSL exception is thrown while we are trying to persist connection secrets/tokens to Hashicorp vault from within the Airbyte UI.
The following required vault configurations are applied and available within the Airbyte server container:
https://docs.airbyte.com/operator-guides/configuring-airbyte/#secrets

Exception from within the Airbyte UI does not give us much information when we try to configure a source and test the connection:
image

After scanning through the logs I noticed that the airbyte-worker failed to r/w a secret to the vault location:

Caused by: io.temporal.failure.ApplicationFailure: message='That secret was not found in the store! Coordinate: airbyte_workspace_00000000-0000-0000-0000-000000000000_secret_xxx_yyy_v1', type='java.lang.RuntimeException', nonRetryable=false

This exception seems to be caused by a failed connection handshake between the Airbyte server and the Hashicorp Vault location.
We can see that an SSL exception gets thrown as the cert is not valid:
Airbyte Server logs:

ERROR�[m i.a.c.p.s.VaultSecretPersistence(write):54 - Vault failed on write
com.bettercloud.vault.VaultException: com.bettercloud.vault.rest.RestException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.bettercloud.vault.api.Logical.write(Logical.java:288) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.api.Logical.write(Logical.java:222) ~[vault-java-driver-5.1.0.jar:?]
	at io.airbyte.config.persistence.split_secrets.VaultSecretPersistence.write(VaultSecretPersistence.java:52) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
	at java.util.HashMap.forEach(HashMap.java:1429) ~[?:?]
	at io.airbyte.config.persistence.SecretsRepositoryWriter.splitSecretConfig(SecretsRepositoryWriter.java:186) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
	at io.airbyte.config.persistence.SecretsRepositoryWriter.statefulSplitEphemeralSecrets(SecretsRepositoryWriter.java:177) ~[io.airbyte.airbyte-config-config-persistence-0.40.29.jar:?]
	at io.airbyte.server.handlers.SchedulerHandler.checkSourceConnectionFromSourceCreate(SchedulerHandler.java:166) ~[io.airbyte-airbyte-server-0.40.29.jar:?]
	at io.airbyte.server.apis.SchedulerApiController.lambda$executeSourceCheckConnection$1(SchedulerApiController.java:45) ~[io.airbyte-airbyte-server-0.40.29.jar:?]
	at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.java:18) ~[io.airbyte-airbyte-server-0.40.29.jar:?]
	at io.airbyte.server.apis.SchedulerApiController.executeSourceCheckConnection(SchedulerApiController.java:45) ~[io.airbyte-airbyte-server-0.40.29.jar:?]
	at io.airbyte.server.apis.$SchedulerApiController$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-server-0.40.29.jar:?]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:378) ~[micronaut-inject-3.8.2.jar:3.8.2]
	at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:594) ~[micronaut-inject-3.8.2.jar:3.8.2]
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303) ~[micronaut-router-3.8.2.jar:3.8.2]
	at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111) ~[micronaut-router-3.8.2.jar:3.8.2]
	at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103) ~[micronaut-http-3.8.2.jar:3.8.2]
	at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:659) ~[micronaut-http-server-3.8.2.jar:3.8.2]
	at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49) ~[reactor-core-3.5.0.jar:3.5.0]
	at reactor.core.publisher.Flux.subscribe(Flux.java:8660) ~[reactor-core-3.5.0.jar:3.5.0]
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:426) ~[reactor-core-3.5.0.jar:3.5.0]
	at io.micronaut.reactive.reactor.instrument.ReactorSubscriber.onNext(ReactorSubscriber.java:57) ~[micronaut-runtime-3.8.2.jar:3.8.2]
	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:172) ~[reactor-core-3.5.0.jar:3.5.0]
	at io.micronaut.http.server.netty.RoutingInBoundHandler$4.doOnComplete(RoutingInBoundHandler.java:965) ~[micronaut-http-server-netty-3.8.2.jar:3.8.2]
	at io.micronaut.core.async.subscriber.CompletionAwareSubscriber.onComplete(CompletionAwareSubscriber.java:79) ~[micronaut-core-reactive-3.8.2.jar:3.8.2]
	at io.micronaut.http.server.netty.jackson.JsonContentProcessor$1.doOnComplete(JsonContentProcessor.java:136) ~[micronaut-http-server-netty-3.8.2.jar:3.8.2]
	at io.micronaut.core.async.subscriber.CompletionAwareSubscriber.onComplete(CompletionAwareSubscriber.java:79) ~[micronaut-core-reactive-3.8.2.jar:3.8.2]
	at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
	at io.micronaut.core.async.processor.SingleThreadedBufferingProcessor.doOnComplete(SingleThreadedBufferingProcessor.java:48) ~[micronaut-core-reactive-3.8.2.jar:3.8.2]
	at io.micronaut.jackson.core.parser.JacksonCoreProcessor.doOnComplete(JacksonCoreProcessor.java:94) ~[micronaut-jackson-core-3.8.2.jar:3.8.2]
	at io.micronaut.core.async.subscriber.SingleThreadedBufferingSubscriber.onComplete(SingleThreadedBufferingSubscriber.java:71) ~[micronaut-core-reactive-3.8.2.jar:3.8.2]
	at io.micronaut.http.server.netty.jackson.JsonContentProcessor.doOnComplete(JsonContentProcessor.java:161) ~[micronaut-http-server-netty-3.8.2.jar:3.8.2]
	at io.micronaut.core.async.subscriber.CompletionAwareSubscriber.onComplete(CompletionAwareSubscriber.java:79) ~[micronaut-core-reactive-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher.publishMessage(HandlerPublisher.java:383) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher.flushBuffer(HandlerPublisher.java:470) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher.publishMessageLater(HandlerPublisher.java:360) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher.complete(HandlerPublisher.java:423) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher.handlerRemoved(HandlerPublisher.java:418) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerRemoved(AbstractChannelHandlerContext.java:1122) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:637) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:477) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:423) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.micronaut.http.netty.stream.HttpStreamsHandler.removeHandlerIfActive(HttpStreamsHandler.java:483) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.stream.HttpStreamsHandler.handleReadHttpContent(HttpStreamsHandler.java:319) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.stream.HttpStreamsHandler.channelRead(HttpStreamsHandler.java:282) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.stream.HttpStreamsServerHandler.channelRead(HttpStreamsServerHandler.java:134) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99) ~[netty-codec-http-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.codec.http.HttpServerKeepAliveHandler.channelRead(HttpServerKeepAliveHandler.java:64) ~[netty-codec-http-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:200) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.handler.flow.FlowControlHandler.read(FlowControlHandler.java:139) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:837) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:814) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.micronaut.http.netty.reactive.HandlerPublisher.requestDemand(HandlerPublisher.java:165) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.stream.HttpStreamsHandler$2.requestDemand(HttpStreamsHandler.java:273) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher$ChannelSubscription.receivedDemand(HandlerPublisher.java:556) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.micronaut.http.netty.reactive.HandlerPublisher$ChannelSubscription.lambda$request$0(HandlerPublisher.java:494) ~[micronaut-http-netty-3.8.2.jar:3.8.2]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: com.bettercloud.vault.rest.RestException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.bettercloud.vault.rest.Rest.postOrPutImpl(Rest.java:416) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.rest.Rest.post(Rest.java:306) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.api.Logical.write(Logical.java:264) ~[vault-java-driver-5.1.0.jar:?]
	... 82 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:578) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1429) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1400) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?]
	at com.bettercloud.vault.rest.Rest.postOrPutImpl(Rest.java:399) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.rest.Rest.post(Rest.java:306) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.api.Logical.write(Logical.java:264) ~[vault-java-driver-5.1.0.jar:?]
	... 82 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:578) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1429) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1400) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?]
	at com.bettercloud.vault.rest.Rest.postOrPutImpl(Rest.java:399) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.rest.Rest.post(Rest.java:306) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.api.Logical.write(Logical.java:264) ~[vault-java-driver-5.1.0.jar:?]
	... 82 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473) ~[?:?]
	at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:578) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:187) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1429) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1400) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?]
	at com.bettercloud.vault.rest.Rest.postOrPutImpl(Rest.java:399) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.rest.Rest.post(Rest.java:306) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.api.Logical.write(Logical.java:264) ~[vault-java-driver-5.1.0.jar:?]
	... 82 more

Resolution attempts

I have already attempted to create a configmap, from the required SSL cert which is needed to connect to our vault URL, and mount it to /etc/ssl/certs/ on both the airbyte-server + airbyte-worker but it has not resolved the issue. See:

  1. Created ca.crt configmap:
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-ca-cert
namespace: airbyte
data:
my-cert.pem: |
  -----BEGIN CERTIFICATE-----
........
  1. Mount the ca.crt configmap to /etc/ssl/certs/my-cert.pem
        volumeMounts:
        - name: vault-ca-cert
          mountPath: /etc/ssl/certs/my-cert.pem
          subPath: my-cert.pem
          readOnly: false
      volumes:
      - name:  vault-ca-cert
        configMap:
          name:  vault-ca-cert

Environment

  • Airbyte version: 0.40.29
  • OS Version / Instance: Linux data-airbyte-server-xxx 5.4.0-1089-azure make server depend on db not scheduler #94~18.04.1-Ubuntu SMP x86_64 GNU/Linux
  • Deployment: Kubernetes via helm
  • Source Connector and version: (if applicable example Salesforce 0.2.3)
  • Destination Connector and version: (if applicable example Postgres 0.3.3)
  • Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

Current Behavior

Airbyte server is failing to connect to the hashicorp vault location.

Expected Behavior

Airbyte server should connect to the vault location to R/W secrets.

Logs

Steps to Reproduce

  1. Pass vault as secret manager configuration
  2. Deploy to k8s
  3. Try to configure a connector

Are you willing to submit a PR?

I can certainly help out here to make any changes that I am qualified to make.
(Data eng background: k8s,python)
@seanglynn-thrive seanglynn-thrive added needs-triage type/bug Something isn't working labels Jan 30, 2023
@seanglynn-thrive seanglynn-thrive mentioned this issue Jan 30, 2023
Open
@seanglynn-thrive
Copy link
Author

If we could somehow add this vault configuration to the airbyte/vault integration, I think this would save our bacon:
https://developer.hashicorp.com/vault/docs/commands#vault_capath

@sajarin sajarin added area/platform issues related to the platform team/prod-eng and removed needs-triage team/tse Technical Support Engineers autoteam labels Jan 31, 2023
@seanglynn-thrive
Copy link
Author

FYI,
We could not resolve this issue so we moved to using GCP Secrets Manager which has been working as expected. This alternative may unblock some people who use GCP.

See docs: https://docs.airbyte.com/operator-guides/security/#credential-management

@octavia-squidington-iii
Copy link
Collaborator

At Airbyte, we seek to be clear about the project priorities and roadmap. This issue has not had any activity for 180 days, suggesting that it's not as critical as others. It's possible it has already been fixed. It is being marked as stale and will be closed in 20 days if there is no activity. To keep it open, please comment to let us know why it is important to you and if it is still reproducible on recent versions of Airbyte.

@octavia-squidington-iii
Copy link
Collaborator

This issue was closed because it has been inactive for 20 days since being marked as stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@octavia-squidington-iii octavia-squidington-iii

Labels
area/platform issues related to the platform community Stale team/prod-eng type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sajarin @octavia-squidington-iii @seanglynn-thrive @octavia-squidington-iv