Source Salesforce not passing discovery phase on Kubernetes #6021

marcosmarxm · 2021-09-13T16:30:23Z

Enviroment

Airbyte version: 0.29.16-alpha
OS Version / Instance: ˜
Deployment: Kubernetes
Source Connector and version: Salesforce
Destination Connector and version: Postgres
Severity: Very Low / Low / Medium / High / Critical
Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

Current Behavior

Screen.Recording.2021-09-13.at.14.02.06.mov

Expected Behavior

Tell us what should happen.

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG


replace this with
your long log
output here

Steps to Reproduce

Are you willing to submit a PR?

Remove this with your answer.

The text was updated successfully, but these errors were encountered:

mhensley-cls · 2021-09-23T16:33:09Z

My team is experiencing this same issue with salesforce... different version info below.

Airbyte version: 0.29.21-alpha
OS Version / Instance:
Deployment: Kubernetes
Source Connector and version: Salesforce (0.1.1)
Destination Connector and version: Postgres
Severity: Very Low / Low / Medium / High / Critical
Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

It just loops back and forth between these two views.

Here's the only section of logs that doesn't look like noise.



2021-09-23 12:29:26 | 2021-09-23 16:29:26 INFO i.a.s.RequestLogger(filter):109 - {cloud_workspace_app_root=/workspace/server/logs} - REQ 10.201.70.165 POST 200 /api/v1/destination_definitions/list_latest - {"workspaceId":"c31f257d-4d8f-4469-8132-d2eb3fa606d8"}
-- | --
  |   | 2021-09-23 12:29:28 | 2021-09-23 16:29:28 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):239 - {cloud_job_log_path=/workspace/262f1f99-20e3-4076-8bec-d653c6c9a9f8/0/logs.log} - Waiting for init container to be ready before copying files...
  |   | 2021-09-23 12:29:32 | 2021-09-23 16:29:32 INFO i.a.s.RequestLogger(filter):109 - {cloud_workspace_app_root=/workspace/server/logs} - REQ 10.201.70.165 POST 200 /api/v1/sources/discover_schema - {"workspaceId":"c31f257d-4d8f-4469-8132-d2eb3fa606d8"}
  |   | 2021-09-23 12:29:32 | Sep 23, 2021 4:29:32 PM org.glassfish.jersey.server.ServerRuntime$Responder writeResponse
  |   | 2021-09-23 12:29:32 | SEVERE: An I/O error has occurred while writing a response message entity to the container output stream.
  |   | 2021-09-23 12:29:32 | org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:638)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:371)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:361)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:569)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1377)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:507)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1292)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.Server.handle(Server.java:501)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
  |   | 2021-09-23 12:29:32 | at java.base/java.lang.Thread.run(Thread.java:832)
  |   | 2021-09-23 12:29:32 | Caused by: org.eclipse.jetty.io.EofException
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:544)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:838)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:910)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:284)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:268)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:833)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:301)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:200)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:276)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2177)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeBytes(UTF8JsonGenerator.java:1266)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeFieldName(UTF8JsonGenerator.java:288)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:726)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1514)
  |   | 2021-09-23 12:29:32 | at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1006)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.jackson.internal.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:647)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
  |   | 2021-09-23 12:29:32 | at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
  |   | 2021-09-23 12:29:32 | ... 45 more
  |   | 2021-09-23 12:29:32 | Caused by: java.io.IOException: Broken pipe
  |   | 2021-09-23 12:29:32 | at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
  |   | 2021-09-23 12:29:32 | at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:66)
  |   | 2021-09-23 12:29:32 | at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
  |   | 2021-09-23 12:29:32 | at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
  |   | 2021-09-23 12:29:32 | at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:514)
  |   | 2021-09-23 12:29:32 | at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:507)
  |   | 2021-09-23 12:29:32 | at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273)
  |   | 2021-09-23 12:29:32 | ... 89 more

cgardens · 2021-10-28T17:27:15Z

@marcosmarxm do we know this does work in docker? like are we sure it is a k8s problem?

@sherifnada we did some saleforce stuff recently any chance it's related

marcosmarxm · 2021-11-15T13:08:46Z

@cgardens yes, in docker took a long time because there are a lot of tables in Salesforce, but works. In Kubernetes (until I know) couldn't pass the discovery step.

sherifnada · 2021-12-09T19:18:27Z

@cgardens @jrhizor based on marcos' comment above this strikes me as an issue with receiving large catalogs from the connector, rather than the connector not being able to output them at all.

@mhensley-cls how many streams/tables do you have in your SF account? this would help us recreate the issue

roypeter · 2021-12-21T11:51:48Z

@sherifnada we are also observing the same error and the number of tables are 550+

roypeter · 2021-12-22T11:21:13Z

Tweaking k8s ingress timeout fixed the issue

misteryeo · 2022-01-06T15:40:31Z

Amazing, can we close this issue now @roypeter?

cc: @marcosmarxm

marcosmarxm · 2022-01-10T20:46:08Z

@misteryeo we should at least write a doc or explain here how increate the timeout in k8s for people need this.

misteryeo · 2022-01-28T03:24:39Z

@cgardens agree with Marcos here. Is there someone on the platform team who could help draft these docs and tag me for review? 🙏

misteryeo · 2022-05-05T23:36:49Z

Resurfacing this @malikdiarra @andyjih can you take a look at this issue and ensure we have next steps to drive clarify for users?

malikdiarra · 2022-05-06T15:40:25Z

We've seen a similar issue recently as well of the discover phase failing because the catalog is too large. We can document the work around that allow people to fix that issue first.

misteryeo · 2022-05-06T21:34:22Z

Cool, assignee you to take over @malikdiarra!

TrustOkoroego · 2022-11-18T13:47:36Z

@malikdiarra Please do you already have the work around documented somewhere.
In my in case I increased the route(openshift) time out to 5 minutes. unfortunately it now fails after 2 minutes which is as a result of the AWS ELB behind the route timeout

malikdiarra · 2022-11-28T23:36:00Z

Hi @TrustOkoroego,

we changed the way we store in the process the catalog during the discoverSchema phase, this should limit issues related to discovering large catalog. This was released in v0.40.12, are you using that version or a more recent one?

arsenlosenko · 2023-03-28T07:50:56Z

I have tried to reproduce this bug by creating a Salesforce and Big Query connection, no issues were found during setup and discovery phase. Most likely it was already fixed.

@YowanR can you (or somebody from TCS) please confirm if users reported such problems recently? Otherwise this issue may be closed.

arsenlosenko · 2023-03-29T11:45:16Z

closing this issue for now, will re-open if the issue comes up again.

marcosmarxm added the type/bug Something isn't working label Sep 13, 2021

marcosmarxm added the kubernetes label Sep 24, 2021

sherifnada added the area/connectors Connector related issues label Nov 15, 2021

sherifnada added area/platform issues related to the platform and removed area/connectors Connector related issues labels Dec 9, 2021

igrankova added connectors/sources-api connectors/source/salesforce labels Jan 14, 2022

bleonard added autoteam team/connectors-python labels Apr 26, 2022

malikdiarra added team/compose area/documentation Improvements or additions to documentation labels May 6, 2022

misteryeo assigned misteryeo and malikdiarra and unassigned misteryeo May 6, 2022

lazebnyi assigned arsenlosenko and unassigned malikdiarra Mar 27, 2023

arsenlosenko closed this as completed Mar 29, 2023

rx007 mentioned this issue Oct 28, 2023

[Snyk] Security upgrade axios from 0.21.1 to 1.6.0 rx007/airbyte#2130

Open

mresposito mentioned this issue Oct 28, 2023

[Snyk] Security upgrade axios from 0.21.1 to 1.6.0 candulabs/airbyte#104

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Salesforce not passing discovery phase on Kubernetes #6021

Source Salesforce not passing discovery phase on Kubernetes #6021

marcosmarxm commented Sep 13, 2021

mhensley-cls commented Sep 23, 2021

cgardens commented Oct 28, 2021 •

edited

Loading

marcosmarxm commented Nov 15, 2021

sherifnada commented Dec 9, 2021

roypeter commented Dec 21, 2021

roypeter commented Dec 22, 2021

misteryeo commented Jan 6, 2022

marcosmarxm commented Jan 10, 2022

misteryeo commented Jan 28, 2022

misteryeo commented May 5, 2022

malikdiarra commented May 6, 2022

misteryeo commented May 6, 2022

TrustOkoroego commented Nov 18, 2022

malikdiarra commented Nov 28, 2022

arsenlosenko commented Mar 28, 2023

arsenlosenko commented Mar 29, 2023

Source Salesforce not passing discovery phase on Kubernetes #6021

Source Salesforce not passing discovery phase on Kubernetes #6021

Comments

marcosmarxm commented Sep 13, 2021

Enviroment

Current Behavior

Expected Behavior

Logs

Steps to Reproduce

Are you willing to submit a PR?

mhensley-cls commented Sep 23, 2021

cgardens commented Oct 28, 2021 • edited Loading

marcosmarxm commented Nov 15, 2021

sherifnada commented Dec 9, 2021

roypeter commented Dec 21, 2021

roypeter commented Dec 22, 2021

misteryeo commented Jan 6, 2022

marcosmarxm commented Jan 10, 2022

misteryeo commented Jan 28, 2022

misteryeo commented May 5, 2022

malikdiarra commented May 6, 2022

misteryeo commented May 6, 2022

TrustOkoroego commented Nov 18, 2022

malikdiarra commented Nov 28, 2022

arsenlosenko commented Mar 28, 2023

arsenlosenko commented Mar 29, 2023

cgardens commented Oct 28, 2021 •

edited

Loading