Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Source Google Sheets: sync fails when google sheets has a Data Connector to Big Query #17100

Closed
marcosmarxm opened this issue Sep 23, 2022 · 10 comments · Fixed by #23292
Closed

Source Google Sheets: sync fails when google sheets has a Data Connector to Big Query #17100

marcosmarxm opened this issue Sep 23, 2022 · 10 comments · Fixed by #23292
Assignees
@midavadim
Labels
area/connectors Connector related issues connectors/source/google-sheets discourse team/connectors-python type/bug Something isn't working zendesk

Comments

@marcosmarxm
Copy link
Member

This Github issue is synchronized with Zendesk:

Ticket ID: #2312
Priority: normal
Group: Community Assistance Engineer
Assignee: Sajarin

Original ticket description:

  • Is this your first time deploying Airbyte?: Yes
  • OS Version / Instance: Linux on AWS EC2. 2 vCPU.
  • Memory / Disk: 8 GBs memory
  • Deployment: Docker
  • Airbyte Version: 0.40.0-alpha
  • Source name/version: Google Sheets / 0.2.17
  • Destination name/version: BigQuery / 1.1.16
  • Step: An old source, while loading and while retesting the connection
  • Description:
    We had this running sync from Google Sheets to BigQuery working for around 15 days, and it was an every 24 hours sync. Two days ago, inside this Google Sheets, a user added a “Data Connector”, which connects to BigQuery, and returns the results in a new tab.
    The source load now started to fail because of this new tab, even though we are not even selecting it.
    Plus, when I try to “retest” the connection to Google Sheets source, I have the same error as when in the sync.

Now I wonder if there is a workaround to this, without having to impact the user’s work and removing this new tab.

Screenshot of the “retest source”:

Full log:

2022-09-15 07:49:49 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.refresh_token: is missing but it is required, $.auth_type: must be a constant value Client
2022-09-15 07:49:49 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.service_account_info: object found, string expected
2022-09-15 07:49:49 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.method: must be a constant value Standard
2022-09-15 07:49:49 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.credential.hmac_key_access_id: object found, string expected, $.credential.hmac_key_secret: object found, string expected
2022-09-15 07:49:49 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):105 - Docker volume job log path: /tmp/workspace/326/0/logs.log
2022-09-15 07:49:49 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.40.0-alpha
2022-09-15 07:49:49 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):83 - Checking if airbyte/source-google-sheets:0.2.17 exists...
2022-09-15 07:49:49 e[32mINFOe[m i.a.c.i.LineGobbler(voidCall):83 - airbyte/source-google-sheets:0.2.17 was found locally.
2022-09-15 07:49:49 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):108 - Creating docker job ID: 326
2022-09-15 07:49:49 e[32mINFOe[m i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/326/0 --log-driver none --name source-google-sheets-check-326-0-fdhqt --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/source-google-sheets:0.2.17 -e AIRBYTE_VERSION=0.40.0-alpha -e WORKER_JOB_ID=326 airbyte/source-google-sheets:0.2.17 check --config source_config.json
2022-09-15 07:49:57 e[1;31mERRORe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - Giving up get(...) after 1 tries (googleapiclient.errors.HttpError: <HttpError 400 when requesting https://sheets.googleapis.com/v4/spreadsheets/1I4CfINSAHY8F7RWahjBy1pyrqase4pJPWWZEA8LnAm8?includeGridData=true&ranges=Connected+sheet+2%211%3A1&alt=json returned "Unable to parse range: Connected sheet 2!1:1". Details: "Unable to parse range: Connected sheet 2!1:1">)
2022-09-15 07:49:57 e[1;31mERRORe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - <HttpError 400 when requesting https://sheets.googleapis.com/v4/spreadsheets/1I4CfINSAHY8F7RWahjBy1pyrqase4pJPWWZEA8LnAm8?includeGridData=true&ranges=Connected+sheet+2%211%3A1&alt=json returned "Unable to parse range: Connected sheet 2!1:1". Details: "Unable to parse range: Connected sheet 2!1:1">
2022-09-15 07:49:57 e[1;31mERRORe[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):95 - Check failed
2022-09-15 07:49:57 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...
2022-09-15 07:49:57 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.refresh_token: is missing but it is required, $.auth_type: must be a constant value Client
2022-09-15 07:49:57 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.service_account_info: object found, string expected
2022-09-15 07:49:57 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.method: must be a constant value Standard
2022-09-15 07:49:57 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed. 
errors: $.credential.hmac_key_access_id: object found, string expected, $.credential.hmac_key_secret: object found, string expected
2022-09-15 07:49:57 e[32mINFOe[m i.a.c.f.EnvVariableFeatureFlags(autoDisablesFailingConnections):17 - Auto Disable Failing Connections: false

[Discourse post]
@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Marcos Marx on 2022-09-15 at 15:03:

Hello there! You are receiving this message because none of your fellow community members has stepped in to respond to your topic post. (If you are a community member and you are reading this response, feel free to jump in if you have the answer!) As a result, the Community Assistance Team has been made aware of this topic and will be investigating and responding as quickly as possible.
Some important considerations that will help your to get your issue solved faster:
* It is best to use our topic creation template; if you haven’t yet, we recommend posting a followup with the requested information. With that information the team will be able to more quickly search for similar issues with connectors and the platform and troubleshoot more quickly your specific question or problem.
* Make sure to upload the complete log file; a common investigation roadblock is that sometimes the error for the issue happens well before the problem is surfaced to the user, and so having the tail of the log is less useful than having the whole log to scan through.
* Be as descriptive and specific as possible; when investigating it is extremely valuable to know what steps were taken to encounter the issue, what version of connector / platform / Java / Python / docker / k8s was used, etc. The more context supplied, the quicker the investigation can start on your topic and the faster we can drive towards an answer.
* We in the Community Assistance Team are glad you’ve made yourself part of our community, and we’ll do our best to answer your questions and resolve the problems as quickly as possible. Expect to hear from a specific team member as soon as possible.

Thank you for your time and attention.
Best,
The Community Assistance Team

@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Sajarin on 2022-09-15 at 17:02:

Hi @patricia.goldberg, 
 
Thanks for your post and welcome to the community! After reading through our documentation around the Google Sheets connector, my guess is that the inclusion of the new tab is causing the sync to fail (something you probably already suspect.)
 
The Google Sheets connector treats each tab as a separate stream and I'm guessing that it's not able to parse the new tab that your user created. It looks like it's failing trying to read "Connected sheet 2", is that the name of a sheet or a tab? 
 
The simplest solution might be to just move the output data from BigQuery to a separate spreadsheet so that it's not part of the daily sync. Other than that, we'll have to investigate why it's not able to parse the rows of the new tab. Could you perhaps share a couple of rows to get a sense of what the data looks like? 
 
Hope this answer helps!

@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Marcos Marx on 2022-09-22 at 13:19:

Yes, that was my suspicion indeed. And yes, the name of the tab is “Connected sheet 2”.

And the output of the data (what I see on Google sheets at least), it’s just like a normal table:

id|name|address|
|A789|Test|test|
|A800|TEST|Teststr.|
|T001|Testmarkt 1|St.-Martin-Str. 72, 81541 München|
|T002|Testmarkt 2|Birkenleiten 43, 81543 München|

[Discourse post]

@sajarin sajarin changed the title Connection to Google Sheets fails when google sheets has a Data Connector to Big Query Source Google Sheets - sync fails when google sheets has a Data Connector to Big Query Sep 23, 2022
@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Sajarin on 2022-09-23 at 16:29:

Hey @patricia.goldberg, 
 
Is the sync successful if you move the output data from BigQuery to another spreadsheet? Furthermore, I escalated the issue to our github repo so please feel free to follow there for any updates from the engineering team relating to this issue: #17100

@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Marcos Marx on 2022-09-27 at 08:39:

Hey!
Yes, it is. Thank you for creating the issue.

[Discourse post]

@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Marcos Marx on 2022-11-16 at 19:46:

Hi @sajarin, Did you manage to fix this issue at airbyte?
I think I have the same problem, but i don’t see any problem with my tab names. The output logs are here:

2022-11-16 19:32:22 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.auth_type: must be a constant value Client, $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.refresh_token: is missing but it is required
2022-11-16 19:32:22 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.service_account_info: object found, string expected
2022-11-16 19:32:22 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.method: must be a constant value Standard
2022-11-16 19:32:22 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.credential.hmac_key_secret: object found, string expected, $.credential.hmac_key_access_id: object found, string expected
2022-11-16 19:32:22 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/2275/0/logs.log
2022-11-16 19:32:22 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.18
2022-11-16 19:32:23 e[32mINFOe[m i.a.c.t.TemporalUtils(withBackgroundHeartbeat):316 - Stopping temporal heartbeating…
2022-11-16 19:32:23 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$4):193 - Completing future exceptionally…
java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:107) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:216) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
… 3 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method. (Service: S3, Status Code: 403, Request ID: null)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:68) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:63) ~[aws-core-2.17.271.jar:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:4483) ~[s3-2.17.271.jar:?]
at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:8373) ~[s3-2.17.271.jar:?]
at io.airbyte.workers.storage.S3DocumentStoreClient.read(S3DocumentStoreClient.java:74) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocument(AsyncOrchestratorPodProcess.java:230) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.checkStatus(AsyncOrchestratorPodProcess.java:234) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocStoreStatus(AsyncOrchestratorPodProcess.java:243) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:168) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
… 3 more
2022-11-16 19:32:23 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling…
2022-11-16 19:32:23 e[32mINFOe[m i.a.c.t.TemporalUtils(withBackgroundHeartbeat):283 - Stopping temporal heartbeating…
2022-11-16 19:32:23 e[33mWARNe[m i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=d01042ca-47bb-38e1-85a8-07bbafee5199, activityType=Replicate, attempt=1
java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:281) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.replicate(ReplicationActivityImpl.java:138) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:578) ~[?:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:95) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:92) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:241) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:57) ~[temporal-serviceclient-1.17.0.jar:?]
at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:461) ~[temporal-sdk-1.17.0.jar:?]
at io.temporal.activity.Activity.wrap(Activity.java:52) ~[temporal-sdk-1.17.0.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:166) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:182) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:276) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
… 14 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:160) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
at io.airbyte.workers.temporal.sync.ReplicationActivityImpl.lambda$replicate$3(ReplicationActivityImpl.java:182) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:276) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
… 14 more
Caused by: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:314) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:107) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
… 1 more
Caused by: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:216) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:107) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
… 1 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method. (Service: S3, Status Code: 403, Request ID: null)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:68) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.17.271.jar:?]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:63) ~[aws-core-2.17.271.jar:?]
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:4483) ~[s3-2.17.271.jar:?]
at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:8373) ~[s3-2.17.271.jar:?]
at io.airbyte.workers.storage.S3DocumentStoreClient.read(S3DocumentStoreClient.java:74) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocument(AsyncOrchestratorPodProcess.java:230) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.checkStatus(AsyncOrchestratorPodProcess.java:234) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.process.AsyncOrchestratorPodProcess.getDocStoreStatus(AsyncOrchestratorPodProcess.java:243) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.sync.LauncherWorker.lambda$run$3(LauncherWorker.java:168) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:309) ~[io.airbyte-airbyte-commons-temporal-0.40.18.jar:?]
at io.airbyte.workers.sync.LauncherWorker.run(LauncherWorker.java:107) ~[io.airbyte-airbyte-commons-worker-0.40.18.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.18.jar:?]
… 1 more
2022-11-16 19:32:58 e[32mINFOe[m i.a.w.t.s.ReplicationActivityImpl(getContainerLauncherWorkerFactory):272 - received response from from jobsApi.getJobInfoLight: class JobInfoLightRead {
job: class JobRead {
id: 2275
configType: sync
configId: 4f13721f-b759-456b-9b9a-d8cb5dfcc4a5
createdAt: 1668627141
updatedAt: 1668627144
status: running
resetConfig: null
}
}
2022-11-16 19:32:24 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.auth_type: must be a constant value Client, $.client_id: is missing but it is required, $.client_secret: is missing but it is required, $.refresh_token: is missing but it is required
2022-11-16 19:32:24 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.service_account_info: object found, string expected
2022-11-16 19:32:24 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.method: must be a constant value Standard
2022-11-16 19:32:24 e[32mINFOe[m i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.credential.hmac_key_secret: object found, string expected, $.credential.hmac_key_access_id: object found, string expected
2022-11-16 19:32:24 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):138 - Cloud storage job log path: /workspace/2275/1/logs.log
2022-11-16 19:32:24 e[32mINFOe[m i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.18
2022-11-16 19:32:24 e[32mINFOe[m i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-google-sheets-check-2275-1-kaolh for airbyte/source-google-sheets:0.2.21 with resources

[Discourse post]

@MantasEql
Copy link

same issues here

@marcosmarxm
Copy link
Member Author

Comment made from Zendesk by Sajarin on 2022-11-30 at 16:33:

Hey @ismaelheelio, 
 
I believe this is still a pending issue. Follow the thread on Github for now 

@MantasEql
Copy link

MantasEql commented Nov 30, 2022 •
edited
Loading

We believe the issue on our end is with the way AirByte sets up the temp tables on BigQuery. For some reason, for Integer fields it sets Float fields, and when AirByte sends an avro file, it fails on BigQuery side. This is when using the de-normalized BigQuery connection.

@marcosmarxm marcosmarxm changed the title Source Google Sheets - sync fails when google sheets has a Data Connector to Big Query Source Google Sheets: sync fails when google sheets has a Data Connector to Big Query Nov 30, 2022
@YowanR
Copy link
Contributor

YowanR commented Jan 4, 2023

@lazebnyi Can you take a look, please?

@lazebnyi lazebnyi reopened this Jan 12, 2023
@grubberr grubberr self-assigned this Feb 15, 2023
@midavadim midavadim self-assigned this Feb 20, 2023
@midavadim midavadim linked a pull request Feb 21, 2023 that will close this issue
🎉 Source Google Sheets - skip non-grid sheets #23292
Merged
37 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@midavadim midavadim

Labels
area/connectors Connector related issues connectors/source/google-sheets discourse team/connectors-python type/bug Something isn't working zendesk
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

🎉 Source Google Sheets - skip non-grid sheets airbytehq/airbyte
8 participants
@grubberr @midavadim @marcosmarxm @sajarin @YowanR @lazebnyi @octavia-squidington-iii @MantasEql