Repository navigation
Destination Bigquery returns 404 when uploading data + resumable #3549
Description
Activity
I'm syncing 3 years of real data here but it still baffles me that this can take so long
It took 12 hours for something to happen and crashed because of a 404 Error
it is still marked as running
yes, maybe the dataset is too big? I reduced the dataset from stripe and removed events and stuff and after that every sync took around an hour even on sync that take 3 years of data
User logs syncing Stripe to Bigquery
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.24.4-alpha
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) TemporalAttemptExecution(get):111 - Executing worker wrapper. Airbyte version: 0.24.4-alpha
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) DefaultReplicationWorker(run):97 - start sync worker. job id: 4 attempt id: 0
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) DefaultReplicationWorker(run):106 - configured sync modes: {null.charges=incremental - append, null.customer_balance_transactions=full_refresh - overwrite, null.coupons=incremental - append, null.products=incremental - append, null.balance_transactions=incremental - append, null.refunds=incremental - append, null.invoice_line_items=full_refresh - overwrite, null.transfers=incremental - append, null.payouts=incremental - append, null.customers=incremental - append, null.plans=incremental - append, null.disputes=incremental - append, null.subscriptions=incremental - append, null.invoice_items=incremental - append, null.events=incremental - append, null.bank_accounts=full_refresh - overwrite, null.subscription_items=full_refresh - overwrite, null.invoices=incremental - append}
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) DefaultAirbyteDestination(start):81 - Running destination...
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) LineGobbler(voidCall):69 - Checking if airbyte/destination-bigquery:0.3.3 exists...
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) LineGobbler(voidCall):69 - airbyte/destination-bigquery:0.3.3 was found locally.
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) DockerProcessFactory(create):111 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4/0 --network host airbyte/destination-bigquery:0.3.3 write --config destination_config.json --catalog destination_catalog.json
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) LineGobbler(voidCall):69 - Checking if airbyte/source-stripe:0.1.11 exists...
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) LineGobbler(voidCall):69 - airbyte/source-stripe:0.1.11 was found locally.
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) DockerProcessFactory(create):111 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/4/0 --network host airbyte/source-stripe:0.1.11 read --config source_config.json --catalog source_catalog.json
2021-05-31 14:42:37 INFO (/tmp/workspace/4/0) DefaultReplicationWorker(run):132 - Waiting for source thread to join.
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - Exception in thread "main" com.google.cloud.bigquery.BigQueryException: 404 Not Found
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/craftnote-live/jobs?uploadType=resumable&upload_id=ABg5-Uyn1T2bRFLs9Ww15UeF2uZYz4c5JKYhSFwjkOAtJxG8ZX7TIXb79sP5EEC3SQJaTrxmvwiXAjedtS4IHYH7t2uxn2Ebkg
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:705)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:59)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.BaseWriteChannel.flush(BaseWriteChannel.java:112)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.BaseWriteChannel.write(BaseWriteChannel.java:139)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.acceptTracked(BigQueryRecordConsumer.java:110)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.accept(FailureTrackingAirbyteMessageConsumer.java:66)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:132)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:113)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:242)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - Suppressed: com.google.cloud.bigquery.BigQueryException: 404 Not Found
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/craftnote-live/jobs?uploadType=resumable&upload_id=ABg5-UwBpKMxykE_ozdsa2iTwzM_dtdMj9SNZ94qlcOyhhmgDSLpYvbqyWTBBynjFrUpL-ja36wn_2a7fwtRqWyqCy2NkEPKdg
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:705)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:59)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:151)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.lambda$close$0(BigQueryRecordConsumer.java:122)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.commons.lang.Exceptions.castCheckedToRuntime(Exceptions.java:72)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.commons.lang.Exceptions.toRuntime(Exceptions.java:59)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.lambda$close$1(BigQueryRecordConsumer.java:122)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1766)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.helpCC(ForkJoinPool.java:1116)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinPool.externalHelpComplete(ForkJoinPool.java:1966)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinTask.tryExternalHelp(ForkJoinTask.java:378)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinTask.externalAwaitDone(ForkJoinTask.java:323)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:412)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.close(BigQueryRecordConsumer.java:122)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.close(FailureTrackingAirbyteMessageConsumer.java:78)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:126)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - ... 2 more
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/craftnote-live/jobs?uploadType=resumable&upload_id=ABg5-UwBpKMxykE_ozdsa2iTwzM_dtdMj9SNZ94qlcOyhhmgDSLpYvbqyWTBBynjFrUpL-ja36wn_2a7fwtRqWyqCy2NkEPKdg
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:688)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - ... 33 more
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/craftnote-live/jobs?uploadType=resumable&upload_id=ABg5-Uyn1T2bRFLs9Ww15UeF2uZYz4c5JKYhSFwjkOAtJxG8ZX7TIXb79sP5EEC3SQJaTrxmvwiXAjedtS4IHYH7t2uxn2Ebkg
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:688)
2021-06-01 06:23:16 ERROR (/tmp/workspace/4/0) LineGobbler(voidCall):69 - ... 13 more
Sherif A. Nada (@sherifnada) tagging you because this is the second case this error happens. Not sure if checkpointing can handle this because is intermittent failing...
Hi, this is happening to me on Postgres => BQ on a huge database first sync. The sync process gets stuck at that point.
2021-06-09 02:06:07 INFO (/tmp/workspace/56/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-06-09 02:06:07 �[32mINFO�[m i.a.i.s.j.AbstractJdbcSource(lambda$createReadIterator$8):308 - {} - Reading stream assessment_responses. Records read: 20000
2021-06-09 02:06:08 INFO (/tmp/workspace/56/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-06-09 02:06:08 �[32mINFO�[m i.a.i.s.j.AbstractJdbcSource(lambda$createReadIterator$8):308 - {} - Reading stream assessment_responses. Records read: 30000
2021-06-09 02:06:08 INFO (/tmp/workspace/56/0) DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-06-09 02:06:08 �[32mINFO�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):77 - {} - hasFailed: true.
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - Exception in thread "main" com.google.cloud.bigquery.BigQueryException: 404 Not Found
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/useful-proposal-454/jobs?uploadType=resumable&upload_id=ABg5-UwHdi3-fSOUNSt1tTQjj7-DE3Gxm2N8x8sExPIVS7G-J6-mJ22EEezApb40TtA6xhvKN9K8NCnap2OoP672QqZYjmr0EA
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:705)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:59)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.BaseWriteChannel.flush(BaseWriteChannel.java:112)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.BaseWriteChannel.write(BaseWriteChannel.java:139)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.acceptTracked(BigQueryRecordConsumer.java:110)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.accept(FailureTrackingAirbyteMessageConsumer.java:66)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:132)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:113)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:242)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - Suppressed: com.google.cloud.bigquery.BigQueryException: 404 Not Found
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/useful-proposal-454/jobs?uploadType=resumable&upload_id=ABg5-Uy6YIL7DnEd9wNiRd27Ms39-N6TcXMELYbqNaQcmktK-3PKH8JAd-1h54mKdKFGKoUnVvQVOrWbi5zoyg3xzD5JtnBClQ
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:705)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:59)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:151)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.lambda$close$0(BigQueryRecordConsumer.java:122)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.commons.lang.Exceptions.castCheckedToRuntime(Exceptions.java:72)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.commons.lang.Exceptions.toRuntime(Exceptions.java:59)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.lambda$close$1(BigQueryRecordConsumer.java:122)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1766)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.close(BigQueryRecordConsumer.java:122)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.close(FailureTrackingAirbyteMessageConsumer.java:78)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:126)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - ... 2 more
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/useful-proposal-454/jobs?uploadType=resumable&upload_id=ABg5-Uy6YIL7DnEd9wNiRd27Ms39-N6TcXMELYbqNaQcmktK-3PKH8JAd-1h54mKdKFGKoUnVvQVOrWbi5zoyg3xzD5JtnBClQ
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:688)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - ... 29 more
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/useful-proposal-454/jobs?uploadType=resumable&upload_id=ABg5-UwHdi3-fSOUNSt1tTQjj7-DE3Gxm2N8x8sExPIVS7G-J6-mJ22EEezApb40TtA6xhvKN9K8NCnap2OoP672QqZYjmr0EA
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1113)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:688)
2021-06-09 02:06:33 ERROR (/tmp/workspace/56/0) LineGobbler(voidCall):85 - ... 13 more
User could sync after selecting only a subset of tables.
I was able to finish the sync by selecting only 10 tables each time but that was a lot of iterations.
my guess is that the right way to deal with this is bulk loading data into BQ via staging on GCS. Andrés Bravo (@andresbravog) would that be an acceptable solution for you?
Sherif A. Nada (@sherifnada) we already have a workaround by creating a script that waits for each job to finish and ads one more table to the connection each time (controlling when the process gets stuck).
My take here is that the process has two main bugs:
- The process gets frozen and does not fail after the error, meaning there is no second attempt and no release of the resources until the user manually clicks on cancel.
- The process is failing due to a single
404with no retry (this affects long processes needing to restart)
Ideally, we would like to make BQ destination error prune.
Running into a similar issue here with Stripe to BigQuery.
2021-07-13 02:58:24 ERROR () LineGobbler(voidCall):85 - Exception in thread "main" com.google.cloud.bigquery.BigQueryException: 404 Not Found followed by more errors. Status in Airbyte continues to say "running" for 24+ hours.
Seeing the same problem when trying to load a few tables, with one of them containing nearly a Billion rows from MSSQL to BigQuery. The exception happens pretty late in the sync, after it already ran for a few hours.
2021-07-17 18:01:29 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-17 18:01:29 �[33mWARN�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):78 - {} - Airbyte message consumer: failed.
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - Exception in thread "main" com.google.cloud.bigquery.BigQueryException: 404 Not Found
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - PUT https://www.googleapis.com/upload/bigquery/v2/projects/<redacted>/jobs?uploadType=resumable&upload_id=<redacted>
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.write(HttpBigQueryRpc.java:705)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:59)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel$1.call(TableDataWriteChannel.java:54)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.bigquery.TableDataWriteChannel.flushBuffer(TableDataWriteChannel.java:53)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.BaseWriteChannel.flush(BaseWriteChannel.java:112)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at com.google.cloud.BaseWriteChannel.write(BaseWriteChannel.java:139)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.acceptTracked(BigQueryRecordConsumer.java:101)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.accept(FailureTrackingAirbyteMessageConsumer.java:66)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at io.airbyte.integrations.base.IntegrationRunner.consumeWriteStream(IntegrationRunner.java:132)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:113)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - at io.airbyte.integrations.destination.bigquery.BigQueryDestination.main(BigQueryDestination.java:279)
2021-07-17 18:01:32 ERROR () LineGobbler(voidCall):85 - Suppressed: com.google.cloud.bigquery.BigQueryException: 404 Not Found
hey everyone, we're working on this in the current sprint. It's not 100% clear yet what the fix will be, so our first step is to identify that, then we'll have a more concrete ETA. But it's something we're currently working on.
18 remaining items
I've tried twice more with the updated connector. Once with default value for chunk size and once with it set to 5. Neither were successful. Here's the logs for both.
5 chunk size.txt
default chunk size.txt
I'm only getting the error with Stripe -> BigQeury so I tried to find an alternative for that source. Looking through the Fivetran docs I noticed that they don't support BigQuery Sandbox accounts which is what I'm using right now. Not sure if that's related with the issue here but felt like it's worth pointing out. The limitations of sandbox are listed here.
It seems like I've found how to reproduce it. The key point here is not how big is the data set that you are going to upload and how long it's being uploaded. But the main point is how much time did it take to read records from source DB. We create a destination connector's container at the very beginning along with the source connector's container as far as I see far now. The google big query clients and tmp dirs are created at the time when the destination container is created. Then we wait for source to read data meanwhile, the destination one is in an idle state.
I was able to reproduce this issue locally from tests even with only one table\stream and only one record.
- Create a destination client and init google big query stream in it (it initiates automatically at the getConsumer() phase.
- Do a Tread.sleep for test thread for 12 hours keeping the destination container in idle state.
- Write a record to the destination.
At the moment when the native google bigQuery client will try to flush its buffer - it fails. It flushes it either when the buffer is full or onClose depends on what happens first.
Next steps:
- Identify is there any TTL arg in bigQuery clients or any kind of "keep alive" requests.
- If no, then think on how can we rewrite a connector to create a table at the first call to write something (acceptTracked() method). Still need to consider, how to make buffer flushed when started getting messages for the next migrated stream. Google SDK client doesn't provide an option for force flush buffer on another way than just close the connection.
By the way, checked how much memory it takes to write 18 streams with chunk buffer equals to 5MB. Here is the result
UPD: it seems like google SDK client doesn't have any kind of "keep alive" requests. It also doesn't have an explicit option to force flush its buffer other than just close a connection. But we do not have an internal message from the source that says that read completed and no more messages to write expected (as far as I see, but maybe I missing something).
UPD2. As far as I was told we do not have any source connectors that may process several streams in parallel. So we may try to use stream name changing as an indicator that we need to close old stream and open a new one
amazing work Eugene (@etsybaev) , seems like you're getting very close here. So it sounds like the main hypothesis then is if the time difference between two interactions with a BQ Stream ("interact" can mean open, write, or close) then we receive a 404?
Like you noted, I'm not sure there is an "easy" way around this. Ideally we need to create a job as late as possible for each stream. So we could potentially do lazy creation. That doesn't solve the "close" issue though where a stream receives all of its data at the beginning of the sync, but the sync takes long enough that by the time we close the stream it has been a long time.
I think the most guaranteed way around this is using Bulk loading as it would allow us to stage the data on GCS entirely before loading it into bigquery. This is already implemented in Snowflake so we'd need something similar implemented for bigquery. WDYT?
If you agree Eugene (@etsybaev) could you create an issue for this and move it into prioritized for scoping? we should work on that soon.
Hi Sherif A. Nada (@sherifnada).
I've created a follow-up ticket.
I can't be a 100% sure that this is the same issue I was able to reproduce, but yes. In my case it also fails with 404 like this which looks pretty similar (full logs can be found in attached file PerftestLogs.txt):
com.google.cloud.bigquery.BigQueryException: 404 Not Found PUT https://www.googleapis.com/upload/bigquery/v2/projects/dataline-integration-testing/jobs?uploadType=resumable&upload_id=ADPycdvsL3_l8k9SNv1ukO5LmDPkcpoZJJf4hpCRgF9baBKPhf_UrgwmGqfTUfX_5MSaJ0QiXhgaq_JYi5biynNDfgg java.lang.RuntimeException: com.google.cloud.bigquery.BigQueryException: 404 Not Found PUT https://www.googleapis.com/upload/bigquery/v2/projects/dataline-integration-testing/jobs?uploadType=resumable&upload_id=ADPycdvsL3_l8k9SNv1ukO5LmDPkcpoZJJf4hpCRgF9baBKPhf_UrgwmGqfTUfX_5MSaJ0QiXhgaq_JYi5biynNDfgg at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.lambda$close$0(BigQueryRecordConsumer.java:149) at io.airbyte.commons.lang.Exceptions.castCheckedToRuntime(Exceptions.java:72) at io.airbyte.commons.lang.Exceptions.toRuntime(Exceptions.java:59) at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.lambda$close$1(BigQueryRecordConsumer.java:141) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1766) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408) at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:736) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661) at io.airbyte.integrations.destination.bigquery.BigQueryRecordConsumer.close(BigQueryRecordConsumer.java:141) at io.airbyte.integrations.base.FailureTrackingAirbyteMessageConsumer.close(FailureTrackingAirbyteMessageConsumer.java:82) at io.airbyte.integrations.destination.bigquery.BigQueryDestinationPerformanceTest.testWriteSuccess(BigQueryDestinationPerformanceTest.java:212) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:688) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84) at org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115) at org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104) at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:210) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:206) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:131) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at
Hi,
The initial sync of my stripe account with BigQuery was failing because of this issue. So, as recommended I switched to the GCS staging method. Thanks to this, the initial sync worked correctly. But now that the big initial load has passed I would like to switch back to the standard sync method for refreshes which are much smaller, without going through GCS. However when I launch a sync it seems to restart from the beginning, as if Airbyte was not recognizing the sync done through GCS. Is it expected and is there any solution to this problem ?
Expected Behavior
Sync between Shopify and BQ works
Current Behavior
It's not totally clear why this happens...
BigQuery returns 404 when uploading data after running a few hours.
From Google Cloud API Page here
Logs
Steps to Reproduce
It's not clear how to reproduce. Maybe this happens when trying to sync large Shopify datasets to BQ.
Severity of the bug for you
Very low / Low / Medium / High / Critical
Airbyte Version
0.23.0-alpha
Connector Version (if applicable)
0.2.2 Shopify
Additional context
Slack convo
┆Issue is synchronized with this Asana task by Unito