Airbyte GitHub Sync Fails: Something went wrong within the destination connector

[horosin]

Describe the bug

I am experiencing an issue were GitHub Sync always fails. It happens at vcs_PullRequestComment stage. Other data gets synced properly and is available via Metabase and Hasura.

I tried it 5 times, sync always fails after 25 minutes to 1h.

It appears as if newly synced items violate unique constraints but I have no idea why that happens.

To Reproduce
Steps to reproduce the behavior:

1. Go to http://localhost:8000
2. Click on 'Launch' next to GitHub connections
3. Go to details and see failed logs

Expected behavior
Sync succeeds and data is available via Metabase and Hasura.

Screenshots and logs

0 Bytesno recordsno records22m 55sSync
Failure Origin: destination, Message: Something went wrong within the destination connector

2023-08-22 10:46:17 - Additional Failure Information: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed

I'm not able to provide unreacted logs because they contain pull request comments with AWS deployment details (from pull request comments).

023-08-22 10:45:46 �[43mdestination�[0m > Error while saving batch: [{"extensions":{"internal":{"statement":"WITH \"vcs_PullRequestComment__mutation_result\" AS (INSERT INTO \"public\".\"vcs_PullRequestComment\" ( \"origin\"
 \"createdAt\"
 \"uid\"
 \"pullRequest\"
 \"author\"
 \"id\"
 \"updatedAt\"
 \"number\"
 \"comment\"
 \"refreshedAt\" ) VALUES (('github_source')::text
 ('2023-02-08T15:26:36Z')::timestamptz
 ('1100296384')::text

')::text
 DEFAULT) ON CONFLICT ON CONSTRAINT "vcs_PullRequestComment_pkey" DO UPDATE SET "origin" = EXCLUDED."origin"
"createdAt" = EXCLUDED."createdAt"
"author" = EXCLUDED."author"
"updatedAt" = EXCLUDED."updatedAt"
"number" = EXCLUDED."number"
"comment" = EXCLUDED."comment" WHERE (('true') AND ('true')) RETURNING * 
 ('true')::boolean AS "check__constraint") SELECT  json_build_object('affected_rows'
 (SELECT  COUNT(*)  FROM "vcs_PullRequestComment__mutation_result"      )
 'returning_columns'
 (SELECT  coalesce(json_agg("root" )
 '[]' ) AS "root" FROM  (SELECT  row_to_json((SELECT  "_1_e"  FROM  (SELECT  ("_0_root.base"."origin")::text AS "origin"
 ("_0_root.base"."createdAt")::text AS "createdAt"
 ("_0_root.base"."uid")::text AS "uid"
 ("_0_root.base"."pullRequest")::text AS "pullRequest"
 ("_0_root.base"."author")::text AS "author"
 ("_0_root.base"."id")::text AS "id"
 ("_0_root.base"."updatedAt")::text AS "updatedAt"
 ("_0_root.base"."number")::text AS "number"
 ("_0_root.base"."comment")::text AS "comment"
 ("_0_root.base"."refreshedAt")::text AS "refreshedAt"       ) AS "_1_e"      ) ) AS "root" FROM  (SELECT  *  FROM "vcs_PullRequestComment__mutation_result" WHERE ('true')     ) AS "_0_root.base"      ) AS "_2_root"      ) ) 
 (SELECT  coalesce(bool_and("check__constraint" )
 'true' )  FROM "vcs_PullRequestComment__mutation_result"      )        \"
\"prepared\":false
\"error\":{\"exec_status\":\"FatalError\"
\"hint\":null
\"message\":\"index row requires 9352 bytes
 maximum size is 8191\"
\"status_code\":\"54000\"
\"description\":null}
\"arguments\":[]}
\"path\":\"$.selectionSet.insert_vcs_PullRequestComment_one.args.object[0]\"
\"code\":\"unexpected\"}
\"message\":\"database query error\"}] - {\"jse_shortmsg\":\"Failed to write vcs_PullRequestComment record with query 'mutation { insert_vcs_PullRequestComment_one (object: {number: 1100296384
 uid: "1100296384"
 comment: "

2023-08-22 10:47:17 �[33mWARN�[m i.a.c.i.LineGobbler(voidCall):119 - airbyte-source gobbler IOException: Stream closed. Typically happens when cancelling a job.
2023-08-22 10:47:17 �[1;31mERROR�[m i.a.w.g.DefaultReplicationWorker(run):188 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:181) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
	Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 1. This warning is normal if the job was cancelled.
		at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:140) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at java.lang.Thread.run(Thread.java:833) ~[?:?]
	Suppressed: java.io.IOException: Stream closed
		at java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:445) ~[?:?]
		at java.io.OutputStream.write(OutputStream.java:162) ~[?:?]
		at java.io.BufferedOutputStream.write(BufferedOutputStream.java:123) ~[?:?]
		at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234) ~[?:?]
		at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:313) ~[?:?]
		at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:318) ~[?:?]
		at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:160) ~[?:?]
		at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:248) ~[?:?]
		at java.io.BufferedWriter.flush(BufferedWriter.java:257) ~[?:?]
		at io.airbyte.workers.internal.DefaultAirbyteDestination.notifyEndOfInput(DefaultAirbyteDestination.java:98) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.internal.DefaultAirbyteDestination.close(DefaultAirbyteDestination.java:111) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:145) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:68) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:159) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
		at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450) ~[io.airbyte-airbyte-workers-0.40.9.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more
2023-08-22 10:47:17 �[33mWARN�[m i.a.w.g.DefaultReplicationWorker(run):220 - Could not reliably determine committed record counts, committed record stats will be set to null
2023-08-22 10:47:17 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):279 - Source output at least one state message
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):285 - State capture: Updated state to: Optional[io.airbyte.config.State@38bf5de4[state=[{"type":"STREAM","stream":{"stream_descriptor":{"name":"review_comments"},"stream_state":
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):300 - sync summary: {
  "status" : "failed",
  "recordsSynced" : 35177,
  "bytesSynced" : 39556741,
  "startTime" : 1692699895332,
  "endTime" : 1692701237993,
  "totalStats" : {
    "recordsEmitted" : 35177,
    "bytesEmitted" : 39556741,
    "sourceStateMessagesEmitted" : 1739,
    "destinationStateMessagesEmitted" : 1717,
    "meanSecondsBeforeSourceStateMessageEmitted" : 3,
    "maxSecondsBeforeSourceStateMessageEmitted" : 250,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 5,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0
  },
  "streamStats" : [ {
    "streamName" : "github_source__github__pull_requests",
    "stats" : {
      "recordsEmitted" : 194,
      "bytesEmitted" : 2803965
    }
  }, {
    "streamName" : "github_source__github__collaborators",
    "stats" : {
      "recordsEmitted" : 14080,
      "bytesEmitted" : 15532594
    }
  }, {
    "streamName" : "github_source__github__repositories",
    "stats" : {
      "recordsEmitted" : 228,
      "bytesEmitted" : 1253144
    }
  }, {
    "streamName" : "github_source__github__releases",
    "stats" : {
      "recordsEmitted" : 16,
      "bytesEmitted" : 35073
    }
  }, {
    "streamName" : "github_source__github__issues",
    "stats" : {
      "recordsEmitted" : 195,
      "bytesEmitted" : 1168345
    }
  }, {
    "streamName" : "github_source__github__commits",
    "stats" : {
      "recordsEmitted" : 524,
      "bytesEmitted" : 1986803
    }
  }, {
    "streamName" : "github_source__github__review_comments",
    "stats" : {
      "recordsEmitted" : 266,
      "bytesEmitted" : 1116811
    }
  }, {
    "streamName" : "github_source__github__organizations",
    "stats" : {
      "recordsEmitted" : 1,
      "bytesEmitted" : 1775
    }
  }, {
    "streamName" : "github_source__github__branches",
    "stats" : {
      "recordsEmitted" : 2608,
      "bytesEmitted" : 774681
    }
  }, {
    "streamName" : "github_source__github__issue_labels",
    "stats" : {
      "recordsEmitted" : 2985,
      "bytesEmitted" : 826570
    }
  }, {
    "streamName" : "github_source__github__assignees",
    "stats" : {
      "recordsEmitted" : 14080,
      "bytesEmitted" : 14056980
    }
  } ]
}
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.g.DefaultReplicationWorker(run):301 - failures: [ {
  "failureOrigin" : "destination",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed",
  "externalMessage" : "Something went wrong within the destination connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 8,
    "connector_command" : "write"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:346)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: java.io.IOException: Broken pipe\n\tat java.base/java.io.FileOutputStream.writeBytes(Native Method)\n\tat java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)\n\tat java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)\n\tat java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)\n\tat java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132)\n\tat java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:205)\n\tat java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:120)\n\tat java.base/java.io.BufferedWriter.write(BufferedWriter.java:233)\n\tat java.base/java.io.Writer.write(Writer.java:249)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.accept(DefaultAirbyteDestination.java:90)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:343)\n\t... 4 more\n",
  "timestamp" : 1692701177835
}, {
  "failureOrigin" : "destination",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1",
  "externalMessage" : "Something went wrong within the destination connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 8,
    "connector_command" : "write"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\n",
  "timestamp" : 1692701177835
} ]
2023-08-22 10:47:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - 
2023-08-22 10:47:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - ----- END REPLICATION -----
2023-08-22 10:47:18 �[32mINFO�[m i.a.c.i.LineGobbler(voidCall):114 - 
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.t.TemporalAttemptExecution(get):132 - Stopping cancellation check scheduling...
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.t.s.ReplicationActivityImpl(lambda$replicate$3):149 - sync summary: io.airbyte.config.StandardSyncOutput@59107209[standardSyncSummary=io.airbyte.config.StandardSyncSummary@608888a8[status=failed,recordsSynced=35177,bytesSynced=39556741,startTime=1692699895332,endTime=1692701237993,totalStats=io.airbyte.config.SyncStats@16490e94[recordsEmitted=35177,bytesEmitted=39556741,sourceStateMessagesEmitted=1739,destinationStateMessagesEmitted=1717,recordsCommitted=<null>,meanSecondsBeforeSourceStateMessageEmitted=3,maxSecondsBeforeSourceStateMessageEmitted=250,maxSecondsBetweenStateMessageEmittedandCommitted=5,meanSecondsBetweenStateMessageEmittedandCommitted=0,additionalProperties={}],streamStats=[io.airbyte.config.StreamSyncStats@72431d5f[streamName=github_source__github__pull_requests,stats=io.airbyte.config.SyncStats@2899b924[recordsEmitted=194,bytesEmitted=2803965,sourceStateMessagesEmitted=<null>,destinationStateMessagesEmitted=<null>,recordsCommitted=<null>,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,additionalProperties={}]], 
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:346)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 3 more
Caused by: java.io.IOException: Broken pipe
	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:349)
	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:234)
	at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:304)
	at java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
	at java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:132)
	at java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:205)
	at java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:120)
	at java.base/java.io.BufferedWriter.write(BufferedWriter.java:233)
	at java.base/java.io.Writer.write(Writer.java:249)
	at io.airbyte.workers.internal.DefaultAirbyteDestination.accept(DefaultAirbyteDestination.java:90)
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:343)
	... 4 more
,retryable=<null>,timestamp=1692701177835], io.airbyte.config.FailureReason@32058b7[failureOrigin=destination,failureType=<null>,internalMessage=io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1,externalMessage=Something went wrong within the destination connector,metadata=io.airbyte.config.Metadata@3c77b2c0[additionalProperties={attemptNumber=0, jobId=8, connector_command=write}],stacktrace=java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process exited with non-zero exit code 1
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getDestinationOutputRunnable$7(DefaultReplicationWorker.java:450)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 3 more
,retryable=<null>,timestamp=1692701177835]]]
2023-08-22 10:47:18 �[32mINFO�[m i.a.w.t.TemporalUtils(withBackgroundHeartbeat):279 - Stopping temporal heartbeating...

Deployment

• Local
• MacOS 13.4 (22F66), MacBook Air M1
• Docker version 23.0.5, build bc4487a
• Docker Compose version v2.17.3

Thank you for providing us with a great tool and help with this bug!

[tovbinm]

Fixed

[thomas-gerber]

@horosin you can go to Airbyte (localhost:8000) > Settings > destinations and set the Faros Destination version to 0.6.11. Alternatively, just stop and start Faros, and it will pick it up.

[horosin]

@thomas-gerber Thank you, I really appreciate the help. Manual change helped!

@tovbinm thank you as well