Intercom is slow

[exalate-issue-sync]

Why intercom is very slow to get data after i upgrade to 0.35.2-alpha
logs-242-0.txt (19.7 KB)

[Discourse post]

[exalate-issue-sync]

Augustin Lafanechère commented: Hey @ayyoub,
Could you please try to downgrade the intercom connector version to 0.1.16 and check if the read speed is different?

Thank you!

[exalate-issue-sync]

ayyoub commented: already upgrade Airbyte again to v0.36.4-alpha and use intercom connector 0.1.16 but still slow even i just get 2 days from start date on conversations

[Discourse post]

[exalate-issue-sync]

Augustin Lafanechère commented: I don't think this can be related to your Airbyte version. It's more likely to be a delay introduced by Intercom for throttling requests.
Could you please share the logs of a sync with 0.35.2 to check how fast it was before your upgrade?

We have two open issues to improve Intercom's performance, I suggest you subscribe to these to receive updates once we work on this.

• Source Intercom: Support high volume syncs #11595
• [EPIC] Source Intercom to GA #11756

[exalate-issue-sync]

ayyoub commented: after running 3 days but worker canceled accidentally @alafanechere
logs-540.txt (2.7 MB)

[Discourse post]

[exalate-issue-sync]

Augustin Lafanechère commented: Hey @ayyoub,
I had the chance to chat with other intercom users and they have similar loading speeds for the conversation stream.
Feel free to reach the intercom support to understand if some throttling is happening on their side.
Airbyte's sync has a maximum duration of three days, this is why your sync stopped.
You can change the value of the SYNC_JOB_MAX_TIMEOUT_DAYS environment variable in your .env file, the default value is 3, so please increase to a superior value.

[exalate-issue-sync]

ayyoub commented: oke got it thanks @alafanechere

[Discourse post]

[exalate-issue-sync]

ayyoub commented: should i restar airbyte after changed the variable ? @alafanechere

[Discourse post]

[exalate-issue-sync]

Augustin Lafanechère commented: Yes, the worker and the scheduler at least.

[exalate-issue-sync]

ayyoub commented: whats the endpoint airbyte used for airbyte to get intercom conversations ? @alafanechere

[Discourse post]

[exalate-issue-sync]

Augustin Lafanechère commented: source-intercom uses this endpoint for the Conversations stream: https://api.intercom.io/conversations
You can find more details about how the connector is implemented here

[exalate-issue-sync]

ayyoub commented: this is new case again i think, after 15 hours running intercom contact force cancel its worker @alafanechere
logs-1679 (1).txt (653.8 KB)

[Discourse post]

[exalate-issue-sync]

Augustin Lafanechère commented: This behavior is interesting:
The job is considered as succeeded according to the logs:

2022-05-06 06:00:05 �[36mDEBUG�[m i.a.s.a.JobSubmitter(lambda$submitJob$2):135 - Job id 1679 succeeded2022-05-06 06:00:05 �[36mDEBUG�[m i.a.s.a.JobSubmitter(lambda$submitJob$4):166 - Job id 1679 cleared

The sources emitted records but these records are not committed to the destinations

recordsEmitted=915851,bytesEmitted=1907436593,stateMessagesEmitted=0,recordsCommitted=0

I'm under the impression the source connector read all the data on your streams (~915000), do you think it's the case?

Maybe the error is on the destination side now. Could you please upgrade your BigQuery connector to the latest version and chose the GCS upload mode?