Source MSSQL: sync 1mm throws error when try 80mm records not OOM

[marcosmarxm]

Enviroment

• Airbyte version: ~
• OS Version / Instance: ~
• Deployment: ~
• Source Connector and version: ~
• Destination Connector and version: ~
• Severity: Very Low / Low / Medium / High / Critical
• Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte

Current Behavior

Slack convo

Now I tried to expand my testing and tried to load a big table having around 80 Mn records. I started the sync and went to the snowflake UI tool and saw table creation was successful and INSERT started into the temp. It was inserting at the rate of 10000 records /2.5 sec. I left it for a while and found the job failed after 3 hours with communication error. Same thing happened when I try to re-run the process. Do we have any limitations with the number of records being processed with the open source version of Airbyte ? Do we have any enterprise version to replicate large chunk of data ?

Error location: Reply.fill() - insufficient data (-1).  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001

Tried syncing a table on DB2 with 1.15 Million records to Snowflake and was succeeded in 15 mins. So the connections are all working fine

Expected Behavior

Tell us what should happen.

Logs

logs-10-1.txt

Steps to Reproduce

Are you willing to submit a PR?

Remove this with your answer.

[anoopsharp]

Airbyte version: ~0.29.13-alpha
OS Version / Instance: ~ Windows 10
Deployment: ~Docker
Source Connector and version: ~IBM DB2 11.1
Destination Connector and version: ~ Snowflake 5.34.1
Severity: Very Low / Low / Medium / High / Critical Medium
Step where error happened: Deploy / Sync job / Setup new connection / Update connector / Upgrade Airbyte Sync Job