Proxy causes Gateway Timeout for dynamic schema discovery

[henriblancke]

Environment

• Airbyte version: >=v0.40.16
• OS Version / Instance: Any
• Deployment: Docker
• Source Connector and version: source-s3
• Step where error happened: Setup new connection -> discover dynamic schema

Current Behavior

When adding a new S3 connection, discovering the schema results in 504 Gateway Timeout. Trying again often resolves the issue. I wonder if the proxy's nginx default timeout should be increased to allow more time for dynamically fetching schemas like with the s3 connector (or any source where discovery can take +60 seconds)

Expected Behavior

The API returns the discovered schema.

Steps to Reproduce

1. Setup source-s3 pointing to a bucket with larger files
2. Setup a connection to any destination
3. Wait for a non json response in the UI, in Inspect > Console it shows a 504

[rajeshkp]

Is it large number of files by count or by size, can you be specific

[marcelopio]

I have the same problem with the Microsoft dataverse connector. And on slack people are also complaining seeing the issue with MSSQL on large databases. It does not seem like a S3 issue

[marcelopio]

We probably need to add these configs to the nginx template for airbyte proxy

proxy_connect_timeout       300;
proxy_send_timeout          300;
proxy_read_timeout          300;
send_timeout                300;

[murat-cetinkaya]

I started to get Failed to fetch schema. Please try again error for the Zuora source after I migrated to v 0.40.15 ( the version that nginx was added). I was trying to find the reason and I thinks this issue is the answer.

@marcelopio how do we add those configs to the nginx template? Have you tried it? Did it work?
Thanks.

[marcelopio]

@murat-cetinkaya , added a PR, if you want you can apply that patch and make a custom proxy image until Airbyte merge the PR