[terraform] Problem Adding 10 or more streams to connection

[corbin-Tprime]

Hi All,

I was working on creating a new connection that uses a redshift source and an s3 destination. I was able to create the source and destinations just fine with Airbyte/terraform. However, when trying to create the connection, the API seems to fail when trying to add more than 10 streams to the configuration.

The error I receive:

Error: failure to invoke API
│ 
│   with airbyte_connection.redacted_name,
│   on connections.tf line 5, in resource "airbyte_connection" "redacted_name":
│    5: resource "airbyte_connection" "redacted_name" {
│ 
│ error sending request: Patch "https://airbyte.redactedname.com/v1/connections/redacted-id": context deadline exceeded (Client.Timeout
│ exceeded while awaiting headers)

I did redact some of the information in the error message due to the nature of the work. That said, it shouldn't be relevant to the issue as creating the connection does work. We just can't add more than 10 streams to the configuration. Once we try to add an 11th+, we receive the API error listed above.

Thanks!

[corbin-Tprime]

@marcosmarxm Any idea on a potential work around or if this has been resolved?

[pgrant87]

The changes can be found here, #29506 This is a similar issue to yours but not exactly the same error. A PR was put up for this, comment on Oct 20:

That release both includes some increased default timeouts as well as a change to how our micronaut server allocates threads in order to avoid timing out.

The main change to the settings is http:// was added to INTERNAL_API_HOST either in .env for docker-compose installs or the env config map for kubernetes/helm. I noticed this tripped a few users up

I have also tried to replicate further this morning using terraform to set up the connection but again mine succeded without error. Can I ask, how long does it take before your terraform apply errors? Currently my requests take about 15 seconds to process but they get applied with no issue

[corbin-Tprime]

@pgrant87 When I try to add http:// in the INTERNAL_API_HOST variable, I get a 502 gateway error when booting my containers back up. What version of airbyte do I need to be on for the above mentioned fix to work?

In terms of timeout, the apply attempts to apply for 60 seconds then produces the timeout error. The connection does get created, but the state is not stored so i have to go in and then import the state from the created connection.

[pgrant87]

That fix should be in 0.50.31 onwards. Would it be possible to attach the file where you'd added the http:// with any sensitive info redacted so I can take a look?

[corbin-Tprime]

@pgrant87 We just have the out of box .env file for version 0.50.33, we use param store to pass in most variables.

Here is the file:
env-file.txt

[pgrant87]

Thanks for passing that on, your .env looks absolutely fine, nothing out of the ordinary. When you received the 502 gateway error when you initially added the http:// to INTERNAL_API_HOST, had all containers started without error or did you notice any other issues when spinning up?

On the Terraform side of things, can I ask, do you have many other large resources? My testing simply consisted of a source, destination and a connection with lots of streams being applied but they were always dealt with within about 20 seconds