In Airbyte Open Source, while configuring File connectors, all are failing with "An unknown error occurred. (HTTP 504)"

[saurabhpandeyt]

Connector Name

File (CSV, JSON, Excel, Feather, Parquet)

Connector Version

v0.63.8 source-file:0.5.3

What step the error happened?

Configuring a new connector

Relevant information

I installed latest abctl 0.8.2.0 version on ubuntu and trying to create file connector. All the connector including https source using https://storage.googleapis.com/covid19-open-data/v2/latest/epidemiology.csv is failing with error "An unknown error occurred. (HTTP 504)"

Relevant log output

There are no logs available this message is displayed on screen.

Contribute

• Yes, I want to contribute

[natikgadzhi]

Not good. Can you try this in Airbyte Cloud and tell me if that works?

[marcosmarxm]

I having the same issue with abctl latest version.

[marcosmarxm]

This is very likely similar to #42404

[saurabhpandeyt]

For web sources this is not happening in the airbyte cloud for public https source Regards, Saurabh Pandey

…

On Sun, Jul 21, 2024 at 1:12 AM Natik Gadzhi ***@***.***> wrote: Not good. Can you try this in Airbyte Cloud and tell me if that works? — Reply to this email directly, view it on GitHub <#42135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUF3BATQHA2HOXMQ5HW2WN3ZNM7LJAVCNFSM6AAAAABLFYET7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBRGQ3TCOBUHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[saurabhpandeyt]

Not good. Can you try this in Airbyte Cloud and tell me if that works?

1. For web sources airbyte cloud is able to connect public https source, however same is not working for local installation.

2. In addition, error message is more clear in airbyte cloud then airbyte local.

AirbyteCloud Error Message: Failed to load scp:///home/saurabh/or.txt. You could have provided an invalid URL, please verify it: TimeoutError(110, 'Connection timed out').

AirbyteLocal Error Message: An unknown error occurred. (HTTP 504)

3. Using airbyte cloud, I am not able to connect to local unix machine as well. The same is not working for local ubuntu installation as well.

Regards,
Saurabh Pandey

[johnwasserman]

Hi @saurabhpandeyt for abctl I was able to reproduce the issue and confirmed 504 was caused by under-resourced deployment in docker

you can verify this by:

1. Running kubectl get po -n airbyte-abctl
2. identifying any pods with status pending and "check" in the name (e.x. my e2e connector was ion-e2e-test-check-adef9c6a-0eaf-43ee-966d-cfcd905adda0-0-mmyfl
3. Describe pod kubectl describe po -n airbyte-abctl [pending-pod-name]
4. Confirming there are logs showing insufficient resourcing

After increasing my Docker Desktop resource limits (8CPU and 12 Gi) and then running abctl local uninstall then abctl local install my 504 issue went away when creating connectors.

Let me know if this was your issue as well or not!

[natikgadzhi]

John, can you confirm that on _any_ default clean setup of abctl this is reproducible? If so, can you please ping Bryce and Cole and ask to tweak resources so it runs out of the box?

[saurabhpandeyt]

Hi, docker image oos installation that was given and I am getting as in point 2 below. 1. installation command and file values.yaml below abctl local install --values ./values.yaml ***@***.***:~$ cat ./values.yaml # values.yaml airbyte: version: 0.8.2 resources: limits: cpu: 8 memory: 12Gi storage: size: 50Gi 2. kubectl get po -n airbyte-abctl ***@***.***:~ I used ***@***.***:~$ kubectl get po -n airbyte-abctl E0723 00:42:04.600495 17990 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused E0723 00:42:04.604372 17990 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused E0723 00:42:04.605867 17990 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused E0723 00:42:04.606209 17990 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused E0723 00:42:04.607637 17990 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused

…

On Mon, Jul 22, 2024 at 11:55 PM Natik Gadzhi ***@***.***> wrote: John, can you confirm that on _any_ default clean setup of abctl this is reproducible? If so, can you please ping Bryce and Cole and ask to tweak resources so it runs out of the box? — Reply to this email directly, view it on GitHub <#42135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUF3BAUKHG2HNB6VRUXIGMTZNXH23AVCNFSM6AAAAABLFYET7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBUGIYDANJQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[marcosmarxm]

@saurabhpandeyt where are you running Airbyte? an EC2 instance? How much memory/cpu/disk the instance has?

[marcosmarxm]

@perangel can we close this?