Unexpected error could not find image: Airbyte/*:version

[ahmedhamidibr]

Connector Name

source-salesforce

Connector Version

2.5.13

What step the error happened?

During the sync

Relevant information

I am encountering an issue with Airbyte (version 0.60.1) deployed via Docker on an Azure VM. Here are the details of my environment:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Issue Description

When running a sync, Airbyte fails to pull the respective source image. This issue occurs not only with the source-salesforce but also with source-google-analytics-data-api. Both sources fail to pull the required images.

Additional Information

-Disk Space: The disk has sufficient space, as shown below:

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       124G   76G   48G  62% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           3.2G  2.7M  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
efivarfs        128M  9.9K  128M   1% /sys/firmware/efi/efivars
/dev/sda15      105M  6.1M   99M   6% /boot/efi
/dev/sdb1        32G  7.1G   23G  24% /mnt
tmpfs           1.6G     0  1.6G   0% /run/user/1000

-Network: The instance has internet access, and there are no firewall rules blocking the network. When pulling the image manually, it succeeds:

docker pull airbyte/source-salesforce:2.5.13  
2.5.13: Pulling from airbyte/source-salesforce
Digest: sha256:1b1961f3df579b0a98dd7e13e8229cdf42880c478ff694b8bde67f5f28a17aec
Status: Image is up to date for airbyte/source-salesforce:2.5.13
docker.io/airbyte/source-salesforce:2.5.13

-Intermittent soultion: Sometimes restarting airbyte (via docker compose) resolves the issue, but after a while, it happens again. Occasionally, even restarting airbyte does not solve the problem.

Can you please provide guidance on resolving this issue?

Thank you.

Relevant log output

2024-06-11 14:32:47 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=4, totalCompleteFailures=4, successivePartialFailures=0, totalPartialFailures=0)
2024-06-11 14:32:47 platform > Backing off for: 4 minutes 30 seconds.
2024-06-11 14:37:18 platform > Docker volume job log path: /tmp/workspace/9097/4/logs.log
2024-06-11 14:37:18 platform > Executing worker wrapper. Airbyte version: 0.60.1
2024-06-11 14:37:18 platform > Attempt 0 to save workflow id for cancellation
2024-06-11 14:37:18 platform > 
2024-06-11 14:37:18 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-06-11 14:37:18 platform > ----- START CHECK -----
2024-06-11 14:37:18 platform > 
2024-06-11 14:37:18 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-06-11 14:37:18 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-06-11 14:37:18 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-06-11 14:37:18 platform > Checking if airbyte/source-salesforce:2.5.13 exists...
2024-06-11 14:37:18 platform > airbyte/source-salesforce:2.5.13 not found locally. Attempting to pull the image...
2024-06-11 14:37:18 platform > Image does not exist.
2024-06-11 14:37:18 platform > Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/source-salesforce:2.5.13
	at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:117) ~[io.airbyte-airbyte-commons-worker-0.60.1.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:149) ~[io.airbyte-airbyte-commons-worker-0.60.1.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.60.1.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:44) ~[io.airbyte-airbyte-commons-worker-0.60.1.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.60.1.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.lambda$runWithJobOutput$1(CheckConnectionActivityImpl.java:226) ~[io.airbyte-airbyte-workers-0.60.1.jar:?]
	at io.airbyte.commons.temporal.HeartbeatUtils.withBackgroundHeartbeat(HeartbeatUtils.java:57) ~[io.airbyte-airbyte-commons-temporal-core-0.60.1.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:211) ~[io.airbyte-airbyte-workers-0.60.1.jar:?]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:278) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:243) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:216) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:105) ~[temporal-sdk-1.22.3.jar:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-06-11 14:37:18 platform > 
2024-06-11 14:37:18 platform > ----- END CHECK -----
2024-06-11 14:37:18 platform > 
2024-06-11 14:37:18 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=5, totalCompleteFailures=5, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 13 minutes 30 seconds
2024-06-11 14:37:18 platform > Failing job: 9097, reason: Job failed after too many retries for connection 15a21571-14e6-4ba7-9b8a-b6ed7a7dc560

Contribute

• Yes, I want to contribute

[marcosmarxm]

Is this happening only for Snowflake or other connectors too?

[ahmedhamidibr]

@marcosmarxm source-google-analytics-data-api:2.4.2 as well

[sbuse]

I work with the same setup, Airbyte (version 0.60.1) deployed via Docker on an Azure VM and I experienced the same with bing-ads and google-ads. Restarting with ./run-ab-platform.sh -b fixed the issue for now. Let's see when it comes back.

2024-06-11 15:31:18 platform > Checking if airbyte/source-bing-ads:2.6.2 exists...
2024-06-11 15:31:18 platform > airbyte/source-bing-ads:2.6.2 not found locally. Attempting to pull the image...
2024-06-11 15:31:19 platform > Image does not exist.
2024-06-11 15:31:19 platform > Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/source-bing-ads:2.6.2

2024-06-11 14:10:49 platform > airbyte/source-google-ads:3.5.0 not found locally. Attempting to pull the image...
2024-06-11 14:10:50 platform > Image does not exist.
2024-06-11 14:10:50 platform > Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/source-google-ads:3.5.0

[wwfch-cyrill]

I can confirm this issue, we need to inspect and analyze the situation when Airbyte is stuck:
--> Try to pull the image with docker "outside" of Airbyte to see whether it's generally not pullable or whether Airbyte can't do it.

It's important to realize: Airbyte does use the same docker mechanism. The var/docker.sock is mounted into Airbyte so basically Airbyte only instructs docker to pull the image for it. It's irrelevant if Airbyte can connect to the web or not, it's the docker daemon running on the VM that performs the pull request.

Could it be a docker hub limiting issue? Would it help if we authorize with docker hub using an account instead of an anonymous pull?

[hwildwood]

Happening to me as well.

Possible related to: #39337

Edit: Restarting docker fixed the issue for me.

[wwfch-cyrill]

## tried to pull the image manually from the VM
docker pull airbyte/source-salesforce:2.5.15
2.5.15: Pulling from airbyte/source-salesforce
Digest: sha256:5296acee7fb4eb1f39f9d220cfa7192440af50d52aece7f2b7747a0798fec0c2
Status: Image is up to date for airbyte/source-salesforce:2.5.15
docker.io/airbyte/source-salesforce:2.5.15
## this confirms that the image is actually there and ready for airbyte - no need to pull at all

## now let's go into airby_worker and try to pull the image like airbyte does

x@y:~/services/airbyte$ docker exec -it airbyte-worker bash
[airbyte@5e84c22df9f6 app]$ docker pull airbyte/source-salesforce:2.5.15
error during connect: Post "http://docker-proxy:2375/v1.24/images/create?fromImage=airbyte%2Fsource-salesforce&tag=2.5.15": dial tcp: lookup docker-proxy on 127.0.0.11:53: server misbehaving

## ah, docker-proxy not there?
[airbyte@5e84c22df9f6 app]$ exit

x@y:~/services/airbyte$ docker compose ls
NAME                    STATUS              CONFIG FILES
dbt-salesforce          running(1)          /home/adminuser/services/dbt-salesforce/docker-compose.yml
airbyte             running(9)          /home/adminuser/services/airbyte/docker-compose.yaml,/home/adminuser/services/airbyte/docker-compose.override.yaml
dagster_workspace   running(5)          /home/adminuser/services/dagster_workspace/docker-compose.yml
metabase            running(2)          /home/adminuser/services/metabase/docker-compose.yml

To fix this issue, add this to a docker-compose.override.yaml or edit the upstream airby docker-compose.yaml file.

  docker-proxy:
  # i also noticed that the container_name is not set by airbyte like for the other services
    container_name: docker-proxy
# this should fix the issue
    restart: unless-stopped

I created: airbytehq/airbyte-platform#334

Bonus information:
If you have a regular cronjob to cleanup docker that might remove the airbyte images, use this to filter them from being deleted, this way the probability of the pull issue should be reduced:

docker image prune --force --all --filter="label!=io.airbyte.name" --filter until=92h

[davinchia]

Hi all, first off, thank you for investigating and proposing solutions to this bug.

Docker compose is going to be deprecated in a month so we aren't going to check in any fixes to the docker set up. I'd encourage you to try out our new abctl tool linked in the same discussion post.

With that in mind, I'm going to close this issue. Sorry for the inconvenience and thank you for your patience!