The check connection failed because of an internal error in the scheduler used by airbyte.

[spunzon]

Platform Version

0.57.1

What step the error happened?

During the Sync

Revelant information

After updating to version 0.57.1 the sync of all connections fail. With version 0.51 it was working without problem. There is no log:

I add info about SO and Docker:

cat /etc/os-release:
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian

uname -r:
6.1.0-18-cloud-amd64

docker --version:
Docker version 26.0.0, build 2ae903e

docker compose version:
Docker Compose version v2.25.0

Relevant log output

Failure reason: The check connection failed because of an internal error
The check connection failed because of an internal error in the scheduler used by airbyte.

[ysguy]

!! See workaround / temporary fix below !!

Can confirm happening after upgrading from 0.54 -> 0.57.1 today. All my scheduled jobs are timing out with the same error. Changing jobs from scheduled to manual does not alter the outcome. Testing via the builder works.

Also seeing this in the logs:
airbyte-worker | ERROR StatusConsoleListener FileManager (/tmp/workspace/287863/0/logs.log) java.io.IOException: Could not create directory /tmp/workspace/287863/0
airbyte-worker | java.io.IOException: Could not create directory /tmp/workspace/287863/0
airbyte-worker | at org.apache.logging.log4j.core.util.FileUtils.mkdir(FileUtils.java:129)
airbyte-worker | at org.apache.logging.log4j.core.util.FileUtils.makeParentDirs(FileUtils.java:143)
airbyte-worker | at org.apache.logging.log4j.core.appender.FileManager$FileManagerFactory.createManager(FileManager.java:504)
airbyte-worker | at org.apache.logging.log4j.core.appender.FileManager$FileManagerFactory.createManager(FileManager.java:488)

Am going to try and shell into the worker and override permissions on /tmp to see if that helps.

it appears the workspace folder of /tmp is owned by root

[airbyte@0f98a20190ee tmp]$ ll
total 1140
drwxr-xr-x 2 airbyte airbyte 4096 Mar 19 01:26 airbyte_local
drwxr-xr-x 2 airbyte airbyte 4096 Apr 3 14:02 hsperfdata_airbyte
drwx------ 2 airbyte airbyte 4096 Apr 3 14:03 scripts5504285348746223521
drwx------ 2 airbyte airbyte 4096 Apr 3 13:57 scripts9539611819066118485
drwxr-xr-x 46022 root root 1146880 Apr 3 12:54 workspace

Confirmed, this resolved this issue.
Running this from the host:

sudo docker exec -u 0 <airbyte_worker docker id> chown -R airbyte:airbyte /tmp/workspace
will allow the jobs to run.

[perangel]

Thank you for reporting this issue @spunzon

This appears to be caused by a recent change that modifies the user the containers run as (no longer running as root). The solution is to recreate the airbyte_workspace docker volume (e.g. delete it and let it get recreated by docker compose) when you start Airbyte up again.

[spunzon]

@perangel thank you very much, it solved the problem

[techxorcist]

To avoid recreating the volume, this can also be resolved by changing the ownership of /tmp as outlined here.