Support running Airbyte on Mac M1 Docker #2017
Comments
|
@jrhizor sorry, seems this is not related with this one...please ignore. |
|
Docker on M1 is now generally available https://docs.docker.com/docker-for-mac/apple-silicon/ |
|
Having the same issue on M1. Would be great to have a solution |
|
Tried running airbyte with Kind on my M1 chip macbook pro but it got stuck saying "cannot reach server. The server may still be starting up.". This is critical to me to stick with airbyte. Since more and more developers are switching to the M1 macs, I'd love to see the solution asap. |
|
M1 support reference: https://docs.docker.com/desktop/multi-arch/ |
|
Hey guys! We are deciding on our primary connector, I'm advocating on airbyte, but all of our engineers have an m1 at this moment, a possible solution to this would help a lot, do we have an eta? |
|
Hey @caiopedroso, thank you for letting us know. There is an increasing number of requests for making Airbyte work on M1. We are evaluating if we can prioritize this issue any time soon. Will provide an update this week (probably this Thursday). |
|
Just confirming that running Airbyte on my M1 is not working. Short version of my experience:
I am moving on to workarounds like using a deployed version and my Intel. Happy to help however possible to get this up and running as soon as possible. |
Update on this. We have allocated one week for this issue in August, and will likely solve it by the end of the month. |
|
Another user affected by this problem: https://airbytehq.slack.com/archives/C01MFR03D5W/p1629153029204200 |
|
That's me! (commenting for alerts when you fix it). |
|
Would love to see a fix for this too! |
|
Getting the same errors as @hbd when trying to use Airbyte for the first time. |
|
Having the same problem. |
|
+1 for this problem also. |
|
+1, same problem running on M1. I can get docker image to run and get into UI. problem arises when trying to add a connector/source, even after pre-loading the connector manually and restarting. |
|
+1 following |
|
+1, ran into the same problem today as @hbd. |
|
It seems we'd also need to upgrade dbt version that is used by airbyte to run on Mac M1: |
|
Any update on this? Eager to use Airbyte on my M1! Happy to help where I can.
|
|
Thanks to @h7kanna, Airbyte can now build and run on M1. Please use open jdk 17, and build the server as follows: export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export JDK_VERSION=17
export NODE_VERSION=16.11.1
SUB_BUILD=PLATFORM ./gradlew build -x testSee doc here for details and #7104 for next steps. Known remaining issues:
|
|
For Temporal:
|
|
I am able to build airbyte locally in M1 machine with a little bit more setting, just share what I have done here:
|
|
@jrhizor just an update on the Temporal side - v1.14.2 onwards should be fine again. only v1.12 to v1.14 were affected by this M1 issue. |
|
For those using macports cd airbyte
git fetch
sudo port install openjdk17-temurin openssl3
export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export ALPINE_IMAGE=arm64v8/alpine:3.14
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine
export JDK_HOME=${JAVA_HOME}
export JDK_VERSION=17
export JAVA_OPTS=-Djdk.lang.Process.launchMechanism=vfork
export LDFLAGS="-L/opt/local/lib -L/opt/local/libexec/openssl3/lib"
export CPPFLAGS="-I/opt/local/include -I/opt/local/libexec/openssl3/include"
docker compose up It now builds, but I still get the |
This seems to be causing a new problem. Gradle fails with |
|
At version 0.35.60-alpha I manage to get it up with: export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export ALPINE_IMAGE=arm64v8/alpine:3.14
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine
export JDK_VERSION=17
VERSION=0.35.60-alpha docker-compose uphowever that crashes: |
|
@evantahler Facing the same issue in Airbyte while trying to run a sample dataset using File connector on M1 Macbook Pro.Tried the workaround but still the same. Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 147, exit value: 1 |
|
I am trying to create simple source and destination Is there a way to fix this? |
|
Is there an ETA on this? |
|
For me AirByte builds on an M1 and I don’t get any errors however it doesn’t connect to any source or destination successfully |
|
In the meantime, we compiled and published Airbyte containers for Apple M1 chips on farosai org on DockerHub. E.g. It would be great to see the official images published as well though. |
|
HI all, we've started publishing official ARM images as of 0.38.2-alpha. A proper announcement will go out soon. This means it is now possible to run Airbyte on ARM laptops without any additional work. Connector support is still spotty. @evantahler and @sherifnada are working on publishing ARM connector images. In the mean time, one way around this is to build the relevant connector images locally (so it's an ARM architecture) and update the connector image tag on the UI. Airbyte will run off the local connector and thus use the right architecture. Thanks for the patience! |
|
With #13004, we now have the plumbing in place to start releasing connectors for multiple platforms (e.g. M1 Macs). Going forward, we will release connectors for multiple architectures! |
|
Looks like |
|
I am going to close this issue since the core Airbyte platform now supports M1. Note that though we've the default connector publishing now publishes M1 images, not all connectors have M1 images published yet. Our approach is to greedily publish M1-compatible connector images as connectors are updated. Please open a new issue for M1 specific connector image or any suspected M1-specific bugs. Thanks again for the patience everyone! |
|
@davinchia Is there a list of connectors that we can refer to, which have M1 images published?
|
|
@void666 we don't have a list at the moment, but if you are curious you can look on dockerhub and check for the
|
and others
Current status
Todos
source-mssqlsource-oraclesource-file: stuck atinstallTestReqssource-mysql&destination-mysqlsource-mysql.Relevant PRs
The text was updated successfully, but these errors were encountered: