Build on MacOS M1 (Apple Silicon)

[h7kanna]

What

Experimental Build on macOS M1 (Apple silicon)

How

Upgrade JDK version to 17 and parameterize through an environment variable
Upgrade Node version to 16 and parameterize through an environment variable
Parameterize the docker build platform

Example

export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export JDK_VERSION=17
export NODE_VERSION=16.11.1
SUB_BUILD=PLATFORM ./gradlew build -x test

Recommended reading order

1. build.gradle
2. airbyte-workers/Dockerfile

Pre-merge Checklist

This is an experiment mainly to speed my dev cycle by solving #2017.
Currently, I have to build on a separate Linux machine which is a frustratingly very slow dev loop.
Changes include fixes for the below issues.

• com.github.johnrengelman.shadow Gradle plugin should be upgraded to 7.1.0 Also watch Plugin is not completely compatible with gradle 7.2 johnrengelman/shadow#713
• There is a weird issue with Gradle node plugin. Currently it does not allow customizing the node version platform. Only way to build on Mac M1 is to use Node 16.x which supports darwin/arm64 Support for Apple Silicons architecture aarch64/arm64 node-gradle/gradle-node-plugin#154
• testcontainers should be upgraded to 1.15.3 or better 1.16.0 (which has good support for arm64). Fails on m1 mac docker preview, java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native testcontainers/testcontainers-java#3610
• JDK 17 which has native darwin/arm64 support. https://www.oracle.com/java/technologies/downloads/#jdk17-mac
• Set Gradle properties for Spotless plugin google-java-format (and removeUnusedImports) broken on JDK 16+ (has workaround) diffplug/spotless#834

TODO

• Make Airbyte worker pull connector docker images specific to platform (Currently docker pull prints a warning) https://docs.docker.com/engine/reference/commandline/pull/#options
• Make gradle pick correct JDK/JVM toolchain https://docs.gradle.org/current/userguide/toolchains.html#sec:vendors
• Temporal can be built for linux/arm64. In fact I added support for arm64 release binaries a while ago. But somehow docker images do not have linux/arm64
  https://github.com/temporalio/temporal/tree/master/docker/base-images
  https://github.com/temporalio/temporal/releases/tag/v1.12.3

I am able to successfully run Airbyte on MacOS M1, created sample Connection, Destination, and Source.

[CLAassistant]

All committers have signed the CLA.

[marcosmarxm]

Thanks @h7kanna I asked @jrhizor and @tuliren to review your contribution! They already discuss about developing in M1 or upgrading to Java 17.

[tuliren]

@h7kanna, huge thanks to you for working on this issue.

I tested it on a 16 GB M1 Mac Pro. The compilation works perfectly. I am able to launch the server and run a sync as well.

Some problems I have seen are:

1. Unit tests fail because the current Mockito does not support Java 17.
2. There are temporal problems from time to time (see an example below). Restarting the server can fix them. I suppose this will be fixed once temporal can pull the correct image?

airbyte-temporal | {"level":"error","ts":"2021-10-19T16:31:04.969Z","msg":"Operation failed with internal error.","service":"matching","error":"UpdateTaskQueue failed. Failed to start transaction. Error: context deadline exceeded","metric-scope":38,"logging-call-at":"persistenceMetricClients.go:804","stacktrace":"go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).updateErrorMetric\n\t/temporal/common/persistence/persistenceMetricClients.go:804\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).UpdateTaskQueue\n\t/temporal/common/persistence/persistenceMetricClients.go:787\ngo.temporal.io/server/service/matching.(*taskQueueDB).UpdateState\n\t/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskReader).persistAckLevel\n\t/temporal/service/matching/taskReader.go:272\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:160"}`

@h7kanna, by the way, do you plan to work on the TODOs? It's ok if you don't. We just need to know what projects we need to prioritize for M1 support.

I also confirmed that this PR does not interfere with Intel mac. Build, unit test, and server can all run without any problem. So I think it is worth merging this PR as is (even if the unit test issue cannot be resolved soon), since it can unblock developers working on M1.

[h7kanna]

@tuliren My intention of this is to unblock myself to develop on M1 and not break anything for existing build.

for the TODO important one is temporal arm64 image. Better way is to engage with the Temporal team if they can include linux/arm64 images be pushed to Docker hub.

Do merge it.. Thanks.

[tuliren]

A small change has been made in the master. Our CI runner has an old version of docker-compose. So we added a M1 specific docker compose build file: docker-compose.build-m1.yaml, which will be used when the arch is arm64.