m1 docker experimentation

[jrhizor]

re: #2017

I was just on Twitter waiting for water to boil for dinner when I saw this blog post: https://til.simonwillison.net/macos/running-docker-on-remote-m1#user-content-platform-linuxamd64-to-the-rescue

First of all, it mentions some ways to rent M1 Macs online easily today if we don't have access to one easily.

Specifying architecture like this has shown to work in some cases Automattic/jetpack#19237 although the support for it across docker-compose versions is a bit problematic: docker/cli#1050

The blog post used 3.1 so I'd be interested in trying it. I'm also not positive if it has to be specified for both the build and the main docker-compose or just the build?

To test this, we'd have to do the following on an M1 mac:

• build source and destination dev images
• build a normalization image and update DefaultNormalizationRunner
• rebuild core
• try running a sync with the custom images and the rebuilt core

[tuliren]

With this branch, the build went a little bit futher than the master, but failed with the following message:

> Task :airbyte-cli:airbyteDocker FAILED
#6 ERROR: executor failed running [/bin/sh -c go get -u github.com/cgardens/restish@e6f8ddda7fb5a3a989537272292c8321022cdb54]: exit code: 2
------
 > [2/2] RUN go get -u github.com/cgardens/restish@e6f8ddda7fb5a3a989537272292c8321022cdb54:
------
executor failed running [/bin/sh -c go get -u github.com/cgardens/restish@e6f8ddda7fb5a3a989537272292c8321022cdb54]: exit code: 2

Commit e6f8ddda7fb5a3a989537272292c8321022cdb54 only exists in the original repo (danielgtaylor/restish, not in cgardens/restish. Hmm, not sure why it works in the master branch.

Also neither github.com/cgardens/restish@e6f8ddda7fb5a3a989537272292c8321022cdb54 or github.com/danielgtaylor/restish@e6f8ddda7fb5a3a989537272292c8321022cdb54 works.

[tuliren]

Wow, changing the failed command to RUN go get -u github.com/danielgtaylor/restish@47de8a55eae6c0ab38e35119bef52ccf2837e3b3 (the latest commit on the original repo) fixed it.

CORE_ONLY=1 ./gradlew build now works 100% and the fan did not run for a second.

[tuliren]

Task :airbyte-integrations:connectors:source-mysql:test failed because there is no ARM image for mysql 8.0:

Caused by:
        org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageName=mysql:8.0, imagePullPolicy=DefaultPullPolicy())
            at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1286)
            at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:615)
            at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:320)
            ... 3 more

            Caused by:
            org.testcontainers.containers.ContainerFetchException: Failed to get Docker client for mysql:8.0
                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:99)
                at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
                at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
                at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
                at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1284)
                ... 5 more

                Caused by:
                com.github.dockerjava.api.exception.DockerClientException: Could not pull image: no matching manifest for linux/arm64/v8 in the manifest list entries

[jrhizor]

If we can specify the platform with testcontainers we should be able to pull things like docker-library/mysql#318 (comment)

[jrhizor]

testcontainers/testcontainers-java#3810 has information on what's necessary to get testcontainers running.

[jrhizor]

I think the biggest question is if Airbyte was able to run locally?

[tuliren]

The server can run into various random issues. Some of them are transient and will go away by restarting. But some are persistent, like this one:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.github.slugify.Slugify

[tuliren]

These connectors have various trouble under M1:

• source-mssql
• source-oracle
• source-file: stuck at installTestReqs

MySQL connectors can compile after switching to MariaDB:

• source-mysql: there are weird unit test failures
• destination-mysql: all tests pass

[tuliren]

Unfortunately, composeBuild for the the platform no longer works due to issue from this image: openjdk:14.0.2-slim. The linux/amd64 version does not work, and there is no linux/arm64 for it.

[jrhizor]

This is quite out of date now (we're well past the experimentation here).