Enable M1 development with docker

[fuziontech]

According to the onboarding docs new hires are to buy an M1 mac as their laptop (if they go the apple route). Currently our stack doesn't fully support M1 and this has slowed down the onboarding of new hires. We should sort out the story for dev on M1.

Ideal solution:

• Docker runs entire stack locally

Alternate solution:

• Use dedicated or shared dev box to develop for clickhouse, kafka, and zookeeper.

[buwilliams]

I've been looking into this since it directly impacts my current work. I think it's possible and what's more I think I can get it working for both ARM64 and AMD64 architectures. I'll report back in a couple of days once I know more.

Success criteria:

• Runs on Macbook M1 with 16GB RAM since that is our companies preferred devbox
• Works for both arm64 and amd64 architectures
• Runs Clickhouse and Postgres

[buwilliams]

Updates:

• You need to run be running Docker 3.3.0+ which you can install via brew install homebrew/cask/docker on mac
• PostHog + Postgres is working on M1 / ARM64
• It does not run bin/start-frontend since it's too slow, you can run is manually bin/start-frontend

Todo:

• Provide an even better CLI experience for launching both docker and start-frontend for getting up and running with PostHog a trivial activity
• Persist data across reboots. Postgres and Redis are not using docker Volumes so the data is being reset on each reboot which is not ideal for localdev.
• Get Clickhouse working for M1 and eventually the whole of PostHog EE

[timgl]

Looks like Docker for m1 has just gone general availability so clickhouse should be fine? https://twitter.com/Docker/status/1382731403389984769

[buwilliams]

Docker postgres support for M1 is now working. While there is ongoing work for making local dev more streamlined this issue resolved. Thanks everyone!

[buwilliams]

Re-opened since Clickhouse doesn't work for M1. I'm very close to full support since I've got Clickhouse building and running via Docker.

Several things to note:

• Compiling Clickhouse from source is no joke since the docs are insufficient. I do not recommend it.
• Clickhouse installation documentation is misleading since macOS is not the binary you need for Docker on M1 since Docker uses a Linux VM to run Docker therefore you need the AArch64 binary.
• We need build the docker images manually so what you get for free with yandex/clickhouse-server
  is unavailable and needs to be configured manually.
• I'm running into a configuration issue with Kafka: kafka.errors.UnrecognizedBrokerVersion: UnrecognizedBrokerVersion that I haven't been able to solve. You'll see this error when running, PYTHONUNBUFFERED=1 DJANGO_SETTINGS_MODULE=posthog.settings DEBUG=1 PRIMARY_DB=clickhouse KAFKA_URL=kafka://localhost python manage.py runserver 0.0.0.0:8000

I believe that full support will be available as soon as I resolve the kafka configuration issue. Anyone interested in helping?

You can track the Docker Clickhouse M1 support progress in the PR #4145

Edit:
The kafka issue was caused because the standard version doesn't work for arm64.

[buwilliams]

Update:

• The standard wurstmeister/kafka image from docker hub doesn't work. Had to fork and rebuild fuziontech/kafka-m1:latest (thanks James)
• Zookeeper is a similar story, had to arm64v8/zookeeper but it's buggy
• Cannot connect to Clickhouse via JDBC for some reason java.net.SocketTimeoutException: Read timed out. which means JetBrains cannot connect.

Suggestion:

• For developers running M1, it may make more sense for Posthog EE to have a shared Clickhouse environment that we could point our dev environments to. It would only need to run clickhouse, kafka, and zookeeper. Support for arm64 is basic at best so it seems like we are fighting against the tide.
• This is what I'll be doing tomorrow morning.

[fuziontech]

The main issue currently is that Kafka and Zookeeper start, but are unstable and quickly crash after startup. I spent some time on this today and am not sure how much effort it would take to shore up the images to make them stable enough to work as a Dev environment.

I do think we could setup one large node with users for everyone. This would make sharing resources easy and get everyone going sooner than later. Basically put running the docked-compose stack on hold.

Another alternative is setting up Kafka, ZK, and potentially clickhouse locally and natively on the m1 with no docker. This would be hard to maintain but could be a nice alternative to running everything on a remote Dev box.

[mariusandra]

Before I go down the rabbit hole myself, I'd like to know if anyone tried running the combo of clickhouse, zookeeper and kafka as Intel x86 containers (linux/amd64)?

Not all images are available for ARM64 architecture. You can add --platform linux/amd64 to run an Intel image under emulation. In particular, the mysql image is not available for ARM64. You can work around this issue by using a mariadb image.

They do add that:

However, attempts to run Intel-based containers on Apple Silicon machines can crash as QEMU sometimes fails to run the container. Filesystem change notification APIs (e.g. inotify) do not work under QEMU emulation, see docker/for-mac#5321

So it's possible things don't work.

[buwilliams]

@mariusandra This was the first approach I took which didn't work. I admit I know much more now than I did then so perhaps I missed an important step. It's worth a shot but I wouldn't spend too much time with it.

[fuziontech]

I did try running x86 containers and the JVM couldn't handle it.

I also tried using ARM images but they were just as unstable. The most unstable container was Kafka as it seemed to crash the JVM randomly and then would hang.

We can definitely spend more time on this later, but for now I would say we put this on the back burner until more of the build infrastructure has been setup to support these devices.