Not able to run the image on aarch64

[chdanielmueller]

I think the issue has to do with the aarch64 on my Raspberry Pi.
Maybe the image is not built for this architecture?
If you could give me a "How-to build" I will be more than happy to try and see if that's the case.

The image "runs" on MacOS and returns an error that variables are missing which is totally understandable.

What did you do to encounter the bug?
Steps to reproduce the behavior:

1. Start the container on a Raspberry Pi
2. Run the Docker Container docker run quay.io/mongodb/mongodb-kubernetes-operator:0.4.0 or via Kubernetes
3. See error

What did you expect?
A running container or the same error as on MacOS when started without parameters.

What happened instead?
I received the following error and the container crashed.
standard_init_linux.go:211: exec user process caused "exec format error"

Operator Information

• Operator Version: 0.4.0
• MongoDB Image used: none

Kubernetes Cluster Information

• Distribution: K3S
• Version: v1.20.2+k3s1
• Image Registry location (quay, or an internal registry)

Additional context
Raspberry Pi 4B
Ubuntu 20.04.1
aarch64

[jamesbroadhead]

@chdanielmueller thanks for the request! We've got some upstream dependencies to wait for, but will do this once it's unblocked

[chdanielmueller]

@jamesbroadhead Do you have an estimate when this could be?

[chdanielmueller]

@jamesbroadhead Some months have passed. Any news on the upstream dependencies?

[jamesbroadhead]

Sadly not.

MongoDB Server support for aarch64 is still being worked on in https://jira.mongodb.org/browse/SERVER-50115, that is one of the things that is currently blocking Agent support for aarch64 support in general.

[idlezombie]

Sadly not.

MongoDB Server support for aarch64 is still being worked on in https://jira.mongodb.org/browse/SERVER-50115, that is one of the things that is currently blocking Agent support for aarch64 support in general.

The link seems to be talking about MacOS specifically and the MongoDB website indicates that MongoDB server supports arm64 and the docker hub images for MongoDB server specify arm64v8.

Am not a programmer but I was trying to deploy this on K3s on RPi 4 cluster and having the same error above as well. Am I missing something here?

[jtembry]

@idlezombie I was attempting to do the same thing last week, (i.e. install mongo-kubernetes-operator on k3s running on a RPI4). It appears you can run the mongodb image in k3s, but not the operator. I was hoping to set up the operator as I need persistent storage and haven't been able to configure it with just the mongodb image.

[jphgardner]

same problem here with a Raspberry Pi4 Cluster, Gonna have to update my cluster to multi-arch

[arshabh-inspectlogs]

@jamesbroadhead Looks like Spidermonkey ESR91 was released on 10th Aug. Any plans on when will mongodb agent for aarch64 will be available ?

[kuznero]

When mongodb agent will support arm64? I managed to build operator images to arm64 and it all works, except mongodb agent. Would be really nice to get this last bit in.

[jamesbroadhead]

There are now builds of the Agent for aarch64 available -
https://mciuploads.s3.amazonaws.com/mms-automation/mongodb-mms-build-agent/builds/automation-agent/hosted/mongodb-mms-automation-agent-11.12.0.7377-1.amzn2_aarch64.tar.gz

We would happily welcome PRs to add support for this.

Note that this is a newer version of the Agent than is currently used by the Operator. An x86 build of the Agent with the same version number is available here:
https://mciuploads.s3.amazonaws.com/mms-automation/mongodb-mms-build-agent/builds/automation-agent/hosted/mongodb-mms-automation-agent-11.12.0.7377-1.linux_x86_64.tar.gz

[jtembry]

@jamesbroadhead is there an aarch64 image available that I would add to the manager.yaml for rpi4 support?

for example:

• name: AGENT_IMAGE
  value: quay.io/mongodb/mongodb-agent:11.12.0.7388-1

[jamesbroadhead]

@jtembry Not at present.
We still would welcome PRs to add support, and could start publishing images if those PRs got merged.

Our feature request portal is here feedback.mongodb.com, and this request is tracked here:
https://feedback.mongodb.com/forums/924355-ops-tools/suggestions/44503347-arm64-support-for-kubernetes-operator

If this item receives votes and comments, it is more likely to have traction.
Thanks for your interest in the Operator!

[HazyFish]

This is very important. It's not only for Raspberry Pis. I have a ARM64 K8s cluster on AWS and many of our team members use ARM64 Mac for local dev as well. Please support this soon!

[Type1J]

I can't run the k8s operator on my Pi4 cluster, and I don't have another option with x86_64. Please add support for aarch64.

[darktempla]

Support for ARM64 in regards to the operator would be appreciated. ARM64 is going to be the way companies start to head for Kubernetes clusters in the cloud as at scale you can save big $$$ so strategically it would be a good move for MongoDB.

[mohsinonxrm]

There are now builds of the Agent for aarch64 available - https://mciuploads.s3.amazonaws.com/mms-automation/mongodb-mms-build-agent/builds/automation-agent/hosted/mongodb-mms-automation-agent-11.12.0.7377-1.amzn2_aarch64.tar.gz

We would happily welcome PRs to add support for this.

Note that this is a newer version of the Agent than is currently used by the Operator. An x86 build of the Agent with the same version number is available here: https://mciuploads.s3.amazonaws.com/mms-automation/mongodb-mms-build-agent/builds/automation-agent/hosted/mongodb-mms-automation-agent-11.12.0.7377-1.linux_x86_64.tar.gz

@jamesbroadhead , I was able to successfully build ARM64 image using the above mentioned mms automation with the version 12.0.10.7591-1 being used with v0.7.6. I also used the correct tools from the mongodb website targeting ARM64 Ubuntu.

However, when it tries to start the pod, the readiness probe fails with the message:
Cluster config did not pass validation for pre-expansion semantics : MongoDB version 4.2.6 for process = example-mongodb-0 was not found in the list of available versions : [ 4.2.6 : &{linux [] amd64 rhel false},&{linux [] amd64 ubuntu false} ]

Looking at the automation builder, looks like the dummy config is missing linux and arm64 combination.

However, I don't believe I can just update and build the agent again as the binaries are pre-built and being referenced in the docker file.

Please review:

mongodb-kubernetes-operator/pkg/automationconfig/automation_config_builder.go

Line 388 in 7ee3ad2

func buildDummyMongoDbVersionConfig(version string) MongoDbVersionConfig {

Thoughts? any help would be much appreciated.

Referencing:
#925

[mohsinonxrm]

Since I'm running a multi-arch cluster, here are the images I've built targeting both amd64 and arm64.

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-agent
mohsinonxrm/mongodb-agent:12.0.10.7591-1

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-kubernetes-operator
mohsinonxrm/mongodb-kubernetes-operator:0.7.6

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook
mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.5

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-kubernetes-readiness
mohsinonxrm/mongodb-kubernetes-readiness:1.0.11

These are built from v0.7.6 tag, however I don't recommend using these in any production workloads. As is - no support.
No changes were made in the source for amd64 build
only dockerfiles and release yaml was updated to target arm64

images were built natively on AMD64 machine and ARM64 machine

[dan-mckean]

Hi all, there's no time currently allotted to working on AArch64/ARM64 support before February at the earliest.

And beyond that point we don't have the order of roadmap items nailed down, so I can't advise an ETA.

The best think you can all do is vote or feedback on https://feedback.mongodb.com/forums/924355-ops-tools/suggestions/44503347-arm64-support-for-kubernetes-operator - doing so helps document the demand for this and increase it's priority!

[mohsinonxrm]

@dan-mckean, thanks for getting back. Could you please just confirm my findings above? and if that is indeed causing the issue, could you perhaps please share or point me to the documentation of building the agent binaries? (Not the agent image, I've already built the image).

I understand the binaries are downloaded from:
https://mciuploads.s3.amazonaws.com/mms-automation/mongodb-mms-build-agent/builds/automation-agent/hosted/mongodb-mms-automation-agent-[version].amzn2_aarch64.tar.gz

But would need to know how I can build the above myself and use that in the docker builder file.

Thanks again for getting back.

[jamesbroadhead]

Hi @mohsinonxrm -

Yes, you're correct that the dummy config would have to contain an entry for amd64 - even though that config is not actually used to locate the mongod binaries (instead: the operator relies on the MongoD container to supply a running binary).

Here is an example of a real config, as used by the Enterprise Operator:
https://opsmanager.mongodb.com/static/version_manifest/6.0.json

Without having tested it, I expect that the change you would have to make would be to add something like this to that Dummy config generation function, then rebuild the Operator to supply this config (not the Agent)

Builds: []BuildConfig{
			{
				Platform:     "linux",
				Architecture: "amd64",
				Flavor:       "rhel",
				Modules:      []string{},
			},
			{
				Platform:     "linux",
				Architecture: "amd64",
				Flavor:       "ubuntu",
				Modules:      []string{},
			},
            {
				Platform:     "linux",
				Architecture: "arm64",
				Flavor:       "rhel",
				Modules:      []string{},
			},
            {
				Platform:     "linux",
				Architecture: "arm64",
				Flavor:       "ubuntu",
				Modules:      []string{},
			},
		},
...

The first version of the Agent to support arm64 was:
https://www.mongodb.com/docs/ops-manager/current/release-notes/mongodb-agent/#mongodb-12-0-8-7575
and it looks like you're using a build newer than that already, so no new release or build of the Agent should be required.

Hope this helps & wishing you success!

James

[mohsinonxrm]

@jamesbroadhead , thanks for getting back to me on this. I'll def. give this a try. No expert here but trying my best to get this working on my cluster for development/learning purposes.

[mohsinonxrm]

SUCCESS!!

Happy to report that the things are working for me in mixed architecture cluster. It is AKS if anyone is curious. Next for me is to try this on my local k3s RPI-4B (7 x arm64) and RPI-X (4 x amd64)

Now the details:
As I suspected and was also directed above to add arm64 in the dummy config - unfortunately that didn't work exactly.

I made the changes, created operator image etc. but I still got the same error as before.

Then I thought maybe it isn't using dpkg --print-architecture format and instead using uname -m, so I added aarch64 stuff in the dummy config, re-created operator image and deployed and that is what worked!

If anyone would like to give it a go here are the images again: BTW, this is as-is, no support and don't use in production.

Here's 0.7.6:
https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-agent
mohsinonxrm/mongodb-agent:12.0.10.7591-1

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-kubernetes-operator
mohsinonxrm/mongodb-kubernetes-operator:0.7.6

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook
mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.5

https://hub.docker.com/repository/docker/mohsinonxrm/mongodb-kubernetes-readiness
mohsinonxrm/mongodb-kubernetes-readiness:1.0.11

Thanks @jamesbroadhead for your help with this, really appreciate it.

[jamesbroadhead]

@mohsinonxrm congratulations!

Note that the released arm64 binaries of mongodb do not run on a raspberry pi, as they use some ARM instructions that are not supported by the Pi (this is for performance when they're running on more modern ARM chips).

As a result, you'll have to find a pi-compatible build of mongodb for your mongodb containers.

To make one yourself, you could adapt the instructions here:
https://www.mongodb.com/community/forums/t/add-mongodb-4-2-arm64-builds-for-raspberry-pi-os-64-bit-debian-buster/5046/5

[jamesbroadhead]

Update - I have been made aware of some unofficial builds here: https://github.com/themattman/mongodb-raspberrypi-binaries/releases/tag/r6.1.0-rc4-rpi-unofficial

[mohsinonxrm]

@jamesbroadhead , thanks for the heads up. Yes, I'm aware of Mongodb 5.0 and greater not being compatible on older x64 or armv8 v1

All good with 4.4 mongodb though.

Thanks again for all the help!

[jamesbroadhead]

@mohsinonxrm ok great!

Just to make sure it's clear - it's possible to compile both MDB 5.0/6.0 and above to work on a Pi

[M4urici0GM]

Any updates on that? using OCI free tier to test development, and it'd be great
For now i'll try @mohsinonxrm images

Really appreciate the effort, bud!

[uwburn]

@mohsinonxrm, thanks for the images! I'm testing this right now on K3S on a cloud box.

Looking forward to have official support for it.

[kennedyoliveira]

@mohsinonxrm could you share how you built the images so we can update them to the latest versions?

Thank you!

[devgig]

Any updates on ARM64 support would be great.

[drivebyer]

+1

[Codestech1]

We are still waiting

[arthursimas1]

+1
this feature needs more traction from the community :(

[Mortega5]

You can compile the mongodb-kubernetes-operator image for arm64 (aarch64) by adding to the buildDummyMongoDbVersionConfig function of the automation_config_builder.go file this architecture:

[...]
  versionConfig := MongoDbVersionConfig{
    Name: version,
    Builds: []BuildConfig{
      {
        Platform:     "linux",
        Architecture: "amd64",
        Flavor:       "rhel",
        Modules:      []string{},
      },
      {
        Platform:     "linux",
        Architecture: "aarch64",
        Flavor:       "rhel",
        Modules:      []string{},
      },
      {
        Platform:     "linux",
        Architecture: "amd64",
        Flavor:       "ubuntu",
        Modules:      []string{},
      },
      {
        Platform:     "linux",
        Architecture: "aarch64",
        Flavor:       "ubuntu",
        Modules:      []string{},
      },
    },
  }
[...]

You can then compile the image using the following Dockerfile:

FROM golang:1.19.7-bullseye AS builder

RUN apt-get update && apt-get install -y unzip
ARG VERSION=0.8.1
RUN mkdir /workspace
WORKDIR /workspace

RUN wget https://github.com/mongodb/mongodb-kubernetes-operator/archive/refs/tags/v${VERSION}.zip
RUN unzip v${VERSION}.zip && rm v${VERSION}.zip && mv mongodb-kubernetes-operator-${VERSION} mongodb-kubernetes-operator
WORKDIR /workspace/mongodb-kubernetes-operator
# Override to support aarch64
COPY ./automation_config_builder.go ./pkg/automationconfig/

RUN go mod download
RUN CGO_ENABLED=0 go build -a -o manager cmd/manager/main.go


FROM registry.access.redhat.com/ubi8/ubi-minimal:latest

ENV OPERATOR=manager \
    USER_UID=2000 \
    USER_NAME=mongodb-kubernetes-operator

WORKDIR /

COPY --from=builder /workspace/mongodb-kubernetes-operator/manager .
COPY --from=builder /workspace/mongodb-kubernetes-operator/build/bin /usr/local/bin

RUN  /usr/local/bin/user_setup

USER ${USER_UID}

ENTRYPOINT ["/usr/local/bin/entrypoint"]

To compile for multiple architectures you can use docker buildx:

docker buildx build . --t <your_private_repository>/mongodb-kubernetes-operator:0.8.1 --platform linux/amd64,linux/arm64 --push

The automation_config_builder.go file must be in the same folder as the Dockerfile. Keep in mind that this file can change between releases, so make sure the version of the automation_config_builder.go file matches the version used in the Dockerfile (ARG VERSION=0.8.1).

[mohsinonxrm]

so... I've updated the images with the latest release. BTW, this is as-is, no support and don't use in production.

Here's v0.8.1 multi-arch (AMD64 and ARM64): ARM64 images are built on native ARM64 machine and AMD64 on AMD64 machine. Docker manifest and uploaded to Docker Hub.

https://hub.docker.com/r/mohsinonxrm/mongodb-agent/tags
mohsinonxrm/mongodb-agent:12.0.24.7719-1

https://hub.docker.com/r/mohsinonxrm/mongodb-kubernetes-operator/tags
mohsinonxrm/mongodb-kubernetes-operator:0.8.1

https://hub.docker.com/r/mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook/tags
mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7

https://hub.docker.com/r/mohsinonxrm/mongodb-kubernetes-readiness/tags
mohsinonxrm/mongodb-kubernetes-readiness:1.0.15

I'm looking into updating the release pipeline and contributing to this repo/PR. The problem isn't just building the images - as I've already figured that out last year - it is making sure e2e tests are done and everything is validated on ARM64 like it is currently done on AMD64 arch.

Anyway, hope you guys find this helpful!

[PhoenixMage]

so... I've updated the images with the latest release. BTW, this is as-is, no support and don't use in production.

Here's v0.8.1 multi-arch (AMD64 and ARM64): ARM64 images are built on native ARM64 machine and AMD64 on AMD64 machine. Docker manifest and uploaded to Docker Hub.

https://hub.docker.com/r/mohsinonxrm/mongodb-agent/tags mohsinonxrm/mongodb-agent:12.0.24.7719-1

https://hub.docker.com/r/mohsinonxrm/mongodb-kubernetes-operator/tags mohsinonxrm/mongodb-kubernetes-operator:0.8.1

https://hub.docker.com/r/mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook/tags mohsinonxrm/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7

https://hub.docker.com/r/mohsinonxrm/mongodb-kubernetes-readiness/tags mohsinonxrm/mongodb-kubernetes-readiness:1.0.15

I'm looking into updating the release pipeline and contributing to this repo/PR. The problem isn't just building the images - as I've already figured that out last year - it is making sure e2e tests are done and everything is validated on ARM64 like it is currently done on AMD64 arch.

Anyway, hope you guys find this helpful!

@mohsinonxrm Very much appreciate your work here.

Wasn't sure where to create an issue as your mongodb-kubernetes-operator fork doesn't have them enabled.

I found when a mixed cluster (arm64/amd64) runs the operator pod on amd64 that the cluster-config.json/mender-mongodb-config secret doesnt contain entries for aarch64 under mongoDbVersions

[mohsinonxrm]

@PhoenixMage , it's been a while since I last worked/used these.

I'm wondering, are you using the operator image from quay.io or from the ones I put together on docker hub?

BTW, best would now be to wait for the official release once the following PR is merged: #1361

[PhoenixMage]

@mohsinonxrm , the images I am using are from docker hub.

In the end I just used a node affinity to keep the operator on an arm board and everything is fine.

Keen for the merge :)

[Julien-Ben]

Hello,
We thank you for your feedback and your patience.
The PR was merged and the operator now supports the ARM64 architecture. We will do a new release soon with this new feature.

[vinnytwice]

@mohsinonxrm Thank you so much for the images, you saved us quite a bit of headaches.
@Julien-Ben I see that basically it's just the agent that doesn't have an arm64 image. Any Idea of when will it be available?

[mohsinonxrm]

@vinnytwice , I'm glad you found it helpful.

[abasu0713]

operator supports arm64 but not mongodb-agent. I am still getting errors. How is this still an issue?
Correction. It works with official images on Orange Pi. so I am sure it will work on Raspberry Pis. as well.
[UPDATE] the Orangepi that I use has a custom kernel with CONFIG_ARM64_VA_BITS_48=y which is usually not the case in RPIs and OPIs. Just an FYI