Unable to build / deploy on ARM

[zwan2]

 - deployment/adservice: container server terminated with exit code 1
    - pod/adservice-5b65c5ff6-zzfkl: container server terminated with exit code 1
      > [adservice-5b65c5ff6-zzfkl server] Error occurred during initialization of VM
      > [adservice-5b65c5ff6-zzfkl server] Could not find agent library /opt/cprof/profiler_java_agent.so in absolute path, with error: /opt/cprof/profiler_java_agent.so: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)
 - deployment/adservice failed. Error: container server terminated with exit code 1.

[xtineskim]

Hello! Thanks for bringing this up!!
At first glance, it seems like it might be a minikube issue,
could you specify your environment (os, minikube version, anything else that may be relevant)?
Thanks!

[zwan2]

• m1 mac Big Sur 11.6x
• minikube:latest using docker

It may be resolved when x86 condition

[jxlwqq]

Does not support deployment it under arm64 architecture, such as Apple chip's Mac, Raspberry Pi. @zwan2

Are there any plans to support arm64? @ckim328

[NimJay]

tl;dr - We will not be (consciously) supporting the deployment of our microservice sample images on ARM anytime soon. I say "consciously" because you might still be able to build the Docker images locally (and target ARM) and get them to deploy on ARM machines.

---

Hi @zwan2 and @jxlwqq!
Thank you for raising this issue! 😁

We've actually been talking about ARM support a lot recently. This issue is becoming a growing concern — especially given the popularity of the M1 chip.

Currently, the Docker images (for each microservice) that we host (e.g., gcr.io/google-samples/microservices-demo/checkoutservice:v0.3.1) use AMD64 architecture.

> docker image inspect gcr.io/google-samples/microservices-demo/checkoutservice:v0.3.1 | grep Arch
"Architecture": "amd64",

And, unfortunately, we do not intend to create sample images that use ARM architecture anytime soon — mainly because Google Kubernetes Engine does not currently support ARM-based image deployments.

@zwan2
In your Kubernetes deployments, are you using the images hosted at gcr.io/google-samples/microservices-demo/...? If so, have you tried building the images locally (and targeting ARM) and using those locally-built Docker images in minikube? I can't guarantee that this works (because I am not sure if the base images used in each of microservices' Dockerfiles support ARM). But it's worth trying. :)

[bourgeoisor]

Not fully supported yet, but note that you can alternatively use docker buildx from your ARM machine to build amd64 images: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

We're currently investigating the use of ARM chips by developers to build our sample applications.

[xtineskim]

cooling down out of our oslo, still investigations being done for the use of ARM

[NimJay]

@zwan2, @jxlwqq

GKE (Google Kubernetes Engine) now supports the ability to run containerized workloads using the Arm architecture. :)
These resources might be of interest to you:

1. YouTube - Run your Arm workloads on GKE!
   
2. Google Cloud Doc - Arm workloads on GKE
3. Google Cloud Blog - Run your Arm workloads on Google Kubernetes Engine with Tau T2A VMs
   

I have not tested your specific user journey myself (i.e., I have not run Arm-based container images on GKE, built on Mac's M1 chips). But I hope this info helps.

[minherz]

@zwan2 , @jxlwqq would you mind to try the tutorial for deploying Online Boutique sample application described in the official documentation?
Please, let us know if it does not work. Deploying the application does not require building containers.

[smazzone]

just checking if there are plans to build and publish arm64 images in the default repo for people who are willing to run the demo locally (i.e. M1 and docker desktop or similar). Thanks again.

[mathieu-benoit]

One more evidence we need to support this: #810

[mathieu-benoit]

Another evidence here: #832

[chenhy97]

need the ARM architecture supports!

[GregCKrause]

Not fully supported yet, but note that you can alternatively use docker buildx from your ARM machine to build amd64 images: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images

We're currently investigating the use of ARM chips by developers to build our sample applications.

Following up on @bourgeoisor's response.

To use this with skaffold, set build.local.useBuildkit to true in skaffold.yaml:

build:
  ...
  local:
    useBuildkit: true

[smazzone]

@GregCKrause I have been running the demo on arm64 for a couple of weeks. I have also used buildx to build images on both architectures. But, changing skaffold.yaml was not enough. I had to slightly re-adapt Dockerfile as health probes are hardcoded for amd64 architecture see example here

RUN GRPC_HEALTH_PROBE_VERSION=v0.4.14 && \ wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \ chmod +x /bin/grpc_health_probe

I hope this helps. Stefano

[minherz]

I will keep this issue open for now and mark it as a "good first issue" for someone who wants to submit a kustomize or Helm configuration that will build and deploy on ARM platform.

[NimJay]

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023.

More specifically:

• A user should be able to run Online Boutique in development mode (skaffold dev), on an ARM-based device.
• A user should be able to deploy Online Boutique, as is, on an ARM-based Kubernetes cluster. In this sentence, “as is” means that the user does not have to build Online Boutique’s container images themselves. Official ARM-supporting container images should be published by Online Boutique.

We hope to achieve this by the end of this year (2023).

Thank you, @smazzone and @GregCKrause for the tips you've shared — those will be helpful.

[odidev]

I have made changes in the Dockerfiles to build the images for arm64 platform and now skaffold is running successfully and pods are in running state.

I have used my docker images that I built for arm64 in kubernetes-manifests.yaml and deployed on minikube cluster. Now all the pods are running successfully but when applying the same kubernetes-mainfests.yaml file in GKE cluster the pods are in pending state for arm64.

I have also built the images using buildx and the pods are running for amd64 on GKE cluster but for arm64 it is still in pending state.
Can you please share your feedback regarding the same?

[NimJay]

Heads-up:

A comment above mentioned:

I had to slightly re-adapt Dockerfile as health probes are hardcoded for amd64 architecture see example here

This additional step (of swapping the grpc-health-probe binaries deployed next to Online Boutique's gRPC microservices) is no longer necessary, since the merge of #1837. In other word, the grpc-health-probe binary is no longer used.

[mtardy]

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023.
[...]
We hope to achieve this by the end of this year (2023).

Hey, we bumped into this, wanting to use the demo as part of our testing pipeline that runs on arm64 cilium/tetragon#2345. Do you still want to publish arm64 images anytime soon? Thanks!

[bourgeoisor]

Hi @mtardy! It's something we wanted to do last year but prioritization went a different direction.

cc: @NimJay might be able to give some more details / if there's a plan for this year.

But yeah this is still something we'd like to do.

[willfindlay]

Are you looking for any help on this or is it more of a case of "we could do this but aren't sure we're in place to maintain another set of images right now"?

[maitrungduc1410]

building cartservice on Apple M2 hang at restore

Dockerfile

FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:8.0.302-noble@sha256:bd836d1c4a19860ee61d1202b82561f0c750edb7a635443cb001042b71d79569 as builder
WORKDIR /app
COPY cartservice.csproj .
RUN dotnet restore cartservice.csproj \
    -r linux-x64
COPY . .
RUN dotnet publish cartservice.csproj \
    -p:PublishSingleFile=true \
    -r linux-x64 \
    --self-contained true \
    -p:PublishTrimmed=true \
    -p:TrimMode=full \
    -c release \
    -o /cartservice

# https://mcr.microsoft.com/product/dotnet/runtime-deps
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/runtime-deps:8.0.6-noble-chiseled@sha256:55d6e41f2e7687c597daa4fdca997b07beb3e23b6283729e19bb8ceb272def1a

WORKDIR /app
COPY --from=builder /cartservice .
EXPOSE 7070
ENV DOTNET_EnableDiagnostics=0 \
    ASPNETCORE_HTTP_PORTS=7070
USER 1000
ENTRYPOINT ["/app/cartservice"]

Logs:

[+] Building 552.2s (10/14)                    docker:desktop-linux
 => [internal] load build definition from Dockerfile           0.0s
 => => transferring dockerfile: 1.51kB                         0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/run  0.1s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk  0.1s
 => [internal] load .dockerignore                              0.0s
 => => transferring context: 92B                               0.0s
 => [builder 1/6] FROM mcr.microsoft.com/dotnet/sdk:8.0.302-n  0.0s
 => [stage-1 1/3] FROM mcr.microsoft.com/dotnet/runtime-deps:  0.0s
 => [internal] load build context                              0.0s
 => => transferring context: 569B                              0.0s
 => CACHED [stage-1 2/3] WORKDIR /app                          0.0s
 => CACHED [builder 2/6] WORKDIR /app                          0.0s
 => CACHED [builder 3/6] COPY cartservice.csproj .             0.0s
 => [builder 4/6] RUN dotnet restore cartservice.csproj      552.0s

I'm wondering, if this issue can't be resolved (it's there for years), why not convert this cartservice (and other services which have issue with Apple chip) to other languages that works for everyone? 🤔🤔🤔

[maitrungduc1410]

finally I got it working, we need to set ENV DOTNET_EnableWriteXorExecute=0

My Dockerfile

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# https://mcr.microsoft.com/product/dotnet/sdk
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:8.0 as builder
ENV DOTNET_EnableWriteXorExecute=0
WORKDIR /app
COPY cartservice.csproj .
RUN dotnet restore cartservice.csproj \
    -r linux-x64
COPY . .
RUN dotnet publish cartservice.csproj \
    -p:PublishSingleFile=true \
    -r linux-x64 \
    --self-contained true \
    -p:PublishTrimmed=true \
    -p:TrimMode=full \
    -c release \
    -o /cartservice

# https://mcr.microsoft.com/product/dotnet/runtime-deps
FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/runtime-deps:8.0

WORKDIR /app
COPY --from=builder /cartservice .
EXPOSE 7070
ENV DOTNET_EnableDiagnostics=0 \
    ASPNETCORE_HTTP_PORTS=7070
USER 1000
ENTRYPOINT ["/app/cartservice"]

[mtardy]

Update (Jan 2023)

Supporting ARM is definitely a priority for 2023.
[...]
We hope to achieve this by the end of this year (2023).

Hey, we bumped into this, wanting to use the demo as part of our testing pipeline that runs on arm64 cilium/tetragon#2345. Do you still want to publish arm64 images anytime soon? Thanks!

By the way, I hope it's not inappropriate to write about "alternatives", but for people looking for similar demos working both on amd64 and arm64, we ended up using https://github.com/open-telemetry/opentelemetry-demo.