[FB:] ARM64 and Podman support verified + with fixes to make it work

[HAHermsen]

[FB:] ARM64 and Podman support verified + with fixes to make it work

Context

Tested on a Mono Gateway DK (NXP LS1046A, aarch64) running Armbian Debian Trixie + k3s v1.35.4. Found three issues blocking ARM64 deployment, all with proposed fixes. These findings demonstrate that both aarch64 and Podman support are relatively straightforward to implement.

Environment:

• Hardware: Mono Gateway DK (NXP LS1046A, 4-core aarch64, 8GB ECC RAM)
• OS: Armbian 26.2.3 Debian Trixie
• Kernel: 6.12.49-current-ls1046a-ask
• Architecture: arm64
• Kubernetes: k3s v1.35.4 (containerd 2.2.3-k3s1)
• Container runtime (host): Podman 5.4.2 aliased as docker — no Docker daemon involved

Podman containers (WFM stack):

NAMES                             STATUS          IMAGE
symphony-api-container            Up 1 minute     ghcr.io/margo/margo-symphony-api:latest
redis                             Up 4 minutes    docker.io/library/redis:7-alpine
workload-fleet-management-client  Up 1 minute     localhost/workload-fleet-management-client:ci-test

k3s pods (Eclipse Symphony k8s control plane):

NAMESPACE   NAME                                                READY   STATUS
symphony    symphony-api-fcf9cdbb8-55t6t                        1/1     Running
symphony    symphony-cert-manager-77f8479489-ftzsg              1/1     Running
symphony    symphony-cert-manager-cainjector-758cc94c44-5t4w7   1/1     Running
symphony    symphony-cert-manager-webhook-6f4546df97-tclmc      1/1     Running
symphony    symphony-redis-575fc56cc9-vr8rg                     1/1     Running
symphony    symphony-zipkin-5c47b4569-jfmtx                     1/1     Running

Device agent onboarded and reporting capabilities:

INFO  Device onboarding successful         {"deviceClientId": "client-896fdc96df18bfa2-1778970065"}
INFO  Capabilities reported successfully   {"deviceClientId": "client-896fdc96df18bfa2-1778970065"}
INFO  Workload Fleet Management Client started successfully
        {"capabilitiesFile": "./config/capabilities.json",
         "hasDeviceSignature": true,
         "stateSeekingInterval": 15,
         "sbiUrl": "https://symphony-service:8082/v1alpha2/margo"}

Symphony API responding:

$ curl http://localhost:32371/v1alpha2/greetings
Hello from Symphony K8s control plane (S8C)

The full Margo PR1 sandbox stack — WFM server, device agent, and Eclipse Symphony k8s control plane — is running on ARM64 hardware after applying the workarounds described below.

---

Issue 1 — Margo WFM client: no ARM64 image published to ghcr.io

docker manifest inspect ghcr.io/margo/margo.org/workload-fleet-management-client:latest shows only amd64. No ARM64 variant is published.

The .goreleaser.yaml correctly lists linux/arm64 as a build target, and the Dockerfile builds cleanly on ARM64 (tested and confirmed). The issue is that the GitHub Actions workflow never builds and pushes the ARM64 image to ghcr.io.

Fix: Add linux/arm64 to the Docker build and push step in the CI workflow.

---

Issue 2 — Margo WFM client: Docker hardcoded as runtime dependency

The WFM client has a hard dependency on Docker in two places:

shared-lib/workloads/dockerCliClient.go:

dockerBinary, err := exec.LookPath("docker")
if err != nil {
    return nil, fmt.Errorf("docker binary not found in PATH: %w", err)
}

poc/device/agent/Dockerfile:

FROM docker:28.3.3-cli AS docker-source
...
COPY --from=docker-source /usr/local/bin/docker /usr/bin/docker
COPY --from=docker-source /usr/local/libexec/docker/cli-plugins/docker-compose /usr/local/libexec/docker/cli-plugins/docker-compose

This directly contradicts Margo's secure-by-design philosophy. Docker requires a persistent root daemon (dockerd) — a container escape on an OT edge node running EtherCAT/PROFINET gives an attacker root on the host. This is unacceptable for industrial deployments.

Podman is the correct default: rootless by default, daemonless, full Docker CLI compatibility (alias docker=podman or symlink), native pod support, OCI-native. The fix is trivial since exec.LookPath("docker") resolves whatever binary is named docker in PATH.

Proposed fix — dockerCliClient.go:

dockerBinary, err := exec.LookPath("podman")
if err != nil {
    dockerBinary, err = exec.LookPath("docker")
    if err != nil {
        return nil, fmt.Errorf("neither podman nor docker binary found in PATH")
    }
}

Proposed fix — Dockerfile:

FROM alpine:3.18
RUN apk add --no-cache ca-certificates tzdata podman podman-compose
RUN ln -s /usr/bin/podman /usr/bin/docker
RUN echo 'alias docker=podman' >> /etc/profile.d/docker-podman.sh
WORKDIR /
COPY --from=builder /app/device-agent .
COPY --from=builder /app/poc/device/agent/config ./config
VOLUME ["/config"]
ENTRYPOINT ["./device-agent"]
CMD ["-config", "config/config.yaml"]

Tested on Podman 5.4.2 with podman-compose 1.3.0 on aarch64. Both docker --version and docker compose version resolve correctly via symlink.

---

Issue 3 — Eclipse Symphony dependency: symphony-k8s:0.48.28 contains an amd64 binary in the ARM64 manifest

The Helm chart for Symphony (which the Margo sandbox depends on) pins symphony-k8s:0.48.28. This image has a multi-arch manifest that claims to include linux/arm64, but the actual binary inside the ARM64 layer is x86-64:

$ file manager
manager: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked...

This results in exec /manager: exec format error when the container starts on ARM64 — a silent wrong-binary-in-manifest bug.

The latest tag (built 2026-05-14) contains a correct ARM64 binary:

$ file manager
manager: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked...

Workaround used to get Symphony running on ARM64:

skopeo copy \
  --override-arch arm64 --override-os linux \
  docker://ghcr.io/eclipse-symphony/symphony-k8s:latest \
  docker-archive:/tmp/symphony-k8s-arm64.tar:ghcr.io/eclipse-symphony/symphony-k8s:0.48.28
k3s ctr -n k8s.io images import /tmp/symphony-k8s-arm64.tar

Fix: Update the sandbox Helm chart to pin a version of symphony-k8s that has a correct ARM64 build, or switch to latest. This should also be reported upstream to the Eclipse Symphony project.

---

Summary

Workarounds needed to run on Podman (undocumented)

1. Build WFM client image locally for ARM64 — no ARM64 image on ghcr.io
2. Tag as ci-test + set CI=true — bypasses the GHCR pull in the start script
3. Mount Podman socket in docker-compose.yaml: /var/run/podman/podman.sock:/var/run/docker.sock
4. Install aardvark-dns — Podman's DNS component not installed by default
5. Enable Podman socket — systemctl enable --now podman.socket
6. Add symphony-service to /etc/hosts — TLS cert hostname matching
7. Generate CA cert with CA:TRUE — the sandbox cert generator has a bug (CA:FALSE), use OpenSSL directly
8. Use margo-symphony-api container instead of Helm Symphony — Helm-deployed Symphony has no Margo onboarding endpoint
9. Run Redis separately — margo-symphony-api needs it but doesn't bundle it

#	Component	Issue	Severity
1	Margo WFM client	No ARM64 image published to ghcr.io	Blocker
2	Margo WFM client	Docker hardcoded 😞 — see related TWG discussion on replacing Docker with Podman for OT edge deployments	Major
3	Eclipse Symphony	symphony-k8s:0.48.28 ARM64 manifest contains amd64 binary	Blocker

All three issues were found on first contact with real ARM64 hardware. Fixes are proposed for all three.

[nilanjan-samajdar]

@HAHermsen ,
Taking issue 1 and 3 as bugs (see sub-issues #282 and #283 ).

Issue 2, we will put in the backlog to bring in proper support for Podman.

Overall, extremely pleased that you were able to take the Margo Sandbox out for a ride and make it work !!

[vireshnavalli]

Hi @HAHermsen
For #Issue 3 , First of all I want to clarify few things, let me know if I am not answering your questions

• There is a separate symphony repo for Margo specific changes to WFM ( it is a forked repo from eclipse symphony https://github.com/eclipse-symphony/symphony which doesn't have margo implemented changes )
• Currently sandbox creates only docker container for WFM (Symphony API) using Github Actions . For this only Dockerfile changes have been implemented
• We haven't created/modified helm chart for Margo version symphony, so currently sandbox support docker container for symphony.

[HAHermsen]

@vireshnavalli
Thanks for the clarification.