Add ARM support

[elmariofredo]

I understand that it's early and not priority just want to raise my hand that my raspberry pi 4 64bit cluster is forced to use prometheus-operator. So please if it's not that big of a issue let's make it happen 🙏

[f41gh7]

Hello,

Wow, pi cluster, it sounds interesting)

I guess, its very simple and i will add cross compilation into next release.

[elmariofredo]

That would be fantastic 😍 let me know if you will need any testing or other support. Thanks!

[f41gh7]

i've added cross compilation, you can test operator with docker image: victoriametrics/operator:cross-compilation-arm64.
I've tested it at aws arm, it was able to start.

Also, you have to have change values for it, apply patch:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: vm-operator
  name: vm-operator
spec:
  template:
    spec:
      containers:
      - name: manager
        env:
        - name: VM_VMALERTDEFAULT_VERSION
          value: v1.40.0-arm64
        - name: VM_VMAGENTDEFAULT_VERSION
          value: v1.40.0-arm64
        - name: VM_VMSINGLEDEFAULT_VERSION
          value: v1.40.0-arm64
        - name: VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION
          value: v1.40.0-cluster-arm64
        - name: VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VERSION
          value: v1.40.0-cluster-arm64
        - name: VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION
          value: v1.40.0-cluster-arm64
        - name: VM_VMALERTMANAGER_ALERTMANAGERVERSION
          value: v0.20.0-arm64
        - name: VM_VMALERTMANAGER_CONFIGRELOADERIMAGE
          value: jimmidyson/configmap-reload:v0.4.0-arm64
        - name: VM_VMALERTDEFAULT_CONFIGRELOADIMAGE
          value: jimmidyson/configmap-reload:v0.4.0-arm64
        - name: VM_VMAGENTDEFAULT_CONFIGRELOADIMAGE
          value: quay.io/coreos/prometheus-config-reloader:v0.41.1-arm64
 

Let me know, if something go wrong.

[elmariofredo]

Great! Operator is up and running I will test it over weekend and will let you know.

p.s. Have you considered multi arch docker image https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/ ? It's under experimental flag for building stage, but it works without any intervention for users. Then we don't need to do image tag patching.

[f41gh7]

Thanks, docker buildx looks interesting. Need to check, is it working with other container-runtime systems, like podman.

[elmariofredo]

For podman building it doesn't seems to work there is ongoing work under buildah containers/buildah#1590, but run should work just fine as it's part of OCI since 2017 https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#manifest-list-field-descriptions and e.g. alpine images are released this way for quite some time.

[elmariofredo]

I had to fix slightly patch file as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: vm-operator
  name: vm-operator
spec:
  template:
    spec:
      containers:
      - name: manager
        env:
        - name: VM_VMALERTDEFAULT_VERSION
          value: v1.40.0-arm64
        - name: VM_VMAGENTDEFAULT_VERSION
          value: v1.40.0-arm64
        - name: VM_VMSINGLEDEFAULT_VERSION
          value: v1.40.0-arm64
        - name: VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION
          value: v1.40.0-cluster-arm64
        - name: VM_VMCLUSTERDEFAULT_VMSTORAGEDEFAULT_VERSION
          value: v1.40.0-cluster-arm64
        - name: VM_VMCLUSTERDEFAULT_VMSELECTDEFAULT_VERSION
          value: v1.40.0-cluster-arm64
        - name: VM_VMALERTMANAGER_ALERTMANAGERVERSION
          value: v0.20.0
        - name: VM_VMALERTMANAGER_CONFIGRELOADERIMAGE
          value: jimmidyson/configmap-reload:v0.4.0
        - name: VM_VMALERTDEFAULT_CONFIGRELOADIMAGE
          value: jimmidyson/configmap-reload:v0.4.0
        - name: VM_VMAGENTDEFAULT_CONFIGRELOADIMAGE
          value: quay.io/coreos/prometheus-config-reloader:v0.41.1-arm64

as jimmidyson/configmap-reload and quay.io/prometheus/alertmanager are multiarch images 😉

I have also run into small issue with alertmanager, but that was simple typo in examples alertmanager config and I have created fixing MR here #79.

Let me know if you want to for now focus on other priorities than figuring multiarch images and I can add ARM patch file to docs examples so other ARM fanatics can start operator easily.

[f41gh7]

Thanks for testing. Docs and examples aren't so pretty yet and there are many typos) Seems like, I've tested alert manager from quick-start docs - it doesn't contain config typo) Need to refactor examples...

Well, I can't promise, that multiarch images for all components will be added soon, it requires some time. Probably at the middle of September. Currently i'm working on OperatorHub integration. Also, I'll create an issue to the main repository - need to apply changes to build process there as well.

So feel free to add your experience and patch to the docs )

[elmariofredo]

Agree I have created following MR #80 to give guides to other fellow ARM builders. I think you can close this one as we have ARM support now. We can open new issue for multiarch images which can track progress for individual components CI changes. WDYT?

[f41gh7]

Thank you for MR. Docs become better)

I've opened an issue at main repo.

Well, let's keep this issue open, first I'm going to add multi-arch image to the operator, then port changes to main repo.

[valyala]

FYI, Docker images for VictoriaMetrics should already support multiple architectures - see https://hub.docker.com/r/victoriametrics/victoria-metrics/tags . The following architectures should be supported at the moment:

• amd64
• arm
• arm64
• 386
• ppc64le

See VictoriaMetrics/VictoriaMetrics#258 for details.

[f41gh7]

Thx @valyala ! I missed it, sry.

So the only operator left, will add multi-arch build soon.

[f41gh7]

I've managed to add docker manifest with multiarch, now operator should work without changes at all platforms. Docker image with changes - victoriametrics/operator:docker-multiarch-manifest.
Need to test it and update docs.

[elmariofredo]

I can confirm that it's working perfectly 👍

Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m30s  default-scheduler  Successfully assigned monitoring-system/vm-operator-vm-operator-685c7dcb7f-jcwwp to n1
  Normal  Pulling    4m30s  kubelet, n1        Pulling image "victoriametrics/operator:docker-multiarch-manifest"
  Normal  Pulled     4m19s  kubelet, n1        Successfully pulled image "victoriametrics/operator:docker-multiarch-manifest"
  Normal  Created    4m18s  kubelet, n1        Created container manager
  Normal  Started    4m17s  kubelet, n1        Started container manager

Thanks for your effort to make this happen 👏

[elmariofredo]

Actually found one issue withprometheus-config-reloader

operator/internal/config/config.go

Line 40 in cea27ec

ConfigReloadImage string `default:"quay.io/coreos/prometheus-config-reloader:v0.30.1"`

it looks like multiarch images for version v0.30.1 are not properly build I get following error::

standard_init_linux.go:211: exec user process caused "exec format error"

In order to make it work we need to either keep patching VM_VMAGENTDEFAULT_CONFIGRELOADIMAGE with arch suffix or upgrade to latest version v0.42.0 I have created #83 for latter.

[f41gh7]

Thanks!

I'm not able to test it, atm. It seems like only aws provides arm based VMs.

[elmariofredo]

If you want me to do some verification for you please do let me know

[f41gh7]

Closing it, thanks for your contribution!