Docker image no longer works on older CPU (CPU does not support x86-64-v2)

[ghudiczius]

After upgrading my Docker based minio installation from RELEASE.2023-10-25T06-33-25Z to RELEASE.2023-11-01T01-57-10Z the container refuses to start on older CPU.

Expected Behavior

The minio container starts.

Current Behavior

The minio container fails to start with the exit code 127 and the following error message: Fatal glibc error: CPU does not support x86-64-v2. No other output is provided.

Steps to Reproduce (for bugs)

1. docker run -it minio/minio:RELEASE.2023-11-01T01-57-10Z

Context

While I understand this is an old CPU, my understanding is that minio itself does not require these CPU instructions. Furthermore, the binary used in the docker image works on the same machine so it seems the issue comes with the new base image. In fact, when trying to run the new base image I encounter the same error.

$ docker run registry.access.redhat.com/ubi9/ubi-minimal:9.2
Fatal glibc error: CPU does not support x86-64-v2

Regression

Yes:

• RELEASE.2023-11-01T01-57-10Z
• upgrade container base image to ubi-9 #18313
• switch minio container base image to ubi-mciro #18329

Your Environment

• Version used (minio --version):

minio version RELEASE.2023-11-01T01-57-10Z (commit-id=55e713db0a367f6cccee00af49f00c269e6ca619)
Runtime: go1.21.3 linux/amd64
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Copyright: 2015-2023 MinIO, Inc.

• Server setup and configuration: N/A

• Operating System and version (uname -a):

Linux mir 6.1.53-gentoo-x86_64-jmk-selinux #1 SMP PREEMPT_DYNAMIC Thu Sep 14 13:12:26 CEST 2023 x86_64 Intel(R) Xeon(R) CPU E5420 @ 2.50GHz GenuineIntel GNU/Linux

• Hardware:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
microcode       : 0x60b
cpu MHz         : 2500.082
cache size      : 6144 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl cpuid tsc_known_freq pni est ssse3 cx16 sse4_1 hypervisor lahf_lm
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips        : 5000.02
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

[ravindk89]

Looks like this issue overall has affected others on ubi-minimal 9:

• Fatal glibc error: CPU does not support x86-64-v2 keycloak/keycloak#17290
• Running ubi9 on Apple silicon breaks on unsupported x86_64-v2 instruction set containers/podman#15456

@kannappanr assuming we want to maintain RHEL9 as our baseline we can add to the docs that the container image uses ubi-minimal and may prevent launching on older CPUs. Not sure if there is a workaround by tweaking the container build.

[harshavardhana]

This is very easy for us, we can just build another container image here for older CPUs and mention this in the doc.

[harshavardhana]

docker pull quay.io/minio/minio:RELEASE.2023-11-01T01-57-10Z-cpuv1

Should fix this problem.

[ravindk89]

@harshavardhana will we be building that tag moving forward as well? If so I can mention it in the docs. I literally have an E5600 series xeon in the closet so this is a nice fix :D

[harshavardhana]

@harshavardhana will we be building that tag moving forward as well? If so I can mention it in the docs. I literally have an E5600 series xeon in the closet so this is a nice fix :D

Yes @ravindk89 we will continue to build it - this is not applicable to most of our customers anyways.

[ravindk89]

Great thanks!

[pbabilas]

Hmm, it seams that not working. On latest and the latest tagged version still has the same error. :(

[ghudiczius]

The latest tag is not affected, you need the tag ending in -cpuv1 (so far it's only RELEASE.2023-11-01T18-37-25Z-cpuv1)

[pbabilas]

cpuv1

gr8, thank you for reply.

[itwasdolly]

Could you add a tag that always points to the latest build? E.g. :cpuv1

[harshavardhana]

Don't use latest - please pin a tag @itwasdolly locally.

[itwasdolly]

So I'm required to constantly monitor for new builds?

[frederikbosch]

My minio/minio Gitlab service also broke. I could switch to quay.io/minio/minio:RELEASE.2023-11-01T01-57-10Z-cpuv1 but I rather refer to a location that receives updates.

[harshavardhana]

My minio/minio Gitlab service also broke. I could switch to quay.io/minio/minio:RELEASE.2023-11-01T01-57-10Z-cpuv1 but I rather refer to a location that receives updates.

A new release provides updates, change the release tag when new release happens.

[frederikbosch]

Sorry, I had a different impression based on the other replies. Very clear now, thank you!

[itwasdolly]

I'm confused. I have to manually monitor when a new release is made? Seems antiquated. Why not just have a tag that points to the latest cpuv1 version?

[frederikbosch]

A new release provides updates, change the release tag when new release happens.

That means there will be a tag containing the update, and there will be continuous release with the update. For now use the manually created tag by @harshavardhana, and change once the update is included in :latest.

[itwasdolly]

Thanks for the clarification.

[CosmicToast]

Current :latest still doesn't work, though multiple releases have been made since then.
There are newer "cpuv1" tags, such as the recent RELEASE.2023-12-02T10-51-33Z-cpuv1, but no tracking tag for future updates.
Can you clarify the intended approach for affected users?

[harshavardhana]

Current :latest still doesn't work, though multiple releases have been made since then. There are newer "cpuv1" tags, such as the recent RELEASE.2023-12-02T10-51-33Z-cpuv1, but no tracking tag for future updates. Can you clarify the intended approach for affected users?

Use the release tag, :latest is for latest CPUs not older ones

[CosmicToast]

So it’s not possible to take advantage of automatic updates?
Unless I missed a :release tag somewhere.

The previous comment suggests that there should be a rolling release tag:

For now use the manually created tag by @harshavardhana, and change once the update is included in :latest.