can not update 19.5 -> 20.0 Fatal glibc error: CPU does not support x86-64-v2

[amg-web]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.

Summary

can not update 19.5 -> 20.0
pods does not start due to:
kubectl logs --previous awx-whz-6d8f7f4589-sw8rv awx-whz-web -n whz Fatal glibc error: CPU does not support x86-64-v2

is it expected results on older CPU with x86-64 ?

AWX version

20.0

Select the relevant components

• UI
• API
• Docs

Installation method

minikube

Modifications

no

Ansible version

No response

Operating system

ubuntu 18.04

Web browser

Chrome

Steps to reproduce

update using awx-oprator:
git checkout 0.17.0
make deploy

Expected results

expect replacing of awx-web containers with new version

Actual results

ee replaced
awx-web still is old.

kubectl get pods -n whz
NAME                                               READY   STATUS             RESTARTS   AGE
awx-operator-controller-manager-646955f456-gdj4w   2/2     Running            4          18d
awx-whz-6d8f7f4589-sw8rv                           2/4     CrashLoopBackOff   10530      18d
awx-whz-855df94bc7-76xlj                           4/4     Running            16         100d
awx-whz-postgres-0                                 1/1     Running            6          155d

Additional information

cpu Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti

[tobgen]

@amg-web: My guess, its related to glibc 9.x require AVX-AES.
So its not really related to AWX, but I might be wrong ofc.

// T

[xyko0]

Hello
I got the exact same problem trying to update with k3s on a Ubuntu 20.04 server (Proxmox)
Is there a solution to avoid this issue ?

[Zguive]

Hello,
I suggest a workaround:

bug container awx-demo-web and awx-ee
Fatal glibc error: CPU does not support x86-64-v2

finalize the installation as indicated in the Quickinstall , then edit the deployment and change the image 20.1.0 by 19.5.1

$ kubectl edit deployment awx-demo -n awx

quay.io/ansible/awx:20.1.0 ==> quay.io/ansible/awx:19.5.1
( https://quay.io/repository/ansible/awx?tab=tags&tag=19.5.1 )

[jelly-is-bean]

i also encountered the issue - @amg-web I'm betting like me you are using visualisation software based in linux (for me it was Proxmox) for your kubernetes/awx install. There is something with the kvm64 cpu emulation/driver and in Proxmox I have the option to select a different type (I selected to past-in the host cpu - no emulation) Anyway - long long day troubleshooting short - I have managed to install and use 0.20.0

Edit: I think the issue is not specific to this project but is an issue with Centos-Stream9 with that kvm64 driver

[S3cBar0n]

i also encountered the issue - @amg-web I'm betting like me you are using visualisation software based in linux (for me it was Proxmox) for your kubernetes/awx install. There is something with the kvm64 cpu emulation/driver and in Proxmox I have the option to select a different type (I selected to past-in the host cpu - no emulation) Anyway - long long day troubleshooting short - I have managed to install and use 0.20.0

Edit: I think the issue is not specific to this project but is an issue with Centos-Stream9 with that kvm64 driver

I can confirm I had this same issue, which was solved by performing the same actions. Also running 0.20.0 just fine.

[amg-web]

jelly-is-bean, S3cBar0n
yes you are right
SolusVM KVM with QEMU (it uses a little old driver and I'm not sure it's possible to update it to the latest version on CentOS 7)
I'll try to play with CPU driver settings.

[jelly-is-bean]

@amg-web this is similar to Proxmox - when you are making the VM for the minikube cluster just passthrough your host cpu

Edit: screenshot taken from documentation for SolusVM KVM

[amg-web]

Yes tweaking CPU driver settings solved the problem

[CluelessTechnologist]

If it's CentOS fault where is the bug report? I mean KVM is a RedHat sponsored project. I was hoping to find a bug report somewhere so I can track the issue. As stated in this issue discussion it's not really a AWX issue.

Interesting Proxmox thread on the same issue: https://forum.proxmox.com/threads/kernel-panic-when-creating-vms-centos-9-stream-iso.104656/
Changing VM CPU to host will work but it has a downside that you might not be able to migrate the VM between hosts as easy, as I understand it from above thread.

[tovoro]

For me, changing Proxmox CPU to "Host" doesnt seem to help either. Does that mean my host CPU doesnt support this and I can stop trying to install AWX on that machine?

[amg-web]

@tovoro 19.5 should work

[tovoro]

Im sorry, I started again, checked out 0.19.0 and still get the same error..
Im on Debian 11 and follow this guide to install

[amg-web]

@tovoro which tag did you use for awx operator?
each awx operator version installs specific version of awx
tag of awx-oprator 0.16.1 is for awx 19.5

[Uzzi]

Hi, same problem with awx-oprator 0.30.0

[andreibrebene]

Hello,

I have the same error when I am trying to run AWX 21.7.0 on docker.
The VM is a RedHat 8.6 running on VBox.
I run to check the CPU on VM and the return is "CPU supports x86-64-v1", is there any tweaks to make the CPU be artificialy x86-v64-v2 supported?

Edit: I solved this issue disabling from Windows features the "Microsoft Hypervisor Platform", for some reason my VM was seeing that the CPU support only x-86-v64-v1...Now the VM see my CPU that can support x86-v64-v2.

[bentamircea]

Hi,

I'm curious to know why is x86-v64-v2 a requirement in the first place? As someone mentioned above it might not be a AWX requirement but rather tools used around it to deploy. I'm trying to deploy it with docker but on build I'm getting the same. I've tried running an already built image and it gives the same error when starting.
I'm not as fortunate as some of the others, my CPU really does not support x86-v64-v2; but I'm trying to figure this thing out.
Also, I've noticed that Ansible Tower does not have issue with older CPUs and seems to run just fine.

[andreibrebene]

@bentamircea x86-64-v2 brings support (among other things) for vector instructions up to Streaming SIMD Extensions 4.2 (SSE4.2) and Supplemental Streaming SIMD Extensions 3 (SSSE3), the POPCNT instruction (useful for data analysis and bit-fiddling in some data structures), and CMPXCHG16B (a two-word compare-and-swap instruction useful for concurrent algorithms).

From RHEL 9 you must have a CPU that supports x-86-64-v2 to run it (older CPUs already support this standard, you don't have to have a new CPU). I have a i5-9600k and I had the same issue, i put the fault on the linux system update but it wasn't this, my problem was because of the Microsoft Hyper-V, check on your windows features (if you run your VMs from VBox on Windows) that "Microsoft Hypervisor Platform" is not enabled, for some reason this feature from windows will limit your CPU and your VM will see that your CPU doesn't support a x86-v64-v2 standard.

Indeed, is not an AWX problem is a OS/Driver problem on how your system is seeing your CPU capabilities.

[bentamircea]

@andreibrebene I'm running a Debian 11 on bare metal so no hypervisor settings to fiddle with.

UPDATE: the images are based on quay.io/centos/centos:stream9 so that's why the containers are failing with non v2 CPUs.

[RaceFPV]

Continuing to see this error on awx 21.10.0, with proxmox and qemu/kvm64. We can't use host based cpu as that restricts the ability to migrate vm's from host to host. This is the only service that has any kind of issue with kvm64 out of the 100s that we host which feels pretty silly to need its own special node pool.

Edit: Changing the cpu type from 'kvm64' to 'max' in proxmox allows the pod to come up successfully while still allowing cross-architecture vm migration, which resolves this for me.

[orcema]

On proxmox I had the same issue with cpu type 'kvm64" and resolved it using the cpu type 'max'.
From now on I will always use the cpu type 'max'. Does anyone know the advantage using cpu type "kvm64' over 'max' ?

[syswe]

We are getting the same error, and we have not encountered such an error in any environment in our current 50-node cluster. Here we encounter this error on the container installed with the centos:stream9 image named init-projects. It would be very helpful for us if the related problem could be solved.

[nodecaf]

In KVM, I changed the CPU configuration from qemu64 to host-passthrough on the VM that the pod was running on, allowing the necessary CPU flags to be accessed by the container engine.

[hurdigurdi]

can confirm on Macbook M1 with kind, this will work once the QEMU cpuType is set to "max" using latest operator and AWX:

cpuType:
	aarch64: max

[a-farahani]

According to https://community.veeam.com/kasten-k10-support-92/fatal-glibc-error-cpu-does-not-support-x86-64-v2-4936?postid=41619#post41619:
I had this problem on ESXI. My EVC mode in ESXI was "Merom". The infrastructure team changed it to "Broadwell" and the problem was solved.