FIPS Kernel Panic on v19 Kernel 5.4.x

[tupacalypse187]

What happened:
Kernel panic on reboot when building v1.19 with FIPS enabled using kernel 5.4.x

What you expected to happen:
Host should reboot without kernel panic and have FIPS enabled

How to reproduce it (as minimally and precisely as possible):
Build v1.19 on 5.4.x kernel with FIPS enabled following #513

Anything else we need to know?:
Builds successfully on all versions prior to kernel upgrade from 4.x to 5.4.x https://www.github.com/awslabs/amazon-eks-ami/commit/e3f1b910f83ad1f27e68312e50474ea6059f052d#diff-0a1b54f1420f4f3189806328b24b37ccR10

Error from EC2 console log:

[    1.389896] alg: ecdh: test failed on vector 2, err=-14
[    1.392566] Kernel panic - not syncing: alg: self-tests for ecdh-generic (ecdh) failed in fips mode!
[    1.396551] CPU: 0 PID: 259 Comm: cryptomgr_test Not tainted 5.4.95-42.163.amzn2.x86_64 #1

Environment:

• AWS Region: All
• Instance Type(s): All
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): All
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.19
• AMI Version: ami-0d45bae218253b811
• Kernel (e.g. uname -a): 5.4.95-42.163.amzn2 amzn2extra-kernel-5.4
• Release information (run cat /etc/eks/release on a node):

[mmerkes]

We're currently engaging with the Amazon Linux team to try to root cause this issue and get the kernel patched to fix it. In the meantime, there are a couple options for customers to unblock themselves until we solve the root issue.

Disclaimer: These steps may not be realistic for all customers, and I have not validated these other than that they successfully use the 4.14 kernel. The 4.14 kernel is tested with AMIs built for k8s version 1.18 and below.

Pass in 4.14 "kernel_version" when building AMI

If a customer builds their own AMI with the amazon-eks-ami GitHub repo, they can set 4.14 as the kernel version and build a new AMI.

Downgrade the kernel

I haven't done any extensive testing on this, so it would be best if customers were able to test in a dev environment, but they could downgrade the kernel before using the AMIs. This worked for me.

Get the install 4.14 version:

$ yum list kernel
Installed Packages
kernel.x86_64             4.14.219-164.354.amzn2                           installed
kernel.x86_64             5.4.95-42.163.amzn2                              @amzn2extra-kernel-5.4

Set the default kernel and verify:

# Update kernel version to match the above result
$ sudo grubby --set-default /boot/vmlinuz-4.14.219-164.354.amzn2.x86_64
$ sudo grubby --default-kernel
/boot/vmlinuz-4.14.219-164.354.amzn2.x86_64

Reboot:

sudo reboot

[ygoodmn]

Still see this error with Kernel, any update ?
Linux ip-10-1-0-112.ec2.internal 5.4.110-54.182.amzn2.x86_64 #1 SMP Fri Apr 9 17:56:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[mmerkes]

Sorry for the delays. The AmazonLinux team is still actively working on this. I will post here as soon as I have word that it's fixed.

[samihoda]

@mmerkes I am an Amazon employee. I have a partner asking about this issue, as a blocker. Will reach out for feedback.

[mmerkes]

AmazonLinux believes that they have a fix for this issue, and we're just waiting on them to make it available to customers. I'll update here when it's pushed.

[mmerkes]

The AL2 fix is now available in the amazon-linux-extras repo, so if you use the AL2 image to build your own AMIs, you should be good to go. EKS is working on releasing new AMIs with the fix as well, so expect that to be coming very soon. We'll update here and resolve the issue when we've published new AMIs.

[mmerkes]

As of release v20210716, this issue should be resolved! If you still have issues, please comment here or open a new issue.