Errors on the TX path of the network device: `Failed to write to tap: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }`

[ShadowCurse]

Description:

All currently supported Firecracker versions (v1.7, v1.8, v1.9) experience error messages: Failed to write to tap: Os { code: 22, kind: InvalidInput, message: "Invalid argument" } and regressed performance on TX path when running on host kernels starting from v6.1.103 up to v6.1.106.

The root cause of the issue is related to the backport of the commit in the linux kernel 6.1 tree, which was fixed later on by the commit present in the kernel v6.1.107 and onward.

If you are impacted by this issue, we suggest pinning your host kernel to a not affected version prior to v6.1.103 or past v6.1.106. Otherwise make sure the commit with a fix is present in your host kernel.

[gudmundur]

What perfect timing, we started running into this issue at Vercel a few days ago when we had an automatic base AMI upgrade to our host AMIs. The affected AMI release is that of al2023-ami-minimal-2023.5.20240903.0-kernel-6.1-x86_64 (release notes). While AWS is looking into this issue, we've pinned our AMIs to the previous version, al2023-ami-minimal-2023.5.20240819.0-kernel-6.1-x86_64 (release notes). The main change between the two seems to be the Kernel upgrade from v6.1.102 to v6.1.106, which would match with what @ShadowCurse points out here above.

The steps for us to reproduce is run Firecracker with a fully configured tap device, and have that VM do network bound loads. The saw around 110k log lines per VM that was affected by this, and performance degradations as well.

[Manciukic]

Amazon Linux has released a new AMI al2023-ami-2023.5.20240916.0-kernel-6.1-x86_64 (release notes) yesterday which is based on v6.1.109 and includes the back-port of the upstream patch.
It doesn't include a following upstream fix (patch) which removes some false positives. This should be included in the Sep 30 release from Amazon Linux. However, internal testing didn't show any issues on the v6.1.109 kernel, which solves the issue of the TX errors.