rawhide: 6.2 kernel fails to boot on AWS Xen instances

[dustymabe]

Our AWS Kola tests that run on the i3.large instance type started failing with 38.20230105.91.0 (kernel-6.2.0-0.rc2.18.fc38).

The last known good kernel is kernel-6.1.0-65.fc38 (note that we don't test debug kernels so none of the kernels in between have been tested). I have verified that the kernel is the culprit by reverting only that package and allow all others to be updated.

The boot logs looks something like:

[    4.523515] udevadm[458]: systemd-udev-settle.service is deprecated. Please fix multipathd-configure.service not to pull it in.^M            
[    4.904874] Invalid max_queues (4), will use default max: 1.^M                                                                               
[    4.911101] ena 0000:00:03.0: ENA device version: 0.10^M                                                                                     
[    4.913730] ena 0000:00:03.0: ENA controller version: 0.0.1 implementation version 1^M                                                       
[    4.933060] blkfront: xvda: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled^M  
[    4.939967] ena 0000:00:03.0: LLQ is not supported Fallback to host mode policy.^M                                                           
[    4.940309] ena 0000:00:03.0 (unnamed net_device) (uninitialized): Failed to enable MSI-X. irq_cnt -524^M                                    
[    4.940311] ena 0000:00:03.0: Can not reserve msix vectors^M                                                                                 
[    4.940313] ena 0000:00:03.0: Failed to enable and set the admin interrupts^M                                                                
[    4.941460] ena: probe of 0000:00:03.0 failed with error -28

The more complete log is at i-0b236cef570dfad9c.log.txt

[dustymabe]

For now since this test doesn't block builds I'm thinking maybe we don't revert the kernel in rawhide so that we get testing on new kernel updates and just in case a future update fixes the problem we'll know sooner.

[davdunc]

Investigating through internal ticket at AWS

[davdunc]

David Woodhouse pointed to his patch upstream https://lore.kernel.org/all/4bffa69a949bfdc92c4a18e5a1c3cbb3b94a0d32.camel@infradead.org/ for the kernel. This is likely the fix for the issue.

[dustymabe]

Looks like a few patches landed upstream to address this that are in v6.2-rc6.

• 0a3a58d
• 6c79699

which should hit rawhide soon.

[dustymabe]

This landed in our rawhide stream in 38.20230204.91.0 (kernel-6.2.0-0.rc6.20230202git9f266ccaa2f5.46.fc38) and the AWS xen tests passed!

[2023-02-04T02:34:55.944Z] --- PASS: non-exclusive-test-bucket-0 (200.51s)
[2023-02-04T02:34:55.944Z]     --- PASS: non-exclusive-test-bucket-0/ext.config.platforms.aws.assert-xen (2.07s)