Decide (and document) whether uefi-test-runner should work outside the QEMU env

[nicholasbishop]

Currently the uefi-test-runner kinda sorta runs on real hardware. I just tried it on a recent Thinkpad, and it drew the usual blue/orange colors to the screen, some output flashed by, and it panicked on something related to serial.

The test runner currently has as lot of conditionals around things like "is this protocol supported?" and will often continue gracefully if something is missing. This has the advantage of making the runner more likely to work on real hardware, but it has the major disadvantage of making it easier to miss an unexpected error when running in the usual QEMU setup. And the more minor disadvantage of making the code a little more complicated.

This comes up from time to time in code review (recent example), so it would be good to make a decision one way or the other and document it.

[nicholasbishop]

My current feeling is that we should probably just make the test runner expect that it is running in our special QEMU setup (cargo xtask run), since even if we wanted to support real hardware it's easy to make mistakes without having a regular test on real hardware. Interested to hear others' thoughts :)

[phip1611]

The test runner currently has as lot of conditionals around things like "is this protocol supported?" and will often continue gracefully if something is missing. This has the advantage of making the runner more likely to work on real hardware, but it has the major disadvantage of making it easier to miss an unexpected error when running in the usual QEMU setup. And the more minor disadvantage of making the code a little more complicated.

Thanks for the initiative. I'm in favor of tighten the bond between the test runner and QEMU/OVMF. This will help us find bugs and errors.

[timrobertsdev]

Something to potentially consider: OvmfPkg in edk2 is missing some protocols that we would be unable to test if we're using the stock OVMF image for QEMU. IIRC I ran into this while implementing DebugSupport or a supporting protocol.

[nicholasbishop]

Something to potentially consider: OvmfPkg in edk2 is missing some protocols that we would be unable to test if we're using the stock OVMF image for QEMU.

For things that OVMF doesn't yet support we probably just don't have a realistic way to test it in CI at all. However, we could add separate programs to https://github.com/rust-osdev/uefi-rs/tree/main/uefi-test-runner/examples that can be manually tested on real hardware or some other env.