flowey: output command for running vmm_tests

[tjones60]

Improves the xflowey vmm-tests command to output the cargo-nextest command need to run the tests outside of the flowey context, on Windows or Linux. This command is written to a script file in the output directory. Also build each package in a separate dir to avoid rebuilding.

[mattkur]

I'm really excited to see vmm-tests and the continued investment here. I was playing around with this.

1. I didn't see where the nextest command is printed out. I tried to reverse engineer what this file should be called, but couldn't grok it. Where do I look?
2. The default set of tests still seems to fail on my machine. Perhaps that means tests are just broke, or perhaps it means that we're not invoking in a "nice" way. Should cargo xflowey vmm-tests return pass?
3. cargo xflowey vmm-tests --build-only does not seem to work:

=== setting up vmm_tests env (flowey_lib_hvlite::init_vmm_tests_env) ===
Error: /home/mattkur/openvmm/flowey-persist/flowey_lib_hvlite__download_openvmm_vmm_tests_artifacts not in /home/mattkur/openvmm/flowey-out/.work/flowey_lib_hvlite___jobs__local_build_and_run_nextest_vmm_tests_1: prefix not found

[tjones60]

Thanks @mattkur for giving this a try.

1. In addition to being logged to the flowey command line, the command will be written to a file called run.sh/run.ps1 in the output folder specified with --dir. For windows tests, it is recommended that this dir be on the windows side: e.g.: --dir /mnt/e/vmm_tests. I think I will change --build-only to require --dir, because it doesn't make sense otherwise.
2. What is the error specifically? I some tests need newer host builds (especially hyper-v tests), so for example my dev box with 23h2 doesn't pass all tests either.
3. Once I do what I described in 1 this shouldn't happen. (It is trying to convert everything to relative paths to make the output portable but it isn't all in the same folder since you didn't specify a folder to copy to)

[mattkur]

nit, as I play with this further: --install-missing-deps does not install llvm or lld (sudo apt install clang-tools-14 lld-14), which seems to be required for building pal (even after . ./build_support/setup_windows_cross.sh)

[mattkur]

Thanks @mattkur for giving this a try.

1. In addition to being logged to the flowey command line, the command will be written to a file called run.sh/run.ps1 in the output folder specified with --dir. For windows tests, it is recommended that this dir be on the windows side: e.g.: --dir /mnt/e/vmm_tests. I think I will change --build-only to require --dir, because it doesn't make sense otherwise.
2. What is the error specifically? I some tests need newer host builds (especially hyper-v tests), so for example my dev box with 23h2 doesn't pass all tests either.
3. Once I do what I described in 1 this shouldn't happen. (It is trying to convert everything to relative paths to make the output portable but it isn't all in the same folder since you didn't specify a folder to copy to)

I don't see the command written to a script. I do see the stdout, but that's lost if tests have failures (since the logs scroll through my terminal buffer).

PS S:\win-vmm-tests> dir -Recurse -Filter *.ps1

In addition, can we do anything to more gracefully detect:

• Lack of running as admin (for HyperV tests)
• Execution policy

[tjones60]

I don't see the command written to a script. I do see the stdout, but that's lost if tests have failures (since the logs scroll through my terminal buffer).

PS S:\win-vmm-tests> dir -Recurse -Filter *.ps1

In addition, can we do anything to more gracefully detect:

• Lack of running as admin (for HyperV tests)
• Execution policy

It only writes the command to a file if you pass --build-only (the command, as run from wsl, is slightly different than the one written to the powershell script). If you think it would be useful, we could always generate that script. I can look into checking for running as admin. By execution policy, do you mean powershell script execution?

[mattkur]

I don't see the command written to a script. I do see the stdout, but that's lost if tests have failures (since the logs scroll through my terminal buffer).

PS S:\win-vmm-tests> dir -Recurse -Filter *.ps1

In addition, can we do anything to more gracefully detect:

• Lack of running as admin (for HyperV tests)
• Execution policy

It only writes the command to a file if you pass --build-only (the command, as run from wsl, is slightly different than the one written to the powershell script). If you think it would be useful, we could always generate that script. I can look into checking for running as admin. By execution policy, do you mean powershell script execution?

• Yep, by execution policy, I mean: Set-ExecutionPolicy Bypass -Scope Process -Force
• For admin, something like https://github.com/gerardog/gsudo might be interesting. (I haven't looked to see if it's something I'd be comfortable recommending from a security pov)

For the test failures I was seeing: the ci pipeline sets the AllowFirmwareLoadFromFile. We should either update our guide for vmm tests to point to the existing hyperv setup instructions, or make it friendlier here.

For the run.ps1 ... I personally think it's confusing to have this difference in behavior between the user specifying --build-only and without. Imagine that the user first runs cargo xflowey vmm-tests and then notices a failure. Re-running just that test to debug would be easier if there's a script to run them.

[justus-camp-microsoft]

Discussed over teams, but we should make the --artifacts flag require --filter so that it's respected by the image dependency auto-installation. Should we update this document to reflect the changes in the dev workflow?