Skip to content

flowey: output command for running vmm_tests #1508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Jun 20, 2025

Conversation

tjones60
Copy link
Contributor

Improves the xflowey vmm-tests command to output the cargo-nextest command need to run the tests outside of the flowey context, on Windows or Linux. This command is written to a script file in the output directory. Also build each package in a separate dir to avoid rebuilding.

@tjones60 tjones60 requested a review from a team as a code owner June 11, 2025 17:32
@mattkur
Copy link
Contributor

mattkur commented Jun 11, 2025

I'm really excited to see vmm-tests and the continued investment here. I was playing around with this.

  1. I didn't see where the nextest command is printed out. I tried to reverse engineer what this file should be called, but couldn't grok it. Where do I look?
  2. The default set of tests still seems to fail on my machine. Perhaps that means tests are just broke, or perhaps it means that we're not invoking in a "nice" way. Should cargo xflowey vmm-tests return pass?
  3. cargo xflowey vmm-tests --build-only does not seem to work:
=== setting up vmm_tests env (flowey_lib_hvlite::init_vmm_tests_env) ===
Error: /home/mattkur/openvmm/flowey-persist/flowey_lib_hvlite__download_openvmm_vmm_tests_artifacts not in /home/mattkur/openvmm/flowey-out/.work/flowey_lib_hvlite___jobs__local_build_and_run_nextest_vmm_tests_1: prefix not found

@tjones60
Copy link
Contributor Author

tjones60 commented Jun 11, 2025

Thanks @mattkur for giving this a try.

  1. In addition to being logged to the flowey command line, the command will be written to a file called run.sh/run.ps1 in the output folder specified with --dir. For windows tests, it is recommended that this dir be on the windows side: e.g.: --dir /mnt/e/vmm_tests. I think I will change --build-only to require --dir, because it doesn't make sense otherwise.
  2. What is the error specifically? I some tests need newer host builds (especially hyper-v tests), so for example my dev box with 23h2 doesn't pass all tests either.
  3. Once I do what I described in 1 this shouldn't happen. (It is trying to convert everything to relative paths to make the output portable but it isn't all in the same folder since you didn't specify a folder to copy to)

@mattkur
Copy link
Contributor

mattkur commented Jun 12, 2025

nit, as I play with this further: --install-missing-deps does not install llvm or lld (sudo apt install clang-tools-14 lld-14), which seems to be required for building pal (even after . ./build_support/setup_windows_cross.sh)

@mattkur
Copy link
Contributor

mattkur commented Jun 13, 2025

Thanks @mattkur for giving this a try.

  1. In addition to being logged to the flowey command line, the command will be written to a file called run.sh/run.ps1 in the output folder specified with --dir. For windows tests, it is recommended that this dir be on the windows side: e.g.: --dir /mnt/e/vmm_tests. I think I will change --build-only to require --dir, because it doesn't make sense otherwise.
  2. What is the error specifically? I some tests need newer host builds (especially hyper-v tests), so for example my dev box with 23h2 doesn't pass all tests either.
  3. Once I do what I described in 1 this shouldn't happen. (It is trying to convert everything to relative paths to make the output portable but it isn't all in the same folder since you didn't specify a folder to copy to)

I don't see the command written to a script. I do see the stdout, but that's lost if tests have failures (since the logs scroll through my terminal buffer).

PS S:\win-vmm-tests> dir -Recurse -Filter *.ps1

In addition, can we do anything to more gracefully detect:

  • Lack of running as admin (for HyperV tests)
  • Execution policy

@tjones60
Copy link
Contributor Author

tjones60 commented Jun 13, 2025

I don't see the command written to a script. I do see the stdout, but that's lost if tests have failures (since the logs scroll through my terminal buffer).

PS S:\win-vmm-tests> dir -Recurse -Filter *.ps1

In addition, can we do anything to more gracefully detect:

  • Lack of running as admin (for HyperV tests)
  • Execution policy

It only writes the command to a file if you pass --build-only (the command, as run from wsl, is slightly different than the one written to the powershell script). If you think it would be useful, we could always generate that script. I can look into checking for running as admin. By execution policy, do you mean powershell script execution?

@mattkur
Copy link
Contributor

mattkur commented Jun 16, 2025

I don't see the command written to a script. I do see the stdout, but that's lost if tests have failures (since the logs scroll through my terminal buffer).

PS S:\win-vmm-tests> dir -Recurse -Filter *.ps1

In addition, can we do anything to more gracefully detect:

  • Lack of running as admin (for HyperV tests)
  • Execution policy

It only writes the command to a file if you pass --build-only (the command, as run from wsl, is slightly different than the one written to the powershell script). If you think it would be useful, we could always generate that script. I can look into checking for running as admin. By execution policy, do you mean powershell script execution?

  • Yep, by execution policy, I mean: Set-ExecutionPolicy Bypass -Scope Process -Force
  • For admin, something like https://github.com/gerardog/gsudo might be interesting. (I haven't looked to see if it's something I'd be comfortable recommending from a security pov)

For the test failures I was seeing: the ci pipeline sets the AllowFirmwareLoadFromFile. We should either update our guide for vmm tests to point to the existing hyperv setup instructions, or make it friendlier here.

For the run.ps1 ... I personally think it's confusing to have this difference in behavior between the user specifying --build-only and without. Imagine that the user first runs cargo xflowey vmm-tests and then notices a failure. Re-running just that test to debug would be easier if there's a script to run them.

Copy link
Contributor

@justus-camp-microsoft justus-camp-microsoft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed over teams, but we should make the --artifacts flag require --filter so that it's respected by the image dependency auto-installation. Should we update this document to reflect the changes in the dev workflow?

@github-actions github-actions bot added the Guide label Jun 20, 2025
@tjones60 tjones60 merged commit da3977b into microsoft:main Jun 20, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants