Skip to content

Conversation

ayylol
Copy link
Contributor

@ayylol ayylol commented Oct 16, 2024

MOVED TO #16016 (to make this not a sycl-devops-pr)

Adds functionality to split the compilation and execution of e2e tests across separate machines.

This functionality is enabled via two new lit features run-mode and build-mode. By default both of these are enabled and tests run just as they have before. We can call lit with the flag --param split-mode= with either build or run to set these features.

When using split-mode=build only the build-mode feature is added. In this mode all tests that can be built on the system will be built. When running a lit test in this mode two key things change:

  • All RUN: lines will be executed unless they contain %{run}, %{run-unfiltered-devices} or %if run-mode.
  • Unsupported and requires statements are ignored. Currently the only way to mark tests as unsupported in build only mode is to include run-mode in a requires statement.

When using split-mode=run only the run-mode is set. In this case tests are not built, they are only executed. Deciding whether a test is supported in this mode works the same way as when the split-mode parameter is not set. To only execute the tests we ignore all lit RUN: lines unless they contain %{run}, %{run-unfiltered-devices} or %if run-mode. Since we do not build the tests in this mode, for any test to pass we must have ran the tests in either build or non-split mode previously.

Tests that are marked as REQUIRES: run-mode, will ignore the run line filtering from the run only mode. This is done since we know in this case we will not have an executable binary from the build only mode.

Some notes/current limitations:

  • Currently only the spir64 triple is supported for the build only mode.
  • Currently tests on build-mode can only be marked as unsupported by using REQUIRES: run-mode
  • The build only mode can be ran manually on CI with the SYCL E2E action, selecting the "Linux, build" runner, and adding --param split-mode=build to LIT_OPTS. This pr does not have the needed CI changes to be able to use the run only mode.

@ayylol
Copy link
Contributor Author

ayylol commented Oct 17, 2024

Dry run of e2e test builds with no logic for setting triple or treating tests as unsupported: https://github.com/intel/llvm/actions/runs/11388738602

68 tests failed, so currently that should be the max amount of unsupported tests for building with the spir64 triple on Linux.

@ayylol
Copy link
Contributor Author

ayylol commented Oct 30, 2024

Here is the results of the latest dry-run, after removing the REQUIRES/UNSUPPORTED logic for the triples: https://github.com/intel/llvm/actions/runs/11596214432

In total there were 63 Failures. 6 are addressed by #15789, another 9 are addressed by #15787. This leaves 48 tests that would need changes.
41 of these tests would need to be marked with REQUIRES: run-mode:

  • 20 are specifically meant for cuda or hip and cannot be built for the spir64 triple
  • 9 are tests only meant for windows
  • 6 are tests marked as TEMPORARY_DISABLED
  • 5 tests require Vulkan
  • 1 tests requires that zstd is not present in the environment.

The remaining 7 are fixed in a different manner:

10 tests are marked as XPASS, these should be addressed by #15891

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants