spack ci: add support for running stand-alone tests #27877
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
3 times, most recently
from
52cd1d8
to
dc273fb
Compare
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
2 times, most recently
from
3e47927
to
9ef24b3
Compare
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
from
578b64d
to
e1df1cf
Compare
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
from
2d88f26
to
817941b
Compare
|
@zackgalbreath @tgamblin @becker33 (Update: Adding Commit ec03ceb generated the output below from a run of
This corresponds to the |
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
from
4d32484
to
c291bcc
Compare
|
Since we want to know if a package does not have tests to run, I created a dummy test part so we can flag/track such packages (starting with commit 2c1ac7c). What do others think of reporting packages in this manner? An example output from running at the command line for the Is there anything missing or you would want changed? |
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
2 times, most recently
from
7012ba2
to
eb0310e
Compare
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
7 times, most recently
from
7dda56f
to
de1fd1d
Compare
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
3 times, most recently
from
9c2454e
to
ebb1e82
Compare
|
Interesting test failures in what appear to be a section untouched by this PR (Updated example below): Note the error seems to be related to a file path being treated as a directory: |
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
2 times, most recently
from
c4bfe48
to
3b50789
Compare
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: isort, mypy, black, flake8
==> Modified files
lib/spack/spack/binary_distribution.py
lib/spack/spack/build_environment.py
lib/spack/spack/ci.py
lib/spack/spack/cmd/__init__.py
lib/spack/spack/cmd/ci.py
lib/spack/spack/cmd/install.py
lib/spack/spack/cmd/test.py
lib/spack/spack/environment/environment.py
lib/spack/spack/install_test.py
lib/spack/spack/installer.py
lib/spack/spack/package_base.py
lib/spack/spack/relocate.py
lib/spack/spack/report.py
lib/spack/spack/reporters/cdash.py
lib/spack/spack/reporters/extract.py
lib/spack/spack/reporters/junit.py
lib/spack/spack/schema/gitlab_ci.py
lib/spack/spack/test/ci.py
lib/spack/spack/test/cmd/ci.py
lib/spack/spack/test/cmd/config.py
lib/spack/spack/test/cmd/install.py
lib/spack/spack/test/cmd/test.py
lib/spack/spack/test/reporters.py
lib/spack/spack/test/test_suite.py
lib/spack/spack/test/util/path.py
lib/spack/spack/util/path.py
==> Running isort checks
isort checks were clean
==> Running mypy checks
Success: no issues found in 557 source files
mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/ci.py
reformatted lib/spack/spack/cmd/install.py
reformatted lib/spack/spack/ci.py
reformatted lib/spack/spack/install_test.py
reformatted lib/spack/spack/installer.py
reformatted lib/spack/spack/package_base.py
reformatted lib/spack/spack/report.py
reformatted lib/spack/spack/reporters/extract.py
reformatted lib/spack/spack/reporters/junit.py
reformatted lib/spack/spack/schema/gitlab_ci.py
reformatted lib/spack/spack/reporters/cdash.py
reformatted lib/spack/spack/test/ci.py
reformatted lib/spack/spack/test/cmd/ci.py
reformatted lib/spack/spack/test/cmd/install.py
reformatted lib/spack/spack/test/reporters.py
reformatted lib/spack/spack/test/util/path.py
reformatted lib/spack/spack/test/test_suite.py
All done! ✨ 🍰 ✨
17 files reformatted, 9 files left unchanged.
black checks were clean
==> Running flake8 checks
flake8 checks were clean
==> spack style checks were clean
I've updated the branch with isort fixes. |
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
from
3372b43
to
2d5af98
Compare
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: isort, mypy, black, flake8
==> Modified files
lib/spack/spack/build_environment.py
lib/spack/spack/ci.py
lib/spack/spack/cmd/__init__.py
lib/spack/spack/cmd/ci.py
lib/spack/spack/cmd/install.py
lib/spack/spack/cmd/test.py
lib/spack/spack/environment/environment.py
lib/spack/spack/install_test.py
lib/spack/spack/installer.py
lib/spack/spack/main.py
lib/spack/spack/package_base.py
lib/spack/spack/report.py
lib/spack/spack/reporters/cdash.py
lib/spack/spack/reporters/extract.py
lib/spack/spack/reporters/junit.py
lib/spack/spack/schema/gitlab_ci.py
lib/spack/spack/test/ci.py
lib/spack/spack/test/cmd/ci.py
lib/spack/spack/test/cmd/config.py
lib/spack/spack/test/cmd/install.py
lib/spack/spack/test/cmd/test.py
lib/spack/spack/test/reporters.py
lib/spack/spack/test/test_suite.py
==> Running isort checks
isort checks were clean
==> Running mypy checks
Success: no issues found in 557 source files
mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/cmd/ci.py
reformatted lib/spack/spack/cmd/install.py
reformatted lib/spack/spack/ci.py
reformatted lib/spack/spack/install_test.py
reformatted lib/spack/spack/installer.py
reformatted lib/spack/spack/report.py
reformatted lib/spack/spack/reporters/cdash.py
reformatted lib/spack/spack/reporters/extract.py
reformatted lib/spack/spack/reporters/junit.py
reformatted lib/spack/spack/schema/gitlab_ci.py
reformatted lib/spack/spack/package_base.py
reformatted lib/spack/spack/test/ci.py
reformatted lib/spack/spack/test/cmd/ci.py
reformatted lib/spack/spack/test/cmd/install.py
reformatted lib/spack/spack/test/reporters.py
reformatted lib/spack/spack/test/test_suite.py
All done! ✨ 🍰 ✨
16 files reformatted, 7 files left unchanged.
black checks were clean
==> Running flake8 checks
flake8 checks were clean
==> spack style checks were clean
I've updated the branch with isort fixes. |
|
The |
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
from
8a42206
to
f51f18b
Compare
|
Rebased to develop in the hopes that that odd |
|
Everything is currently passing so I am going to drop the temporary mirror, etc. |
This support requires adding the '--tests' option to 'spack ci rebuild'. Packages whose stand-alone tests are broken (in the CI environment) can be configured in gitlab-ci to be skipped by adding them to broken-tests-packages. Highlights include: - Restructured 'spack ci' help to provide better subcommand summaries; - Ensured only one InstallError (i.e., installer's) rather than allowing build_environment to have its own; and - Refactored CI and CDash reporting to keep CDash-related properties and behavior in a separate class.
tldahlgren
force-pushed
the
spack-ci-stand-alone-testing
branch
from
f51f18b
to
cb2bbff
Compare
|
@tgamblin Looks like all checks have passed and this PR is RFR. |
|
ping @tgamblin |
tgamblin
approved these changes
Aug 23, 2022
There was a problem hiding this comment.
This looks great to me @tldahlgren! Thanks for all the work debugging pipelines to get this running properly.
Spack v0.19.0 release
automation
moved this from Review in progress
to Reviewer approved
Aug 23, 2022
ma595
pushed a commit
to ma595/spack
that referenced
this pull request
Sep 13, 2022
This support requires adding the '--tests' option to 'spack ci rebuild'.
Packages whose stand-alone tests are broken (in the CI environment) can
be configured in gitlab-ci to be skipped by adding them to
broken-tests-packages.
Highlights include:
- Restructured 'spack ci' help to provide better subcommand summaries;
- Ensured only one InstallError (i.e., installer's) rather than allowing
build_environment to have its own; and
- Refactored CI and CDash reporting to keep CDash-related properties and
behavior in a separate class.
This allows stand-alone tests from `spack ci` to run when the `--tests`
option is used. With `--tests`, stand-alone tests are run **after** a
**successful** (re)build of the package. Test results are collected
and report(able) using CDash.
This PR adds the following features:
- Adds `-t` and `--tests` to `spack ci rebuild` to run stand-alone tests;
- Adds `--fail-fast` to stop stand-alone tests after the first failure;
- Ensures a *single* `InstallError` across packages
(i.e., removes second class from build environment);
- Captures skipping tests for externals and uninstalled packages
(for CDash reporting);
- Copies test logs and outputs to the CI artifacts directory to facilitate
debugging;
- Parses stand-alone test results to report outputs from each `run_test` as
separate test parts (CDash reporting);
- Logs a test completion message to allow capture of timing of the last
`run_test` part;
- Adds the runner description to the CDash site to better distinguish entries
in CDash tables;
- Adds `gitlab-ci` `broken-tests-packages` to CI configuration to skip
stand-alone testing for packages with known issues;
- Changes `spack ci --help` so description of each subcommand is a single line;
- Changes `spack ci <subcommand> --help` to provide the full description of
each command (versus no description); and
- Ensures `junit` test log file ends in an `.xml` extension (versus default where
it does not).
Tasks:
- [x] Include the equivalent of the architecture information, or at least the host target, in the CDash output
- [x] Upload stand-alone test results files as `test` artifacts
- [x] Confirm tests are run in GitLab
- [x] Ensure CDash results are uploaded as artifacts
- [x] Resolve issues with CDash build-and test results appearing on same row of the table
- [x] Add unit tests as needed
- [x] Investigate why some (dependency) packages don't have test results (e.g., related from other pipelines)
- [x] Ensure proper parsing and reporting of skipped tests (as `not run`) .. post- spack#28701 merge
- [x] Restore the proper CDash URLand or mirror ONCE out-of-band testing completed
alalazo
reviewed
Dec 27, 2022
| @@ -245,6 +245,7 @@ def __init__(self, cls, function, format_name, args): | |||
| self.cls = cls | |||
| self.function = function | |||
| self.filename = None | |||
| self.ctest_parsing = getattr(args, "ctest_parsing", False) | |||
There was a problem hiding this comment.
@zackgalbreath @tldahlgren Where is args.ctest_parsing defined? I'm currently trying to refactor these classes to make them independent of argparse namespaces and for some reason I can't find references to this argument.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this work is to run stand-alone tests from
spack ciwhen the--testsoption is used. When the option is used, the stand-alone tests are run after a successful (re)build of the package. Test results are collected and report(able) using CDash.This PR adds the following features:
-tand--teststospack ci rebuildto run stand-alone tests;--fail-fastto stop stand-alone tests after the first failure;InstallErroracross packages (i.e., removes second class from build environment);run_testas separate test parts (CDash reporting);run_testpart;gitlab-cibroken-tests-packagesto CI configuration to skip stand-alone testing for packages with known issues;spack ci --helpso description of each subcommand is a single, concise line;spack ci <subcommand> --helpto provide the full description of each command (versus no description); andjunittest log file ends in an.xmlextension (versus default where it does not).Update (2022 Mar 28): Handling of concretization failures that do not get reported to CDash is expected to be addressed in a subsequent issue/PR.
Update (2022 Apr 7):
spack ci rebuild --testsshould not be be trying to run tests for job spec dependencies, only the job spec. If the build jobs of dependency specs fail to upload to the build cache (e.g., configuration, permissions), then those jobs should be failing. (TBD: Should this PR ensure that check or is it better to have a separate PR?)Tasks:
testartifacts- [ ] Possibly add full hash in build name -- Scott Wittenburg is working on a significant change in this area ATMnot run) .. post- Testing: Optionally run tests on externally installed packages #28701 merge- [ ] Determine why concretization failure jobs and associated reports -- actually justUpdate.xml-- are not showing up in CDash