Windows Install tests #34696

markus-ferrell · 2022-12-26T15:45:43Z

This PR changes the install tests to to make them windows compatible

markus-ferrell · 2023-01-02T19:00:39Z

@spackbot fix style

spackbot-app · 2023-01-02T19:00:41Z

Let me see if I can fix that for you!

spackbot-app · 2023-01-02T19:01:40Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/llnl/util/filesystem.py
  lib/spack/spack/build_environment.py
  lib/spack/spack/cmd/unit_test.py
  lib/spack/spack/installer.py
  lib/spack/spack/test/cmd/find.py
  lib/spack/spack/test/cmd/install.py
  lib/spack/spack/util/url.py
  lib/spack/spack/version.py
  var/spack/repos/builtin.mock/packages/build-error/package.py
  var/spack/repos/builtin.mock/packages/build-warnings/package.py
  var/spack/repos/builtin.mock/packages/callpath/package.py
  var/spack/repos/builtin.mock/packages/dyninst/package.py
  var/spack/repos/builtin.mock/packages/printing-package/package.py
==> Running isort checks
Fixing /tmp/tmpyinx4m1c/spack/var/spack/repos/builtin.mock/packages/build-error/package.py
Fixing /tmp/tmpyinx4m1c/spack/var/spack/repos/builtin.mock/packages/build-warnings/package.py
Fixing /tmp/tmpyinx4m1c/spack/var/spack/repos/builtin.mock/packages/callpath/package.py
Fixing /tmp/tmpyinx4m1c/spack/var/spack/repos/builtin.mock/packages/dyninst/package.py
Fixing /tmp/tmpyinx4m1c/spack/var/spack/repos/builtin.mock/packages/printing-package/package.py
Fixing /tmp/tmpyinx4m1c/spack/lib/spack/spack/version.py
  isort checks were clean
==> Running mypy checks
Success: no issues found in 571 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/build_environment.py
reformatted lib/spack/llnl/util/filesystem.py
reformatted lib/spack/spack/test/cmd/find.py
reformatted var/spack/repos/builtin.mock/packages/build-error/package.py
reformatted var/spack/repos/builtin.mock/packages/build-warnings/package.py
reformatted var/spack/repos/builtin.mock/packages/dyninst/package.py
reformatted var/spack/repos/builtin.mock/packages/printing-package/package.py
reformatted lib/spack/spack/installer.py
All done! ✨ 🍰 ✨
8 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/test/cmd/find.py:9: [F401] 'sys' imported but unused
var/spack/repos/builtin.mock/packages/build-warnings/package.py:23: [E501] line too long (101 > 99 characters)
  flake8 found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

markus-ferrell · 2023-01-02T19:31:00Z

@spackbot fix style

spackbot-app · 2023-01-02T19:31:02Z

Let me see if I can fix that for you!

spackbot-app · 2023-01-02T19:32:07Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, mypy, black, flake8
==> Modified files
  lib/spack/llnl/util/filesystem.py
  lib/spack/spack/build_environment.py
  lib/spack/spack/cmd/unit_test.py
  lib/spack/spack/installer.py
  lib/spack/spack/test/cmd/find.py
  lib/spack/spack/test/cmd/install.py
  lib/spack/spack/util/url.py
  lib/spack/spack/version.py
  var/spack/repos/builtin.mock/packages/build-error/package.py
  var/spack/repos/builtin.mock/packages/build-warnings/package.py
  var/spack/repos/builtin.mock/packages/callpath/package.py
  var/spack/repos/builtin.mock/packages/dyninst/package.py
  var/spack/repos/builtin.mock/packages/printing-package/package.py
==> Running isort checks
Fixing /tmp/tmp_6l2r73l/spack/var/spack/repos/builtin.mock/packages/build-error/package.py
Fixing /tmp/tmp_6l2r73l/spack/var/spack/repos/builtin.mock/packages/build-warnings/package.py
Fixing /tmp/tmp_6l2r73l/spack/var/spack/repos/builtin.mock/packages/callpath/package.py
Fixing /tmp/tmp_6l2r73l/spack/var/spack/repos/builtin.mock/packages/dyninst/package.py
Fixing /tmp/tmp_6l2r73l/spack/var/spack/repos/builtin.mock/packages/printing-package/package.py
Fixing /tmp/tmp_6l2r73l/spack/lib/spack/spack/version.py
  isort checks were clean
==> Running mypy checks
Success: no issues found in 571 source files
  mypy checks were clean
==> Running black checks
reformatted lib/spack/spack/build_environment.py
reformatted lib/spack/llnl/util/filesystem.py
reformatted lib/spack/spack/test/cmd/find.py
reformatted var/spack/repos/builtin.mock/packages/build-error/package.py
reformatted var/spack/repos/builtin.mock/packages/build-warnings/package.py
reformatted var/spack/repos/builtin.mock/packages/dyninst/package.py
reformatted var/spack/repos/builtin.mock/packages/printing-package/package.py
reformatted lib/spack/spack/installer.py
All done! ✨ 🍰 ✨
8 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

markus-ferrell · 2023-02-02T18:30:49Z

@spackbot fix style

spackbot-app · 2023-02-02T18:30:52Z

Let me see if I can fix that for you!

spackbot-app · 2023-02-02T18:31:48Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/llnl/util/filesystem.py
  lib/spack/spack/build_environment.py
  lib/spack/spack/cmd/unit_test.py
  lib/spack/spack/installer.py
  lib/spack/spack/test/cmd/find.py
  lib/spack/spack/test/cmd/install.py
  lib/spack/spack/util/url.py
  lib/spack/spack/version.py
  var/spack/repos/builtin.mock/packages/build-error/package.py
  var/spack/repos/builtin.mock/packages/build-warnings/package.py
  var/spack/repos/builtin.mock/packages/callpath/package.py
  var/spack/repos/builtin.mock/packages/printing-package/package.py
==> Running isort checks
Fixing /tmp/tmpnjl52605/spack/var/spack/repos/builtin.mock/packages/build-error/package.py
Fixing /tmp/tmpnjl52605/spack/var/spack/repos/builtin.mock/packages/build-warnings/package.py
Fixing /tmp/tmpnjl52605/spack/var/spack/repos/builtin.mock/packages/callpath/package.py
Fixing /tmp/tmpnjl52605/spack/var/spack/repos/builtin.mock/packages/printing-package/package.py
Fixing /tmp/tmpnjl52605/spack/lib/spack/spack/version.py
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/build_environment.py
reformatted lib/spack/llnl/util/filesystem.py
reformatted lib/spack/spack/installer.py
reformatted lib/spack/spack/test/cmd/install.py
reformatted lib/spack/spack/util/url.py
reformatted var/spack/repos/builtin.mock/packages/build-error/package.py
reformatted var/spack/repos/builtin.mock/packages/build-warnings/package.py
All done! ✨ 🍰 ✨
7 files reformatted, 5 files left unchanged.
  black checks were clean
==> Running flake8 checks
lib/spack/spack/test/cmd/find.py:136: [F821] undefined name 'sys'
lib/spack/spack/test/cmd/find.py:362: [F821] undefined name 'sys'
  flake8 found errors
==> Running mypy checks
lib/spack/spack/test/cmd/find.py:362: error: Name "sys" is not defined  [name-defined]
Found 1 error in 1 file (checked 576 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

markus-ferrell · 2023-02-02T19:01:41Z

lib/spack/spack/installer.py

@@ -2034,6 +2034,9 @@ def _real_install(self):

                        # Catch any errors to report to logging
                        self.timer.start(phase_fn.name)
+                        import time
+
+                        time.sleep(1)


This is to prevent a race condition in the logger for windows. This should be removed and replaced with a logger refactor.

What's the error here again? I know we went through this a little while ago, but I'd like to record that conversation here. Also can we sleep for less time?

The error was when running test_cdash_upload_build_error. This test is skipped in this PR but this change was more here for the discussion.

The installer would output the line Executing phase install after the actual install output. This makes cdash's build output reader fail because the build output is out of order.

lib/spack/llnl/util/filesystem.py

lib/spack/spack/test/cmd/install.py

var/spack/repos/builtin.mock/packages/build-warnings/package.py

lib/spack/spack/version.py

… into install_tests

lib/spack/spack/version/git_ref_lookup.py

markus-ferrell · 2023-08-10T21:21:04Z

@spackbot fix style

spackbot-app · 2023-08-10T21:21:06Z

Let me see if I can fix that for you!

spackbot-app · 2023-08-10T21:22:20Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/unit_test.py
  lib/spack/spack/directory_layout.py
  lib/spack/spack/test/cmd/install.py
  lib/spack/spack/test/directory_layout.py
  lib/spack/spack/test/util/util_url.py
  lib/spack/spack/test/versions.py
  lib/spack/spack/util/url.py
  lib/spack/spack/version/git_ref_lookup.py
  var/spack/repos/builtin.mock/packages/build-error/package.py
  var/spack/repos/builtin.mock/packages/build-warnings/package.py
  var/spack/repos/builtin.mock/packages/printing-package/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/cmd/install.py
All done! ✨ 🍰 ✨
1 file reformatted, 10 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 583 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

lib/spack/spack/test/cmd/install.py

… into install_tests

lib/spack/spack/version/git_ref_lookup.py

markus-ferrell · 2023-08-14T18:30:38Z

@spackbot fix style

spackbot-app · 2023-08-14T18:30:40Z

Let me see if I can fix that for you!

spackbot-app · 2023-08-14T18:31:56Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/cmd/unit_test.py
  lib/spack/spack/directory_layout.py
  lib/spack/spack/test/cmd/install.py
  lib/spack/spack/test/directory_layout.py
  lib/spack/spack/test/util/util_url.py
  lib/spack/spack/test/versions.py
  lib/spack/spack/util/url.py
  lib/spack/spack/version/git_ref_lookup.py
  var/spack/repos/builtin.mock/packages/build-error/package.py
  var/spack/repos/builtin.mock/packages/build-warnings/package.py
  var/spack/repos/builtin.mock/packages/printing-package/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/cmd/install.py
All done! ✨ 🍰 ✨
1 file reformatted, 10 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
Success: no issues found in 583 source files
  mypy checks were clean
==> spack style checks were clean

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

* The module-level skip for tests in `cmd.install` on Windows is removed. A few classes of errors still persist: * Cdash tests are not working on Windows * Tests for failed installs are also not working (this will require investigating bugs in output redirection) * Environments are not yet supported on Windows overall though, this enables testing of most basic uses of "spack install" * Git repositories cached for version lookups were using a layout that mimicked the URL as much as possible. This was useful for listing the cache directory and understanding what was present at a glance, but the paths were overly long on Windows. On all systems, the layout is now a single directory based on a hash of the Git URL and is shortened (which ensures a consistent and acceptable length, and also avoids special characters). * In particular, this removes util.url.parse_git_url and its associated test, which were used exclusively for generating the git cache layout * Bootstrapping is now enabled for unit tests on Windows

haampie · 2023-10-03T11:12:34Z

@scheibelp / @markus-ferrell I'm uncomfortable with this path normalization you've introduced in lib/spack/spack/directory_layout.py.

Prefixes of installations are registered in the database. Now there's a change in how those paths are computed, also on Linux/macOS/BSD, since str(Path(...)) normalizes x/./y to x/y.

This may lead to very hard to discover issues with relocation, when the prefix is recomputed (for some reason).

Can you leave the old behavior on non-Windows? Or fix it in another way?

johnwparent · 2023-10-03T13:47:07Z

@haampie FYI Markus is no longer on the project, so feel free to ping me in his place for this and all future issues related to his PRs.

I will look into alternative solutions, but if that looks like it'll be a long refactor, I'll just fall back on to the old behavior for this operation on *nix.

spackbot-app bot added build-environment commands compilers core PR affects Spack core functionality fetching tests General test capability(ies) utilities versions labels Dec 26, 2022

markus-ferrell force-pushed the install_tests branch from 6ec754f to db00ade Compare December 26, 2022 16:25

markus-ferrell marked this pull request as draft December 26, 2022 16:27

markus-ferrell force-pushed the install_tests branch from 809b6ac to 4132605 Compare January 2, 2023 19:29

markus-ferrell force-pushed the install_tests branch from 55a06d0 to 71b7e79 Compare January 4, 2023 19:06

markus-ferrell force-pushed the install_tests branch 2 times, most recently from c89f238 to bf5d539 Compare January 26, 2023 16:26

markus-ferrell marked this pull request as ready for review February 2, 2023 18:59

markus-ferrell commented Feb 2, 2023

View reviewed changes

lib/spack/llnl/util/filesystem.py Outdated Show resolved Hide resolved

johnwparent requested a review from scheibelp February 3, 2023 17:10

johnwparent reviewed Feb 3, 2023

View reviewed changes

lib/spack/spack/test/cmd/install.py Outdated Show resolved Hide resolved

johnwparent reviewed Feb 3, 2023

View reviewed changes

var/spack/repos/builtin.mock/packages/build-warnings/package.py Show resolved Hide resolved

johnwparent reviewed Feb 3, 2023

View reviewed changes

lib/spack/spack/version.py Outdated Show resolved Hide resolved

markus-ferrell added 4 commits August 9, 2023 14:58

remove unused imports

949cca6

Merge branch 'install_tests' of https://github.com/markus-ferrell/spack…

ef28449

… into install_tests

cleanups and hashing for unique folders for repository_uri

bdf423e

extra touch ups

ae3a0cc

markus-ferrell requested a review from scheibelp August 9, 2023 19:40

style

8834127

scheibelp reviewed Aug 10, 2023

View reviewed changes

lib/spack/spack/version/git_ref_lookup.py Outdated Show resolved Hide resolved

markus-ferrell added 2 commits August 10, 2023 16:50

Just use hashing for everything

d22fcd2

more descriptive skip messages

6bfb501

[@spackbot] updating style on behalf of markus-ferrell

7590f48

scheibelp reviewed Aug 10, 2023

View reviewed changes

lib/spack/spack/test/cmd/install.py Outdated Show resolved Hide resolved

markus-ferrell added 2 commits August 11, 2023 16:22

Add more specificity to skip messages

64b2ccf

Merge branch 'install_tests' of https://github.com/markus-ferrell/spack…

5432cf1

… into install_tests

scheibelp requested changes Aug 14, 2023

View reviewed changes

lib/spack/spack/version/git_ref_lookup.py Outdated Show resolved Hide resolved

Review changes

0c25cbe

[@spackbot] updating style on behalf of markus-ferrell

2b6718c

scheibelp approved these changes Aug 14, 2023

View reviewed changes

Spack on Windows automation moved this from In Progress to Awaiting Review Aug 14, 2023

scheibelp merged commit d823037 into spack:develop Aug 14, 2023
33 of 34 checks passed

Spack on Windows automation moved this from Awaiting Review to Done Aug 14, 2023

haampie mentioned this pull request Oct 3, 2023

spec.prefix / database inconsistencies around projections / paths due to Windows changes #40294

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows Install tests #34696

Windows Install tests #34696

markus-ferrell commented Dec 26, 2022 •

edited

markus-ferrell commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

markus-ferrell commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

markus-ferrell commented Feb 2, 2023

spackbot-app bot commented Feb 2, 2023

spackbot-app bot commented Feb 2, 2023

markus-ferrell Feb 2, 2023

johnwparent Feb 3, 2023

markus-ferrell Feb 6, 2023

markus-ferrell commented Aug 10, 2023

spackbot-app bot commented Aug 10, 2023

spackbot-app bot commented Aug 10, 2023

markus-ferrell commented Aug 14, 2023

spackbot-app bot commented Aug 14, 2023

spackbot-app bot commented Aug 14, 2023

haampie commented Oct 3, 2023 •

edited

johnwparent commented Oct 3, 2023

Windows Install tests #34696

Windows Install tests #34696

Conversation

markus-ferrell commented Dec 26, 2022 • edited

markus-ferrell commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

markus-ferrell commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

spackbot-app bot commented Jan 2, 2023

markus-ferrell commented Feb 2, 2023

spackbot-app bot commented Feb 2, 2023

spackbot-app bot commented Feb 2, 2023

markus-ferrell Feb 2, 2023

Choose a reason for hiding this comment

johnwparent Feb 3, 2023

Choose a reason for hiding this comment

markus-ferrell Feb 6, 2023

Choose a reason for hiding this comment

markus-ferrell commented Aug 10, 2023

spackbot-app bot commented Aug 10, 2023

spackbot-app bot commented Aug 10, 2023

markus-ferrell commented Aug 14, 2023

spackbot-app bot commented Aug 14, 2023

spackbot-app bot commented Aug 14, 2023

haampie commented Oct 3, 2023 • edited

johnwparent commented Oct 3, 2023

markus-ferrell commented Dec 26, 2022 •

edited

haampie commented Oct 3, 2023 •

edited