Initial contribution of LibPressio ecosystem #32630
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
LibPressio is a generic abstraction for the compression of data. It supports over 40 compressors and is used at over 10 institutions (that we know about). Previously these packages were part of a 3rd party [package repository](https://github.com/robertu94/spack_packages). This PR contains packages (and modifications for 2 existing packages) to expose the LibPressio support. I expect this PR to take some time to settle and finalize because of its size. Modifications: - adios2 - adds libpressio support to the existing package. - cusz the old package was not portable and lacked key dependencies this re-write is down with the knowledge of the maintainers @dingwentao and @jtian0. New Packages: - arc an data corruption tool for compressed data developed by the lab of @joncalhoun40. @joncalhoun40 would you be a maintainer for this? - bitgroomingz a standalone implementation of bitgrooming from NetCDF packaged by @disheng222. - fpzip a specialized compressor for floating point data developed by @lindstro and others and a predecessor to ZFP. - libdistributed a MPI work queue with cancelation used by libpressio and libpressio_opt since this work queue is used by multiple projects outside of libpressio so is separate. Developed by @robertu94 - libpressio-adios2 a adios2 bp file reader for libpressio. This is a separate package to solve a circular dependency. I've worked most with @jychoi-hpc on ADIOS2 - libpressio-errorinjector allows simulating compression error with C++ standard library random distributions. It adds additional interfaces so is separate.Developed by @robertu94 - libpressio-opt automatically configure libpressio compressors to meet quality objectives. It adds additional interfaces so is separate.Developed by @robertu94 - libpressio-rmetric - compute libpressio metrics using the R programming language in a way that embeds into other programming languages. It depends on a GPL dependency so is separate. Developed by @robertu94 - sperr and libpressio-sperr - SPERR is GPL licensed so cannot be in core libpresio which is under a BSD2 license. SPERR is an advanced compressor from the climate community. SPERR is developed by @shaomeng. @shaomeng would you like to be listed as maintainer for this? LibPressio sperr is the libpressio integration. - libpressio-tools high level tools for LibPressio like a CLI interface and a distributed compressor experiment frontend. It also provides integration for the 3rd party compressor libraries. Developed by @robertu94 - libpressio-tthresh - tthresh is a compressor based on Higher order SVD. It is GPL licensed so in a separate repository. TThresh was written by @rballester - libpressio the core libpressio library which supports C, C++, and Python and contains the bulk of the compressor plugins. Developed by @robertu94 - libstdcompat a backwards compatability shim for LibPressio. Developed by @robertu94 - mgard an advanced compressor developed by a team at ORNL as part of ECP (IIRC, @qliu21 is the maintainer of MGARD, but I also interface with @jychoi-hpc from time to time) - mgardx a variant of MGARD developed by at least in part by @lxAltria - ndzip an advanced compressor developed by @fknorr - nvcomp the last open source release of NVCOMP. Unless @NVIDIA starts providing source code again, or provides a stable public download link under an appropriate license we're kind of stuck at this version - qoz a version of SZ3 that allows stable grid sizes developed by the team of @disheng222 - qcat a quality analysis tool for binary data developed by @disheng222 - r-libpressio R bindings for libpressio in a seperate package because it depends on a GPL dependency. Developed by @robertu94 - sam2p a library that converts binary files to postscript used in libpressio's quality analysis example scripts. - sol2 a library that allows embedding lua in to C++ used in LibPressio and LibPressioOpt to allow scripting programmatic responses - sz-cpp a predecessor of SZ3 with a few different features supported by LibPressio developed by the teams of @lxAltria and @disheng222 - szauto a predecessor of SZ3 with autotuning builtin supported by LibPressio developed by the teams of and @disheng222 and @ayzk IIRC. - sz3 the next generation of the SZ compressor family developed by @ayzk and others. - szx an ultrafast variant of SZ developed by @ayzk, @xiaodong-yu, and @disheng222
spackbot-app
bot
added
conflicts
dependencies
extends
maintainers
new-package
new-variant
new-version
R
stand-alone-tests
|
Hi @robertu94! I noticed that the following package(s) don't yet have maintainers:
Are you interested in adopting any of these package(s)? If so, simply add the following to the package class: maintainers = ['robertu94']If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with $ spack blame bitgroomingzThank you for your help! Please don't add maintainers without their consent. You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer. |
|
@scheibelp You asked me to ping you when I submitted this. |
|
@spackbot style fix |
|
@spackbot fix style |
|
Let me see if I can fix that for you! |
|
I was able to run spack style --fix==> Running style checks on spack
selected: isort, mypy, black, flake8
==> Modified files
var/spack/repos/builtin/packages/adios2/package.py
var/spack/repos/builtin/packages/arc/package.py
var/spack/repos/builtin/packages/bitgroomingz/package.py
var/spack/repos/builtin/packages/cusz/package.py
var/spack/repos/builtin/packages/digitrounding/package.py
var/spack/repos/builtin/packages/fpzip/package.py
var/spack/repos/builtin/packages/libdistributed/package.py
var/spack/repos/builtin/packages/libpressio-adios2/package.py
var/spack/repos/builtin/packages/libpressio-errorinjector/package.py
var/spack/repos/builtin/packages/libpressio-frsz/package.py
var/spack/repos/builtin/packages/libpressio-opt/package.py
var/spack/repos/builtin/packages/libpressio-rmetric/package.py
var/spack/repos/builtin/packages/libpressio-sperr/package.py
var/spack/repos/builtin/packages/libpressio-tools/package.py
var/spack/repos/builtin/packages/libpressio-tthresh/package.py
var/spack/repos/builtin/packages/libpressio/package.py
var/spack/repos/builtin/packages/libstdcompat/package.py
var/spack/repos/builtin/packages/mgard/package.py
var/spack/repos/builtin/packages/mgardx/package.py
var/spack/repos/builtin/packages/ndzip/package.py
var/spack/repos/builtin/packages/nvcomp/package.py
var/spack/repos/builtin/packages/qcat/package.py
var/spack/repos/builtin/packages/qoz/package.py
var/spack/repos/builtin/packages/r-libpressio/package.py
var/spack/repos/builtin/packages/sam2p/package.py
var/spack/repos/builtin/packages/sol2/package.py
var/spack/repos/builtin/packages/sperr/package.py
var/spack/repos/builtin/packages/sz-cpp/package.py
var/spack/repos/builtin/packages/sz3/package.py
var/spack/repos/builtin/packages/szauto/package.py
var/spack/repos/builtin/packages/szx/package.py
==> Running isort checks
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/arc/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/bitgroomingz/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/cusz/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/digitrounding/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/fpzip/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libdistributed/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-adios2/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-errorinjector/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-frsz/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-opt/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-rmetric/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-sperr/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-tools/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio-tthresh/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libpressio/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/libstdcompat/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/mgard/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/mgardx/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/ndzip/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/nvcomp/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/qcat/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/qoz/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/r-libpressio/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/sam2p/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/sol2/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/sperr/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/sz-cpp/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/sz3/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/szauto/package.py
Fixing /tmp/tmpdw4k23wk/spack/var/spack/repos/builtin/packages/szx/package.py
isort checks were clean
==> Running mypy checks
Success: no issues found in 558 source files
mypy checks were clean
==> Running black checks
reformatted var/spack/repos/builtin/packages/arc/package.py
reformatted var/spack/repos/builtin/packages/bitgroomingz/package.py
reformatted var/spack/repos/builtin/packages/cusz/package.py
reformatted var/spack/repos/builtin/packages/digitrounding/package.py
reformatted var/spack/repos/builtin/packages/fpzip/package.py
reformatted var/spack/repos/builtin/packages/libdistributed/package.py
reformatted var/spack/repos/builtin/packages/adios2/package.py
reformatted var/spack/repos/builtin/packages/libpressio-adios2/package.py
reformatted var/spack/repos/builtin/packages/libpressio-errorinjector/package.py
reformatted var/spack/repos/builtin/packages/libpressio-frsz/package.py
reformatted var/spack/repos/builtin/packages/libpressio-opt/package.py
reformatted var/spack/repos/builtin/packages/libpressio-rmetric/package.py
reformatted var/spack/repos/builtin/packages/libpressio-sperr/package.py
reformatted var/spack/repos/builtin/packages/libpressio-tools/package.py
reformatted var/spack/repos/builtin/packages/libpressio-tthresh/package.py
reformatted var/spack/repos/builtin/packages/libstdcompat/package.py
reformatted var/spack/repos/builtin/packages/libpressio/package.py
reformatted var/spack/repos/builtin/packages/mgard/package.py
reformatted var/spack/repos/builtin/packages/mgardx/package.py
reformatted var/spack/repos/builtin/packages/nvcomp/package.py
reformatted var/spack/repos/builtin/packages/ndzip/package.py
reformatted var/spack/repos/builtin/packages/qcat/package.py
reformatted var/spack/repos/builtin/packages/qoz/package.py
reformatted var/spack/repos/builtin/packages/sam2p/package.py
reformatted var/spack/repos/builtin/packages/sol2/package.py
reformatted var/spack/repos/builtin/packages/r-libpressio/package.py
reformatted var/spack/repos/builtin/packages/sz-cpp/package.py
reformatted var/spack/repos/builtin/packages/sperr/package.py
reformatted var/spack/repos/builtin/packages/sz3/package.py
reformatted var/spack/repos/builtin/packages/szauto/package.py
reformatted var/spack/repos/builtin/packages/szx/package.py
All done! ✨ 🍰 ✨
31 files reformatted.
black checks were clean
==> Running flake8 checks
var/spack/repos/builtin/packages/arc/package.py:27: [E501] line too long (131 > 99 characters)
var/spack/repos/builtin/packages/libdistributed/package.py:5: [E501] line too long (107 > 99 characters)
var/spack/repos/builtin/packages/sperr/package.py:27: [E501] line too long (141 > 99 characters)
var/spack/repos/builtin/packages/szx/package.py:24: [F811] redefinition of unused 'spack.package.*' from line 6
flake8 found errors
I've updated the branch with isort fixes. |
LibPressio is a generic abstraction for the compression of data. It supports over 40 compressors and is used at over 10 institutions (that we know about). Previously these packages were part of a 3rd party [package repository](https://github.com/robertu94/spack_packages). This PR contains packages (and modifications for 2 existing packages) to expose the LibPressio support. I expect this PR to take some time to settle and finalize because of its size. Modifications: - adios2 - adds libpressio support to the existing package. - cusz the old package was not portable and lacked key dependencies this re-write is down with the knowledge of the maintainers @dingwentao and @jtian0. New Packages: - arc an data corruption tool for compressed data developed by the lab of @joncalhoun40. @joncalhoun40 would you be a maintainer for this? - bitgroomingz a standalone implementation of bitgrooming from NetCDF packaged by @disheng222. - fpzip a specialized compressor for floating point data developed by @lindstro and others and a predecessor to ZFP. - libdistributed a MPI work queue with cancelation used by libpressio and libpressio_opt since this work queue is used by multiple projects outside of libpressio so is separate. Developed by @robertu94 - libpressio-adios2 a adios2 bp file reader for libpressio. This is a separate package to solve a circular dependency. I've worked most with @jychoi-hpc on ADIOS2 - libpressio-errorinjector allows simulating compression error with C++ standard library random distributions. It adds additional interfaces so is separate.Developed by @robertu94 - libpressio-opt automatically configure libpressio compressors to meet quality objectives. It adds additional interfaces so is separate.Developed by @robertu94 - libpressio-rmetric - compute libpressio metrics using the R programming language in a way that embeds into other programming languages. It depends on a GPL dependency so is separate. Developed by @robertu94 - sperr and libpressio-sperr - SPERR is GPL licensed so cannot be in core libpresio which is under a BSD2 license. SPERR is an advanced compressor from the climate community. SPERR is developed by @shaomeng. @shaomeng would you like to be listed as maintainer for this? LibPressio sperr is the libpressio integration. - libpressio-tools high level tools for LibPressio like a CLI interface and a distributed compressor experiment frontend. It also provides integration for the 3rd party compressor libraries. Developed by @robertu94 - libpressio-tthresh - tthresh is a compressor based on Higher order SVD. It is GPL licensed so in a separate repository. TThresh was written by @rballester - libpressio the core libpressio library which supports C, C++, and Python and contains the bulk of the compressor plugins. Developed by @robertu94 - libstdcompat a backwards compatability shim for LibPressio. Developed by @robertu94 - mgard an advanced compressor developed by a team at ORNL as part of ECP (IIRC, @qliu21 is the maintainer of MGARD, but I also interface with @jychoi-hpc from time to time) - mgardx a variant of MGARD developed by at least in part by @lxAltria - ndzip an advanced compressor developed by @fknorr - nvcomp the last open source release of NVCOMP. Unless @NVIDIA starts providing source code again, or provides a stable public download link under an appropriate license we're kind of stuck at this version - qoz a version of SZ3 that allows stable grid sizes developed by the team of @disheng222 - qcat a quality analysis tool for binary data developed by @disheng222 - r-libpressio R bindings for libpressio in a seperate package because it depends on a GPL dependency. Developed by @robertu94 - sam2p a library that converts binary files to postscript used in libpressio's quality analysis example scripts. - sol2 a library that allows embedding lua in to C++ used in LibPressio and LibPressioOpt to allow scripting programmatic responses - sz-cpp a predecessor of SZ3 with a few different features supported by LibPressio developed by the teams of @lxAltria and @disheng222 - szauto a predecessor of SZ3 with autotuning builtin supported by LibPressio developed by the teams of and @disheng222 and @ayzk IIRC. - sz3 the next generation of the SZ compressor family developed by @ayzk and others. - szx an ultrafast variant of SZ developed by @ayzk, @xiaodong-yu, and @disheng222
robertu94
force-pushed
the
libpressio_merge
branch
from
a052359
to
9a76dfa
Compare
shaomeng
reviewed
Sep 13, 2022
There was a problem hiding this comment.
Hi @robertu94 , thanks for including SPERR in the newest LibPressio and I'm happy to be listed as a maintainer for SPERR. Also, I've suggested a few changes, largely because the software has evolved since we spoke last time. Please take a look and let me know what you think!
Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com>
shaomeng
reviewed
Sep 13, 2022
scheibelp
requested changes
Sep 13, 2022
There was a problem hiding this comment.
High level notes:
- There is at least one package here (
sol2) which already has a Spack package. I checked several other packages but not all of them in this PR. - Several packages retrieve source from your personal repository and I'm curious if you can pull from the repo they fork from (if you are in fact the primary maintainer of these repos, then using your forks makes sense)
- A couple of links appear to be broken (I mention this in specific comments)
var/spack/repos/builtin/packages/libpressio-errorinjector/package.py
Outdated
Show resolved
Hide resolved
|
@scheibelp I checked the other packages in the PR, and |
robertu94
force-pushed
the
libpressio_merge
branch
3 times, most recently
from
f5e9244
to
f48768a
Compare
+ new package versions + found a dependency `dlib` that didn't correspond to the same `dlib` that libpressio-opt refers to. Renamed the old package because it had way fewer users, and davisking/dlib hold the dlib name in Ubuntu, Fedora, Homebrew, Pip. Discused with @adamjstewart who added the old dlib package, and he was okay with the replacement + added missing fix for CUB relating to spack#32547 needed for cuSZ to build using Spack.
robertu94
force-pushed
the
libpressio_merge
branch
from
f48768a
to
06ca6e2
Compare
spackbot-app
bot
added
build-environment
core
+ Removed hack for R_HOME/R_ENVION in lipbressio-rmetric + Pin versions rather than rely on branches + Use upstream repositories whenever possible
|
@scheibelp I think I have completed all requested changes. Can you please take another look? Also the CI failures seems to be github related and not related to these changes specifically:
|
4 tasks
scheibelp
requested changes
Oct 14, 2022
robertu94
force-pushed
the
libpressio_merge
branch
from
9644779
to
a70411d
Compare
robertu94
force-pushed
the
libpressio_merge
branch
from
a70411d
to
1287c3a
Compare
scheibelp
previously approved these changes
Oct 14, 2022
|
@scheibelp I looked at the CI failures. I looked at many of them, and all happened in Adios2. As far as I can tell these are unrelated to the changes we made in Adios2. The exact error message is: /tmp/root/spack-stage/spack-stage-adios2-2.7.1-tbawoed3qm3m5xqq5exm6fghdnkmam3d/spack-src/source/adios2/helper/adiosMemory.cpp:32:22: error: no match for 'operator==' (operand types are 'const adios2::DataType' and 'const char [1]')
if (destType == "")
~~~~~~~~~^~~~~
make[2]: *** [source/adios2/CMakeFiles/adios2_core.dir/helper/adiosMemory.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....As far as I can tell, the files we change neither include this file or are included by this file. |
|
@robertu94 Could you try setting |
|
@robertu94 if you are able to try reverting the latest commit, we can see if this was a temporary error, but I suspect the changes to |
|
@scheibelp It builds for me. Let's go ahead and merge this now (with [runderwood@defiant ~]$ spack install adios2+libpressio ^ sz3
[+] /usr (external diffutils-3.8-cmrhmnpvzetjh4wqaveie64jxqq2ssfk)
[+] /usr (external cmake-3.22.2-yoahtgzvn7dkcx5324hjg5zuggvtlcjr)
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/lz4-1.9.3-t5j62ojcexh336hgz5qm4443s3glm5ps
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/zlib-1.2.12-itsi2i4tzpvkirehmpm4jjffkhtq6ooc
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/zstd-1.5.2-htznodvlcs5qakvtbnkl67ceg7q27iql
[+] /usr (external bison-3.8.2-7jz6xa76ansjnduissm2ent3z44oio3l)
[+] /usr (external flex-2.6.4-zff3q4r4itu2heylabncthhsheisht4s)
[+] /usr (external m4-1.4.19-sjyxxeuj7m4chyhjjkjuncqsypascr5x)
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/pkg-config-0.29.2-2sqlxuvivzcplfjzerwkcjislinhcxan
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libmd-1.0.4-kw2q33lp5noau3xh2x2hsn2qnt6sr2yp
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libiconv-1.16-opvjp3hx2hutzswdnem32battgxgoc2b
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/xz-5.2.5-q7fw4xs7gr7pkxtfaecdvvwl32wxccgl
[+] /usr (external tar-1.34-imgd4ijz4m7gmk55rb6afiu7huysfcty)
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libffi-3.4.2-kq3zs7rl7udxhaszrzvdvw7feuadb5m3
[+] /usr (external openssl-3.0.5-va446ch5fjo4qlggbp6iclpgdwhlywat)
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/gsl-2.7.1-6qxuvfmjcopnkiwiqhvfbmngfizuzjy4
[+] /usr (external findutils-4.9.0-loprc5ure6udqaj556zfw5hrva2yieff)
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/util-macros-1.19.3-brffbhnadauippxxk5pdwajmxkdjhtbs
[+] /usr (external autoconf-2.71-5nareulkxpgffxqi26eyjkzs2qjwp4yq)
[+] /usr (external automake-1.16.5-4dacek4lg433rt333kehaasumzuvthgf)
[+] /usr (external libtool-2.4.7-xlut7mvuolw7wpsa3tossu2u2vmhm6ip)
[+] /usr (external m4-1.4.19-4iyjtisldmzbmwgc6rh64r4u5csb5cvi)
[+] /usr (external tar-1.34-3433k5ltb2nzx75k6ssedprvrlwo76i5)
[+] /usr (external openssl-3.0.5-mkzafu5op32esconrswcoouhyd2bxysr)
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/bzip2-1.0.8-mf6yxiy7vp5in2szmobwn2556d4puxn6
==> Installing zfp-0.5.5-qvjmkfzezpkdlyp3ytl4zlhlfmhhtr4e
==> No binary for zfp-0.5.5-qvjmkfzezpkdlyp3ytl4zlhlfmhhtr4e found: installing from source
==> Using cached archive: /home/runderwood/git/contrib/spack/var/spack/cache/_source-cache/archive/fd/fdf7b948bab1f4e5dccfe2c2048fd98c24e417ad8fb8a51ed3463d04147393c5.tar.gz
==> No patches needed for zfp
==> zfp: Executing phase: 'cmake'
==> zfp: Executing phase: 'build'
==> zfp: Executing phase: 'install'
==> zfp: Successfully installed zfp-0.5.5-qvjmkfzezpkdlyp3ytl4zlhlfmhhtr4e
Fetch: 0.00s. Build: 3.62s. Total: 3.62s.
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/zfp-0.5.5-qvjmkfzezpkdlyp3ytl4zlhlfmhhtr4e
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/snappy-1.1.9-k2x4226qpxyplaklpgciawqfym7dfdaj
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libstdcompat-0.0.15-vnqxcovbimynryngnwgurtihy5zdiy7l
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libpng-1.6.37-ei2satrjdtmqht25aocudmpouzawd6l6
==> Installing sz-2.1.12.2-3tpk6ojpefei73i5ekdmrtn3iagrlq46
==> No binary for sz-2.1.12.2-3tpk6ojpefei73i5ekdmrtn3iagrlq46 found: installing from source
==> Using cached archive: /home/runderwood/git/contrib/spack/var/spack/cache/_source-cache/archive/42/427e263e1fed1b0a56e13e0aff8e6a19c6d78d5f35dd16856876c70ab6066dc6.tar.gz
==> No patches needed for sz
==> sz: Executing phase: 'cmake'
==> sz: Executing phase: 'build'
==> sz: Executing phase: 'install'
==> sz: Successfully installed sz-2.1.12.2-3tpk6ojpefei73i5ekdmrtn3iagrlq46
Fetch: 0.01s. Build: 13.08s. Total: 13.09s.
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/sz-2.1.12.2-3tpk6ojpefei73i5ekdmrtn3iagrlq46
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libnl-3.3.0-um6spgfa2wfugtht4uo2wgk77vpcuykh
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libtool-2.4.7-wx7hqjbqkmrj6y4742dlvwke3imm4opw
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/ncurses-6.2-4mzcgqljnvgmwgyvphshs2qbgrbjblsr
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/util-linux-uuid-2.37.4-i77ukptbd4nk3lhw7cdpz6jbsp3vncz5
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libbsd-0.11.5-ocorunpo4wvtutnru73igc234rh2zpl4
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libxml2-2.9.13-2ecdaroxqcwmvnyr36jotfpcqopwmbfg
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/sz3-3.1.5.4-axrs3ixd2etbf7ercl33nclqhhjygdc6
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/c-blosc-1.21.1-2tmwwb23neosp5xz25wvg7c7dusifjom
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libpciaccess-0.16-oullxthmgmmffibvvnehffrjbtzlsrt5
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/readline-8.1.2-2xekztcft4i3qhsjcawjl3hqfdjncoyl
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/expat-2.4.8-qkousqa4bhqqaj6k72d3rr2r3xfqnfjy
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/gettext-0.21-gopkzmfd6vgvxzqqp535utf5pozw65id
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/gettext-0.21-5i4ihvd2hwwirh54wwqis5hcuaztj4oh
==> Installing libpressio-0.88.0-oujutp6554ghll6moqyz7ljv5msuzntd
==> No binary for libpressio-0.88.0-oujutp6554ghll6moqyz7ljv5msuzntd found: installing from source
==> Using cached archive: /home/runderwood/git/contrib/spack/var/spack/cache/_source-cache/archive/43/4358441f0d10559d571327162a216617d16d09569a80e13ad286e3b7c41c5b9b.tar.gz
==> No patches needed for libpressio
==> libpressio: Executing phase: 'cmake'
==> libpressio: Executing phase: 'build'
==> libpressio: Executing phase: 'install'
==> libpressio: Successfully installed libpressio-0.88.0-oujutp6554ghll6moqyz7ljv5msuzntd
Fetch: 0.00s. Build: 5m 9.72s. Total: 5m 9.72s.
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libpressio-0.88.0-oujutp6554ghll6moqyz7ljv5msuzntd
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/hwloc-2.7.1-qlcsa55rtgouzrtwv5ttgdw2eu5geuth
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/sqlite-3.38.5-apalj6gzw3ei6ffcx3qdj5rroupcq64x
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/gdbm-1.19-opkw2gqyl3gejgw6h7vurkyvohxjzegc
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/python-3.9.13-pxf7n5fzzra6kizlwlt7urk6gbqichca
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/python-3.8.13-4cxgwmduyr33xd4uvr624f32oty6weg2
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/yaksa-0.2-rfx2vxmknpa5eu2zjmvgfzxdzt5epbez
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/py-pip-21.3.1-3uf2g56rytmvs46hwgflgvffrkzuafoc
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/py-setuptools-63.0.0-utuqdev43aiecsz6oj7op6h5m3awldxl
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/py-wheel-0.37.0-gaenfizagxzmrzao5db2s4zcfcjgqhwc
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/py-docutils-0.18.1-v2oasbc66aecps4t4el2fu33bemxkvbb
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/rdma-core-39.1-shwtocpyncyz3z46zlv7vo2w4w7jtgux
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/libfabric-1.14.1-7fxswahuf6pofnbbwjywamsz5aqbk27x
==> Installing mpich-4.0.2-w2omnlxraxv5jtxmmvyt7kd64j23vhzc
==> No binary for mpich-4.0.2-w2omnlxraxv5jtxmmvyt7kd64j23vhzc found: installing from source
==> Using cached archive: /home/runderwood/git/contrib/spack/var/spack/cache/_source-cache/archive/5a/5a42f1a889d4a2d996c26e48cbf9c595cbf4316c6814f7c181e3320d21dedd42.tar.gz
==> Using cached archive: /home/runderwood/git/contrib/spack/var/spack/cache/_source-cache/archive/d4/d4c0e99a80f6cb0cb0ced91f6ad5da776c4a70f70f805f08096939ec9a92483e
==> Applied patch https://github.com/pmodels/mpich/commit/3a1f618e017547c9710ab4fb01ae258a01477190.patch?full_index=1
==> mpich: Executing phase: 'autoreconf'
==> mpich: Executing phase: 'configure'
==> mpich: Executing phase: 'build'
==> mpich: Executing phase: 'install'
==> mpich: Successfully installed mpich-4.0.2-w2omnlxraxv5jtxmmvyt7kd64j23vhzc
Fetch: 0.06s. Build: 7m 20.49s. Total: 7m 20.56s.
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/mpich-4.0.2-w2omnlxraxv5jtxmmvyt7kd64j23vhzc
==> Installing adios2-2.8.3-m7datpc2p3xb3ilgnmst2tpgiz4rf32w
==> No binary for adios2-2.8.3-m7datpc2p3xb3ilgnmst2tpgiz4rf32w found: installing from source
==> Fetching https://github.com/ornladios/ADIOS2/archive/v2.8.3.tar.gz
==> No patches needed for adios2
==> adios2: Executing phase: 'cmake'
==> adios2: Executing phase: 'build'
==> adios2: Executing phase: 'install'
==> adios2: Successfully installed adios2-2.8.3-m7datpc2p3xb3ilgnmst2tpgiz4rf32w
Fetch: 2.01s. Build: 3m 32.73s. Total: 3m 34.74s.
[+] /home/runderwood/git/contrib/spack/opt/spack/linux-fedora36-icelake/gcc-12.1.1/adios2-2.8.3-m7datpc2p3xb3ilgnmst2tpgiz4rf32w
|
scheibelp
approved these changes
Oct 17, 2022
luke-dt
added a commit
to dantaslab/spack
that referenced
this pull request
Oct 19, 2022
* hypre 2.26.0 (spack#33299) * Github Discussions can be used for Q&A (spack#33364) * Support spackbot rebuilding all specs from source (spack#32596) Support spackbot rebuilding all specs from source when asked (with "rebuild everything") - Allow overriding --prune-dag cli opt with env var - Use job variable to optionally prevent rebuild jobs early exit behavior - ci rebuild: Use new install argument to insist deps are always installed from binary, but package is only installed from source. - gitlab: fix bug w/ untouched pruning - ci rebuild: install from hash rather than json file - When doing a "rebuild everything" pipeline, make sure that each install job only consumes binary dependencies from the mirror being populated by the current pipeline. This avoids using, e.g. binaries from develop, when rebuilding everything on a PR. - When running a pipeline to rebuild everything, do not die because we generated a hash on the broken specs list. Instead only warn in that case. - bugfix: Replace broken no-args tty.die() with sys.exit(1) * python: add 3.10.7, 3.9.14, 3.8.14, 3.7.14 (spack#32623) * Add checksum for py-secretstorage 3.3.3 (spack#33366) * Add checksum for py-ipykernel 6.15.2 (spack#33360) * GnuPG: add v2.3.8 and update stack (spack#33368) * installer.py: traverse_dependencies has local deptype (spack#33367) Currently `traverse_dependencies` fixes deptypes to traverse once and for all in the recursion, but this is incorrect, since deptypes depend on the node (e.g. if it's a dependency and cache-only, don't follow build type edges, even if the parent is build from sources and needs build deps.) * py-horovod: add v0.26 (spack#33311) * py-horovod: add v0.26 * py-petastorm: add v0.12.0 * database: don't warn adding missing build deps (spack#33361) When installing an individual spec `spack --only=package --cache-only /xyz` from a buildcache, Spack currently issues tons of warnings about missing build deps (and their deps) in the database. This PR disables these warnings, since it's fine to have a spec without its build deps in the db (they are just "missing"). * Classic Intel compilers do not support gcc-toolchain (spack#33281) * Classic Intel compilers do not support gcc-toolchain This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel compilers. AFAIK this option is only supported for Clang based compilers. This lead to an issue when installing cmake. Reproducer: ``` spack install cmake@3.24.2%intel@2021.7.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512 ``` Tagging maintainer @rscohn2 * Add `-gcc-name` for icc .. and `-gxx-name` for icpc. AFAIK this is used for modern C++ support, so we can ignore `ifort`. Co-authored-by: Stephen Sachs <stesachs@amazon.com> * py-xopen: version bump to 1.6.0 (spack#33231) * version bump to 1.6.0 * added py-isal, updated URL * Initial contribution of LibPressio ecosystem (spack#32630) * Add libpressio and dependencies; some of these packages are maintained as forks of the original repositories and in those cases the docstring mentions this. * Add optional dependency in adios2 on libpressio * cub package: set CUB_DIR environment variable for dependent installations * Clear R_HOME/R_ENVIRON before Spack installation (avoid sources outside of Spack from affecting the installation in Spack) * Rename dlib to dorian3d-dlib and update dependents; add new dlib implementation. Pending an official policy on how to handle packages with short names, reviewer unilaterally decided that the rename was acceptable given that the new Spack dlib package is referenced more widely (by orders of magnitude) than the original Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com> * New packages: libbigwig, methyldackel (spack#33273) * libbigwig: adding new package libbigwig * methyldackel: adding new package methyldackel * libbigwig: tighten up curl variant * grid: reference `fftw-api` instead of `fftw` (spack#33374) This makes it possible to compile with, e.g., `cray-fftw`, not just `fftw`. * Bugfix HIP and aluminum rocm build (spack#33344) * Fixed two bugs in the HIP package recipe. The first is that the HIP_PATH was being set to the actual spec, and not the spec prefix. The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and /opt/rocm-x.y.z/lib. This means that the external detection logic may find it in either and it turns out that some modules only expose one of those two locations. Logic is added to ensure that the internal HIP_PATH and associated ROCM_PATH are correctly set in both cases. * Added support for Aluminum to use the libfabric plugin with either RCCL or NCCL. * lz4: add 1.9.4 (spack#33334) * e4s ci stack: add trilinos +rocm (spack#31601) * py-fiona: add v1.8.22 (spack#33372) * intel-oneapi-compilers: fix Python 2.7 compliance (spack#33383) * build(deps): bump docker/setup-buildx-action from 2.1.0 to 2.2.0 (spack#33384) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@95cb08c...c74574e) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * mothur: add v1.48.0 and variants (spack#33326) * vsearch: add v2.22.1 (spack#33327) * papi: fix for Intel OneAPI compiler (spack#33225) Without this patch one hits this error trying to compiler papi with Intel OneAPI: icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization] Signed-off-by: Howard Pritchard <howardp@lanl.gov> Signed-off-by: Howard Pritchard <howardp@lanl.gov> * go,gcc: Support external go compilers for Go bootstrap (spack#27769) For ARM64, fallback to gccgo. (go-bootstrap@1.4 can't support ARM64) * Reusable --use-buildcache with better validation (spack#33388) Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Docs: Spack info option updates (spack#33376) * intel-oneapi-compilers: do not pass -Wno-unused-command-line-argument to icc + refactor (spack#33389) * spack checksum: warn if version is deprecated (spack#32438) Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Update the binary index before attempting direct fetches (spack#32137) "spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> * Relocate "run" type dependencies too (spack#33191) When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `autoconf@2.69` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <stesachs@amazon.com> * Add a command to bootstrap Spack right now (spack#33407) * axom: python only reliably available when +python, +devtools (spack#33414) * pilercr: new package (spack#33251) * new package * fixed style * actually building now * raja +rocm: use hipcc as CMAKE_CXX_COMPILER (spack#33375) Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Howard Pritchard <howardp@lanl.gov> Co-authored-by: Rui Peng Li <li50@llnl.gov> Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com> Co-authored-by: Carlos Bederián <carlos.bederian@unc.edu.ar> Co-authored-by: iarspider <iarspider@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Stephen Sachs <stephenmsachs@gmail.com> Co-authored-by: Stephen Sachs <stesachs@amazon.com> Co-authored-by: Robert Underwood <robertu94@users.noreply.github.com> Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com> Co-authored-by: snehring <7978778+snehring@users.noreply.github.com> Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com> Co-authored-by: Brian Van Essen <vanessen1@llnl.gov> Co-authored-by: Michael Kuhn <michael.kuhn@ovgu.de> Co-authored-by: eugeneswalker <38933153+eugeneswalker@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Howard Pritchard <howardp@lanl.gov> Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Robert Cohn <robert.s.cohn@intel.com> Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
luke-dt
added a commit
to dantaslab/spack
that referenced
this pull request
Oct 19, 2022
* hypre 2.26.0 (spack#33299) * Github Discussions can be used for Q&A (spack#33364) * Support spackbot rebuilding all specs from source (spack#32596) Support spackbot rebuilding all specs from source when asked (with "rebuild everything") - Allow overriding --prune-dag cli opt with env var - Use job variable to optionally prevent rebuild jobs early exit behavior - ci rebuild: Use new install argument to insist deps are always installed from binary, but package is only installed from source. - gitlab: fix bug w/ untouched pruning - ci rebuild: install from hash rather than json file - When doing a "rebuild everything" pipeline, make sure that each install job only consumes binary dependencies from the mirror being populated by the current pipeline. This avoids using, e.g. binaries from develop, when rebuilding everything on a PR. - When running a pipeline to rebuild everything, do not die because we generated a hash on the broken specs list. Instead only warn in that case. - bugfix: Replace broken no-args tty.die() with sys.exit(1) * python: add 3.10.7, 3.9.14, 3.8.14, 3.7.14 (spack#32623) * Add checksum for py-secretstorage 3.3.3 (spack#33366) * Add checksum for py-ipykernel 6.15.2 (spack#33360) * GnuPG: add v2.3.8 and update stack (spack#33368) * installer.py: traverse_dependencies has local deptype (spack#33367) Currently `traverse_dependencies` fixes deptypes to traverse once and for all in the recursion, but this is incorrect, since deptypes depend on the node (e.g. if it's a dependency and cache-only, don't follow build type edges, even if the parent is build from sources and needs build deps.) * py-horovod: add v0.26 (spack#33311) * py-horovod: add v0.26 * py-petastorm: add v0.12.0 * database: don't warn adding missing build deps (spack#33361) When installing an individual spec `spack --only=package --cache-only /xyz` from a buildcache, Spack currently issues tons of warnings about missing build deps (and their deps) in the database. This PR disables these warnings, since it's fine to have a spec without its build deps in the db (they are just "missing"). * Classic Intel compilers do not support gcc-toolchain (spack#33281) * Classic Intel compilers do not support gcc-toolchain This fix removes `--gcc-toolchain=` from the ~.fcg` files for the classic Intel compilers. AFAIK this option is only supported for Clang based compilers. This lead to an issue when installing cmake. Reproducer: ``` spack install cmake@3.24.2%intel@2021.7.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-amzn2-skylake_avx512 ``` Tagging maintainer @rscohn2 * Add `-gcc-name` for icc .. and `-gxx-name` for icpc. AFAIK this is used for modern C++ support, so we can ignore `ifort`. Co-authored-by: Stephen Sachs <stesachs@amazon.com> * py-xopen: version bump to 1.6.0 (spack#33231) * version bump to 1.6.0 * added py-isal, updated URL * Initial contribution of LibPressio ecosystem (spack#32630) * Add libpressio and dependencies; some of these packages are maintained as forks of the original repositories and in those cases the docstring mentions this. * Add optional dependency in adios2 on libpressio * cub package: set CUB_DIR environment variable for dependent installations * Clear R_HOME/R_ENVIRON before Spack installation (avoid sources outside of Spack from affecting the installation in Spack) * Rename dlib to dorian3d-dlib and update dependents; add new dlib implementation. Pending an official policy on how to handle packages with short names, reviewer unilaterally decided that the rename was acceptable given that the new Spack dlib package is referenced more widely (by orders of magnitude) than the original Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com> * New packages: libbigwig, methyldackel (spack#33273) * libbigwig: adding new package libbigwig * methyldackel: adding new package methyldackel * libbigwig: tighten up curl variant * grid: reference `fftw-api` instead of `fftw` (spack#33374) This makes it possible to compile with, e.g., `cray-fftw`, not just `fftw`. * Bugfix HIP and aluminum rocm build (spack#33344) * Fixed two bugs in the HIP package recipe. The first is that the HIP_PATH was being set to the actual spec, and not the spec prefix. The second bug is that HIP is expected to be in /opt/rocm-x.y.z/hip but it's libraries can exist at both /opt/rocm-x.y.z/hip/lib and /opt/rocm-x.y.z/lib. This means that the external detection logic may find it in either and it turns out that some modules only expose one of those two locations. Logic is added to ensure that the internal HIP_PATH and associated ROCM_PATH are correctly set in both cases. * Added support for Aluminum to use the libfabric plugin with either RCCL or NCCL. * lz4: add 1.9.4 (spack#33334) * e4s ci stack: add trilinos +rocm (spack#31601) * py-fiona: add v1.8.22 (spack#33372) * intel-oneapi-compilers: fix Python 2.7 compliance (spack#33383) * build(deps): bump docker/setup-buildx-action from 2.1.0 to 2.2.0 (spack#33384) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.1.0 to 2.2.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@95cb08c...c74574e) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * mothur: add v1.48.0 and variants (spack#33326) * vsearch: add v2.22.1 (spack#33327) * papi: fix for Intel OneAPI compiler (spack#33225) Without this patch one hits this error trying to compiler papi with Intel OneAPI: icx: error: Note that use of '-g' without any optimization-level option will turn off most compiler optimizations similar to use of '-O0' [-Werror,-Wdebug-disables-optimization] Signed-off-by: Howard Pritchard <howardp@lanl.gov> Signed-off-by: Howard Pritchard <howardp@lanl.gov> * go,gcc: Support external go compilers for Go bootstrap (spack#27769) For ARM64, fallback to gccgo. (go-bootstrap@1.4 can't support ARM64) * Reusable --use-buildcache with better validation (spack#33388) Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Docs: Spack info option updates (spack#33376) * intel-oneapi-compilers: do not pass -Wno-unused-command-line-argument to icc + refactor (spack#33389) * spack checksum: warn if version is deprecated (spack#32438) Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> * Update the binary index before attempting direct fetches (spack#32137) "spack install" will not update the binary index if given a concrete spec, which causes it to fall back to direct fetches when a simple index update would have helped. For S3 buckets in particular, this significantly and needlessly slows down the install process. This commit alters the logic so that the binary index is updated whenever a by-hash lookup fails. The lookup is attempted again with the updated index before falling back to direct fetches. To avoid updating too frequently (potentially once for each spec being installed), BinaryCacheIndex.update now includes a "cooldown" option, and when this option is enabled it will not update more than once in a cooldown window (set in config.yaml). Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> * Relocate "run" type dependencies too (spack#33191) When downloading from binary cache not only replace RPATHs to dependencies, but also text references to dependencies. Example: `autoconf@2.69` contains a text reference to the executable of its dependency `perl`: ``` $ grep perl-5 /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/autoconf-2.69-q3lo/bin/autoreconf eval 'case $# in 0) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0";; *) exec /shared/spack/opt/spack/linux-amzn2-x86_64_v3/gcc-7.3.1/perl-5.34.1-yphg/bin/perl -S "$0" "$@";; esac' ``` These references need to be replace or any package using `autoreconf` will fail as it cannot find the installed `perl`. Co-authored-by: Stephen Sachs <stesachs@amazon.com> * Add a command to bootstrap Spack right now (spack#33407) * axom: python only reliably available when +python, +devtools (spack#33414) * pilercr: new package (spack#33251) * new package * fixed style * actually building now * raja +rocm: use hipcc as CMAKE_CXX_COMPILER (spack#33375) Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Howard Pritchard <howardp@lanl.gov> Co-authored-by: Rui Peng Li <li50@llnl.gov> Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com> Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com> Co-authored-by: Carlos Bederián <carlos.bederian@unc.edu.ar> Co-authored-by: iarspider <iarspider@gmail.com> Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com> Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com> Co-authored-by: Stephen Sachs <stephenmsachs@gmail.com> Co-authored-by: Stephen Sachs <stesachs@amazon.com> Co-authored-by: Robert Underwood <robertu94@users.noreply.github.com> Co-authored-by: Samuel Li <shaomeng@users.noreply.github.com> Co-authored-by: snehring <7978778+snehring@users.noreply.github.com> Co-authored-by: Mosè Giordano <giordano@users.noreply.github.com> Co-authored-by: Brian Van Essen <vanessen1@llnl.gov> Co-authored-by: Michael Kuhn <michael.kuhn@ovgu.de> Co-authored-by: eugeneswalker <38933153+eugeneswalker@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Howard Pritchard <howardp@lanl.gov> Co-authored-by: Bernhard Kaindl <43588962+bernhardkaindl@users.noreply.github.com> Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com> Co-authored-by: Robert Cohn <robert.s.cohn@intel.com> Co-authored-by: Jonathon Anderson <17242663+blue42u@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LibPressio is a generic abstraction for the compression of data. It supports over 40 compressors and is used at over 10 institutions (that we know about). Previously these packages were part of a 3rd party package repository. This PR contains packages (and modifications for 2 existing packages) to expose the LibPressio support.
I expect this PR to take some time to settle and finalize because of its size.
Modifications:
New Packages:
meet quality objectives. It adds additional interfaces so is
separate.Developed by @robertu94
programming language in a way that embeds into other programming
languages. It depends on a GPL dependency so is separate.
Developed by @robertu94
libpresio which is under a BSD2 license. SPERR is an advanced
compressor from the climate community. SPERR is developed by
@shaomeng. @shaomeng would you like to be listed as maintainer for
this? LibPressio sperr is the libpressio integration.
and a distributed compressor experiment frontend. It also provides
integration for the 3rd party compressor libraries. Developed by @robertu94
SVD. It is GPL licensed so in a separate repository. TThresh was
written by @rballester
Python and contains the bulk of the compressor plugins. Developed by
@robertu94
by @robertu94
ECP (IIRC, @qliu21 is the maintainer of MGARD, but I also interface
with @jychoi-hpc from time to time)
providing source code again, or provides a stable public download link
under an appropriate license we're kind of stuck at this version
team of @disheng222
it depends on a GPL dependency. Developed by @robertu94
libpressio's quality analysis example scripts.
and LibPressioOpt to allow scripting programmatic responses
LibPressio developed by the teams of @lxAltria and @disheng222
LibPressio developed by the teams of and @disheng222 and @ayzk IIRC.
and others.
@disheng222