-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Add support for binary package creation and install #445
Conversation
Add recursion and force-overwrite to create-tarball.
Fix dependency recursion.
o Add lazy policy for install o Use Stage class for download o Add meta-data o Improve logging
@hegner: Sorry I haven't had a chance to look at this in detail yet but in general having a mechanism like this to host Spack binary packaging would be awesome. We talked about this a bit at the last week's telcon. High level stuff:
Thanks for the PR! I think submitting WIP is really useful. |
@tgamblin - I am happy you find the useful. Let me answer to these points:
|
This needs documentation added to the user manual (.rst files). |
|
||
# handle meta-data | ||
cp = which("cp", required = True) | ||
spec_file = os.path.join(spec.prefix,".spack","spec.yaml") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tgamblin I am trying to use this pull request with the current llnl/develop but the path generated by spec.prefix does not match the actual install path. What is the correct path to use to find an installed package?
spack create-tarball geant4
/usr/bin/cp: cannot stat ‘/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn/.spack/spec.yaml’: No such file or directory
==> Error: Command exited with status 1:
/usr/bin/cp /home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn/.spack/spec.yaml ./redhat7-x86_64-geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn.tar.gz.yaml
ls -d /home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4*
/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gartung - I just updated https://github.com/hegner/spack/tree/devel with a fresh merge of llvm/develop and this PR. Please try that. After automatic merging I've seen that git created a package.py that had duplication of operations, which may have hit you.
Eventually I will update this PR as well, but this I'd like to do w/ a cleaner git history.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hegner Thanks. It works for the geant4 package I built today but not for packages that were built last. I think has to do with a change in package hashes:
spack create_tarball gcc@4.9.3
/usr/bin/tar: gcc-4.9.3-pewb6tme5g5rrkyqlnnirlljpzasefln: Cannot stat: No such file or directory
/usr/bin/tar: Exiting with failure status due to previous errors
==> Error: Command exited with status 2:
/usr/bin/tar --directory=/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3 -cvf redhat7-x86_64-gcc-pewb6tme5g5rrkyqlnnirlljpzasefln.tar.gz gcc-4.9.3-pewb6tme5g5rrkyqlnnirlljpzasefln
spack create_tarball geant4
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4graphics_reps.so
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4visXXX.so
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4track.so
....
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sometimes it works, sometimes it doesn't. I think it might have to do with how the package.py file is hashed. For instance tcl/package.py is simple, but root/package.py has conditionals in it.
spack create_tarball root%gcc@4.9.3
tar: root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
==> Error: Command exited with status 1:
/usr/bin/tar --directory=/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3 -cvf macos1010-x86_64-root-kqcdv2o55aal3jpzhzniqh3si3tdkpj2.tar.gz root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2
mac-127459:gcc-4.9.3 gartung$ spack create_tarball tcl%gcc@4.9.3
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6/.spack
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6/bin
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whatever "spack location -i root%gcc" does finds the correct path.
spack create_tarball root%gcc
tar: root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
==> Error: Command exited with status 1:
/usr/bin/tar --directory=/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3 -cvf macos1010-x86_64-root-kqcdv2o55aal3jpzhzniqh3si3tdkpj2.tar.gz root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2
mac-127459:.spack gartung$ spack location -i root%gcc
/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
mac-127459:.spack gartung$ ls -d /Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hegner#4 should fix this.
@hegner I was able to verify on my SL7 desktop that I can create binary tarballs in one spack install tree and install them in another spack install tree. I did have to rebuild a couple of packages because the has was different, probably because of changes in upstream. One curious behavior that needs to be addressed is that spack downloads the source tarball as if it were doing a real build and then completes by expanding the tarfile. |
There was a problem building patchelf when I used the compiler built with spack on another machine. |
Was the compiler built by Spack on another machine in the same absolute On Fri, Apr 29, 2016 at 3:28 PM, Patrick Gartung notifications@github.com
|
This is in reference to a compiler built with spack on one machine and then relocated to a different path on another machine. When I relocate to another path on the same machine, the original path is still available and the rpaths referring to the original were valid. This is something I hope to work out with Benedikt during the HSF workshop spack hackathon. |
…ation of some code
So the gcc compiler is relocatable if built with the binutils and gold variant turned off. With the binutils variant turned on the gcc configure flags --with-sysroot=/, --with-ld=abs path, --with-as=abs path are used. The relocated gcc binaries were configured for ld and as the didn't exist in the original paths after relocation. By default, gcc looked for ld and as in $PATH, namely /usr/bin/ld which does not support --with-sysroot. I could work around by loading the environment module for binutils but decided to turn off the bintuls and gold variants for gcc. |
@hegner: I wouldn't worry too much about the coveralls check. It's useful but not mandatory for merge. |
@tgamblin - nice to have them useful though. |
lib/spack/spack/relocate.py
Outdated
last_cmd = rhs | ||
if lhs == 'path' and last_cmd == 'LC_RPATH': | ||
path.add(rhs) | ||
return path |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hegner This should line up with for on line 22.
@adamjstewart This can be closed now. The work was merged in my branch. |
…from_spack_dev_20240628 Update ESMF and MAPL from spack develop as of 2024/06/28
This work-in-progress pull request adds support for binary packages. It enables the creation of tarballs of build artifacts via e.g.
A tarball created that way can be downloaded and installed via
At the moment it relies on the dag hash and a platform naming convention to guarantee compatbility between build and install environment. As build and install location may however be different, it contains a placeholder for applying relocation commands.
Consider the current state of the code as a proof-of-principle. I however wanted to share my activities early in the process.