Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add support for binary package creation and install #445

Closed
wants to merge 31 commits into from

Conversation

hegner
Copy link
Contributor

@hegner hegner commented Feb 18, 2016

This work-in-progress pull request adds support for binary packages. It enables the creation of tarballs of build artifacts via e.g.

spack create_tarball libelf

A tarball created that way can be downloaded and installed via

export SPACK_DOWNLOAD_URL=http://your.own.webserver
spack install --install-policy download libelf

At the moment it relies on the dag hash and a platform naming convention to guarantee compatbility between build and install environment. As build and install location may however be different, it contains a placeholder for applying relocation commands.

Consider the current state of the code as a proof-of-principle. I however wanted to share my activities early in the process.

@tgamblin
Copy link
Member

tgamblin commented Mar 7, 2016

@hegner: Sorry I haven't had a chance to look at this in detail yet but in general having a mechanism like this to host Spack binary packaging would be awesome. We talked about this a bit at the last week's telcon.

High level stuff:

  1. Rather than setting an environment variable, could binary tarballs be hosted in a mirror? That would allow users to set up source and binary mirrors in the same place. If a mirror only has source or only has binaries, then Spack will happily check the next mirror in the mirror list for a download. Maybe we could mess with the mirror directory layout to have two subdirectories -- one for sources and one for binaries.
  2. It looks like the fetching for binaries is in do_install -- why not put that in do_fetch? The fetch step could decide if what has been downloaded is binary, then install can decide whether it needs to do a binary install or a build. That would make the results of spack stage and spack fetch a little less surprising, too.
  3. I think you'll like the OS stuff we're near merging -- We'll be including a platform, os revision, and target identifier in the spec, so it should be easier to figure out how to lay out the binary mirror, and to ensure consistent hashing across sites. Another thing we're going to add is hashing package.py files.
  4. How are you thinking of doing relocations? patchelf is one way, but we'd need to either a) build with padded RPATHs, or b) start using relative RPATHs ($ORIGIN) to make it work. a) is easy enough in the compiler wrappers. b) is a bit harder as I would have to think about how to handle binaries installed at arbitrary depths within the prefix. A more serious issue is whether the package itself encodes absolute paths in scripts and/or binaries/symbols/whatever. Many do, and I find it pretty hard to keep up with. Do you think packages should be labeled with a relocatable = True property so that downloaders know whether to try to relocate them?
  5. (minor) Can the create_tarball command be renamed to create-tarball? I'm trying to stick to dashes in commands.

Thanks for the PR! I think submitting WIP is really useful.

@hegner
Copy link
Contributor Author

hegner commented Mar 7, 2016

@tgamblin - I am happy you find the useful. Let me answer to these points:

  1. Sure. That would be the best option. The env-variable was there to get things off-the-ground for the proof-of-principle.
  2. I agree that's the much better approach. Point 1) is a prerequisite to get that going.
  3. Great! Not sure what you discussed/where to make comments, but I'd have a few proposals/considerations/questions to add. In particular about ABI-changing variants. And their handling.
  4. Is non trivial. At the moment I store the origin as to know what to replace. And that's not only limited to binaries, but to scripts as well. So using $ORIGIN in RPATH is as you say not enough. And in fact, we'd need to move to using RUNPATH as well. If we ever want to be able/have to override things manually. Due to the lookup precedence. Marking packages as relocatable by the package maintainer would indeed be nice. Would however require some helper tool to check that. I think it should be feasible to add this as a check to even the packaging step. Would need some more time to think through various possible policies.
  5. Will do.

@citibeth
Copy link
Member

citibeth commented Apr 1, 2016

This needs documentation added to the user manual (.rst files).


# handle meta-data
cp = which("cp", required = True)
spec_file = os.path.join(spec.prefix,".spack","spec.yaml")
Copy link
Member

@gartung gartung Apr 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tgamblin I am trying to use this pull request with the current llnl/develop but the path generated by spec.prefix does not match the actual install path. What is the correct path to use to find an installed package?

spack create-tarball geant4
/usr/bin/cp: cannot stat ‘/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn/.spack/spec.yaml’: No such file or directory
==> Error: Command exited with status 1:
/usr/bin/cp /home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn/.spack/spec.yaml ./redhat7-x86_64-geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn.tar.gz.yaml

ls -d /home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4*
/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gartung - I just updated https://github.com/hegner/spack/tree/devel with a fresh merge of llvm/develop and this PR. Please try that. After automatic merging I've seen that git created a package.py that had duplication of operations, which may have hit you.
Eventually I will update this PR as well, but this I'd like to do w/ a cleaner git history.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hegner Thanks. It works for the geant4 package I built today but not for packages that were built last. I think has to do with a change in package hashes:

spack create_tarball gcc@4.9.3
/usr/bin/tar: gcc-4.9.3-pewb6tme5g5rrkyqlnnirlljpzasefln: Cannot stat: No such file or directory
/usr/bin/tar: Exiting with failure status due to previous errors
==> Error: Command exited with status 2:
/usr/bin/tar --directory=/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3 -cvf redhat7-x86_64-gcc-pewb6tme5g5rrkyqlnnirlljpzasefln.tar.gz gcc-4.9.3-pewb6tme5g5rrkyqlnnirlljpzasefln

spack create_tarball geant4
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4graphics_reps.so
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4visXXX.so
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4track.so
....

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes it works, sometimes it doesn't. I think it might have to do with how the package.py file is hashed. For instance tcl/package.py is simple, but root/package.py has conditionals in it.

spack create_tarball root%gcc@4.9.3
tar: root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
==> Error: Command exited with status 1:
/usr/bin/tar --directory=/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3 -cvf macos1010-x86_64-root-kqcdv2o55aal3jpzhzniqh3si3tdkpj2.tar.gz root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2

mac-127459:gcc-4.9.3 gartung$ spack create_tarball tcl%gcc@4.9.3
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6/.spack
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6/bin

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whatever "spack location -i root%gcc" does finds the correct path.

spack create_tarball root%gcc
tar: root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
==> Error: Command exited with status 1:
/usr/bin/tar --directory=/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3 -cvf macos1010-x86_64-root-kqcdv2o55aal3jpzhzniqh3si3tdkpj2.tar.gz root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2

mac-127459:.spack gartung$ spack location -i root%gcc
/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
mac-127459:.spack gartung$ ls -d /Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hegner#4 should fix this.

@gartung
Copy link
Member

gartung commented Apr 29, 2016

@hegner I was able to verify on my SL7 desktop that I can create binary tarballs in one spack install tree and install them in another spack install tree. I did have to rebuild a couple of packages because the has was different, probably because of changes in upstream. One curious behavior that needs to be addressed is that spack downloads the source tarball as if it were doing a real build and then completes by expanding the tarfile.

@gartung
Copy link
Member

gartung commented Apr 29, 2016

There was a problem building patchelf when I used the compiler built with spack on another machine.

@citibeth
Copy link
Member

citibeth commented May 2, 2016

Was the compiler built by Spack on another machine in the same absolute
path location, or a different one?

On Fri, Apr 29, 2016 at 3:28 PM, Patrick Gartung notifications@github.com
wrote:

There was a problem building patchelf when I used the compiler built with
spack on another machine.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#445 (comment)

@gartung
Copy link
Member

gartung commented May 2, 2016

This is in reference to a compiler built with spack on one machine and then relocated to a different path on another machine. When I relocate to another path on the same machine, the original path is still available and the rpaths referring to the original were valid. This is something I hope to work out with Benedikt during the HSF workshop spack hackathon.

@gartung
Copy link
Member

gartung commented May 13, 2016

So the gcc compiler is relocatable if built with the binutils and gold variant turned off. With the binutils variant turned on the gcc configure flags --with-sysroot=/, --with-ld=abs path, --with-as=abs path are used. The relocated gcc binaries were configured for ld and as the didn't exist in the original paths after relocation. By default, gcc looked for ld and as in $PATH, namely /usr/bin/ld which does not support --with-sysroot. I could work around by loading the environment module for binutils but decided to turn off the bintuls and gold variants for gcc.

@hegner
Copy link
Contributor Author

hegner commented May 15, 2016

@tgamblin @alalazo - any idea how to get coveralis not getting confused by platform specific execution branches?

@tgamblin
Copy link
Member

@hegner: I wouldn't worry too much about the coveralls check. It's useful but not mandatory for merge.

@hegner
Copy link
Contributor Author

hegner commented May 16, 2016

@tgamblin - nice to have them useful though.

last_cmd = rhs
if lhs == 'path' and last_cmd == 'LC_RPATH':
path.add(rhs)
return path
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hegner This should line up with for on line 22.

@gartung
Copy link
Member

gartung commented Aug 16, 2017

@adamjstewart This can be closed now. The work was merged in my branch.

climbfuji added a commit to climbfuji/spack that referenced this pull request Jul 3, 2024
…from_spack_dev_20240628

Update ESMF and MAPL from spack develop as of 2024/06/28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants