[WIP] Add support for binary package creation and install #445

hegner · 2016-02-18T21:27:09Z

This work-in-progress pull request adds support for binary packages. It enables the creation of tarballs of build artifacts via e.g.

spack create_tarball libelf

A tarball created that way can be downloaded and installed via

export SPACK_DOWNLOAD_URL=http://your.own.webserver
spack install --install-policy download libelf

At the moment it relies on the dag hash and a platform naming convention to guarantee compatbility between build and install environment. As build and install location may however be different, it contains a placeholder for applying relocation commands.

Consider the current state of the code as a proof-of-principle. I however wanted to share my activities early in the process.

…ackages

Add recursion and force-overwrite to create-tarball.

Fix dependency recursion.

o Add lazy policy for install o Use Stage class for download o Add meta-data o Improve logging

tgamblin · 2016-03-07T11:09:04Z

@hegner: Sorry I haven't had a chance to look at this in detail yet but in general having a mechanism like this to host Spack binary packaging would be awesome. We talked about this a bit at the last week's telcon.

High level stuff:

Rather than setting an environment variable, could binary tarballs be hosted in a mirror? That would allow users to set up source and binary mirrors in the same place. If a mirror only has source or only has binaries, then Spack will happily check the next mirror in the mirror list for a download. Maybe we could mess with the mirror directory layout to have two subdirectories -- one for sources and one for binaries.
It looks like the fetching for binaries is in do_install -- why not put that in do_fetch? The fetch step could decide if what has been downloaded is binary, then install can decide whether it needs to do a binary install or a build. That would make the results of spack stage and spack fetch a little less surprising, too.
I think you'll like the OS stuff we're near merging -- We'll be including a platform, os revision, and target identifier in the spec, so it should be easier to figure out how to lay out the binary mirror, and to ensure consistent hashing across sites. Another thing we're going to add is hashing package.py files.
How are you thinking of doing relocations? patchelf is one way, but we'd need to either a) build with padded RPATHs, or b) start using relative RPATHs ($ORIGIN) to make it work. a) is easy enough in the compiler wrappers. b) is a bit harder as I would have to think about how to handle binaries installed at arbitrary depths within the prefix. A more serious issue is whether the package itself encodes absolute paths in scripts and/or binaries/symbols/whatever. Many do, and I find it pretty hard to keep up with. Do you think packages should be labeled with a relocatable = True property so that downloaders know whether to try to relocate them?
(minor) Can the create_tarball command be renamed to create-tarball? I'm trying to stick to dashes in commands.

Thanks for the PR! I think submitting WIP is really useful.

hegner · 2016-03-07T12:26:36Z

@tgamblin - I am happy you find the useful. Let me answer to these points:

Sure. That would be the best option. The env-variable was there to get things off-the-ground for the proof-of-principle.
I agree that's the much better approach. Point 1) is a prerequisite to get that going.
Great! Not sure what you discussed/where to make comments, but I'd have a few proposals/considerations/questions to add. In particular about ABI-changing variants. And their handling.
Is non trivial. At the moment I store the origin as to know what to replace. And that's not only limited to binaries, but to scripts as well. So using $ORIGIN in RPATH is as you say not enough. And in fact, we'd need to move to using RUNPATH as well. If we ever want to be able/have to override things manually. Due to the lookup precedence. Marking packages as relocatable by the package maintainer would indeed be nice. Would however require some helper tool to check that. I think it should be feasible to add this as a check to even the packaging step. Would need some more time to think through various possible policies.
Will do.

citibeth · 2016-04-01T20:45:31Z

This needs documentation added to the user manual (.rst files).

gartung · 2016-04-26T17:40:30Z

lib/spack/spack/binary_distribution.py

+
+    # handle meta-data
+    cp = which("cp", required = True)
+    spec_file = os.path.join(spec.prefix,".spack","spec.yaml")


@tgamblin I am trying to use this pull request with the current llnl/develop but the path generated by spec.prefix does not match the actual install path. What is the correct path to use to find an installed package?

spack create-tarball geant4
/usr/bin/cp: cannot stat ‘/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn/.spack/spec.yaml’: No such file or directory
==> Error: Command exited with status 1:
/usr/bin/cp /home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn/.spack/spec.yaml ./redhat7-x86_64-geant4-10.01.p03-ikfpnwhvd2qmqm3lqnqe25blwwaxgegn.tar.gz.yaml

ls -d /home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4*
/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3/geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7

@gartung - I just updated https://github.com/hegner/spack/tree/devel with a fresh merge of llvm/develop and this PR. Please try that. After automatic merging I've seen that git created a package.py that had duplication of operations, which may have hit you.
Eventually I will update this PR as well, but this I'd like to do w/ a cleaner git history.

@hegner Thanks. It works for the geant4 package I built today but not for packages that were built last. I think has to do with a change in package hashes:

spack create_tarball gcc@4.9.3
/usr/bin/tar: gcc-4.9.3-pewb6tme5g5rrkyqlnnirlljpzasefln: Cannot stat: No such file or directory
/usr/bin/tar: Exiting with failure status due to previous errors
==> Error: Command exited with status 2:
/usr/bin/tar --directory=/home/gartung/spack/opt/spack/linux-x86_64/gcc-4.9.3 -cvf redhat7-x86_64-gcc-pewb6tme5g5rrkyqlnnirlljpzasefln.tar.gz gcc-4.9.3-pewb6tme5g5rrkyqlnnirlljpzasefln

spack create_tarball geant4
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4graphics_reps.so
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4visXXX.so
geant4-10.01.p03-diz5zpnvbv3rovdetwqq2dnf4ck3qmz7/lib64/libG4track.so
....

Sometimes it works, sometimes it doesn't. I think it might have to do with how the package.py file is hashed. For instance tcl/package.py is simple, but root/package.py has conditionals in it.

spack create_tarball root%gcc@4.9.3
tar: root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
==> Error: Command exited with status 1:
/usr/bin/tar --directory=/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3 -cvf macos1010-x86_64-root-kqcdv2o55aal3jpzhzniqh3si3tdkpj2.tar.gz root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2

mac-127459:gcc-4.9.3 gartung$ spack create_tarball tcl%gcc@4.9.3
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6/.spack
a tcl-8.6.5-gxongf7lt5hnuvgf2llbqng5qkiqchv6/bin

Whatever "spack location -i root%gcc" does finds the correct path.

spack create_tarball root%gcc
tar: root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors.
==> Error: Command exited with status 1:
/usr/bin/tar --directory=/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3 -cvf macos1010-x86_64-root-kqcdv2o55aal3jpzhzniqh3si3tdkpj2.tar.gz root-6.07.02-kqcdv2o55aal3jpzhzniqh3si3tdkpj2

mac-127459:.spack gartung$ spack location -i root%gcc
/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
mac-127459:.spack gartung$ ls -d /Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5
/Users/gartung/spack/opt/spack/darwin-x86_64/gcc-4.9.3/root-6.07.02-pgd2qeki6kj7wrantg5dw52gb3i63te5

hegner#4 should fix this.

gartung · 2016-04-29T18:22:15Z

@hegner I was able to verify on my SL7 desktop that I can create binary tarballs in one spack install tree and install them in another spack install tree. I did have to rebuild a couple of packages because the has was different, probably because of changes in upstream. One curious behavior that needs to be addressed is that spack downloads the source tarball as if it were doing a real build and then completes by expanding the tarfile.

gartung · 2016-04-29T19:28:03Z

There was a problem building patchelf when I used the compiler built with spack on another machine.

citibeth · 2016-05-02T03:38:47Z

Was the compiler built by Spack on another machine in the same absolute
path location, or a different one?

On Fri, Apr 29, 2016 at 3:28 PM, Patrick Gartung notifications@github.com
wrote:

There was a problem building patchelf when I used the compiler built with
spack on another machine.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#445 (comment)

gartung · 2016-05-02T08:19:48Z

This is in reference to a compiler built with spack on one machine and then relocated to a different path on another machine. When I relocate to another path on the same machine, the original path is still available and the rpaths referring to the original were valid. This is something I hope to work out with Benedikt during the HSF workshop spack hackathon.

…ackages

…ation of some code

…ackages

gartung · 2016-05-13T18:05:35Z

So the gcc compiler is relocatable if built with the binutils and gold variant turned off. With the binutils variant turned on the gcc configure flags --with-sysroot=/, --with-ld=abs path, --with-as=abs path are used. The relocated gcc binaries were configured for ld and as the didn't exist in the original paths after relocation. By default, gcc looked for ld and as in $PATH, namely /usr/bin/ld which does not support --with-sysroot. I could work around by loading the environment module for binutils but decided to turn off the bintuls and gold variants for gcc.

hegner · 2016-05-15T19:09:13Z

@tgamblin @alalazo - any idea how to get coveralis not getting confused by platform specific execution branches?

tgamblin · 2016-05-16T07:06:23Z

@hegner: I wouldn't worry too much about the coveralls check. It's useful but not mandatory for merge.

hegner · 2016-05-16T07:57:54Z

@tgamblin - nice to have them useful though.

gartung · 2016-05-16T21:47:03Z

lib/spack/spack/relocate.py

+                    last_cmd = rhs
+                if lhs == 'path' and last_cmd == 'LC_RPATH':
+                    path.add(rhs)
+            return path


@hegner This should line up with for on line 22.

gartung · 2017-08-16T18:25:58Z

@adamjstewart This can be closed now. The work was merged in my branch.

…from_spack_dev_20240628 Update ESMF and MAPL from spack develop as of 2024/06/28

hegner and others added 12 commits February 17, 2016 02:23

add create_tarball command to package build artifacts into tarballs

e67b7eb

move to non-CERN specific naming conventions

84de4be

first test version of binary install procedures

9c67751

remove debug printout

c053647

add version to tarball name

d31ba16

Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…

2ef4768

…ackages

Add recursion and force-overwrite to create-tarball.

90fcba0

Merge pull request #1 from brettviren/features/binary-packages

95fe15a

Add recursion and force-overwrite to create-tarball.

Fix dependency recursion.

860e5df

Merge pull request #2 from brettviren/features/binary-packages

7719192

Fix dependency recursion.

Extend binary packaging features

4498737

o Add lazy policy for install o Use Stage class for download o Add meta-data o Improve logging

remove accidentally added files

e7773ce

gartung reviewed Apr 26, 2016
View reviewed changes

hegner and others added 10 commits May 7, 2016 10:42

Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…

4c8c82a

…ackages

remove duplicate creation of install layout

8324fcf

use mirrors for downloading binary tarballs

3210dd2

use full version of linux distribution

5fb14db

remove duplicated fetch; remnant from last merge which lead to duplic…

8446096

…ation of some code

add relocation for shebang and config files

80dbd10

reduce verbosity

6bad462

refine relocation

bfa5f2c

fix wrong ident

5911b0b

Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…

779b8dc

…ackages

hegner and others added 6 commits May 12, 2016 11:30

address flake8 checks

1f9a01d

quote pathnames in case there are spaces

a476bd7

fix compression flag

132993d

making this work on OSX

a506bf9

clean up mac relocation

ebd7fe9

code cleanup

4282850

hegner added 2 commits May 15, 2016 20:23

refactoring; documentation; testing

f9cd9a2

pep8

66a812d

gartung reviewed May 16, 2016
View reviewed changes

various fixes

0607152

gartung mentioned this pull request Jun 1, 2016

WIP: Feature: creation of relocatable tarballs from installed packages #1013

Closed

gartung mentioned this pull request Sep 9, 2016

Create rpms and update rpms in a DB #1154

Closed

tgamblin added the WIP label Sep 21, 2016

davydden added the feature label Sep 21, 2016

tgamblin mentioned this pull request Oct 17, 2016

[xSDK] Trilinos package review. #2001

Closed

tgamblin mentioned this pull request Nov 4, 2016

Discussion on Changing Hashes #2247

Closed

tgamblin mentioned this pull request Jan 19, 2017

Losing Users: Perception that Spack is Too Complicated #2873

Closed

tgamblin mentioned this pull request Jan 27, 2017

RPATH with Python and virtualenv easybuilders/easybuild-framework#2076

Open

trws mentioned this pull request Mar 27, 2017

test spack package on release flux-framework/flux-core#920

Closed

scheibelp mentioned this pull request Apr 1, 2017

Allow installation directory layout to be configured #2789

Merged

mathstuf mentioned this pull request May 3, 2017

gpg: add a gpg subcommand #3845

Merged

2 tasks

gartung mentioned this pull request Jul 21, 2017

Create, install and relocate tarballs of installed packages. #4854

Merged

adamjstewart closed this Aug 16, 2017

climbfuji added a commit to climbfuji/spack that referenced this pull request Jul 3, 2024

Merge pull request spack#445 from climbfuji/feature/update_esmf_mapl_…

831c093

…from_spack_dev_20240628 Update ESMF and MAPL from spack develop as of 2024/06/28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add support for binary package creation and install #445

[WIP] Add support for binary package creation and install #445

hegner commented Feb 18, 2016

tgamblin commented Mar 7, 2016

hegner commented Mar 7, 2016

citibeth commented Apr 1, 2016

gartung Apr 26, 2016 •

edited

Loading

hegner Apr 26, 2016

gartung Apr 26, 2016

gartung Apr 26, 2016

gartung Apr 27, 2016

gartung Apr 27, 2016

gartung commented Apr 29, 2016

gartung commented Apr 29, 2016

citibeth commented May 2, 2016

gartung commented May 2, 2016 •

edited

Loading

gartung commented May 13, 2016

hegner commented May 15, 2016

tgamblin commented May 16, 2016

hegner commented May 16, 2016

gartung May 16, 2016

gartung commented Aug 16, 2017

[WIP] Add support for binary package creation and install #445

[WIP] Add support for binary package creation and install #445

Conversation

hegner commented Feb 18, 2016

tgamblin commented Mar 7, 2016

hegner commented Mar 7, 2016

citibeth commented Apr 1, 2016

gartung Apr 26, 2016 • edited Loading

Choose a reason for hiding this comment

hegner Apr 26, 2016

Choose a reason for hiding this comment

gartung Apr 26, 2016

Choose a reason for hiding this comment

gartung Apr 26, 2016

Choose a reason for hiding this comment

gartung Apr 27, 2016

Choose a reason for hiding this comment

gartung Apr 27, 2016

Choose a reason for hiding this comment

gartung commented Apr 29, 2016

gartung commented Apr 29, 2016

citibeth commented May 2, 2016

gartung commented May 2, 2016 • edited Loading

gartung commented May 13, 2016

hegner commented May 15, 2016

tgamblin commented May 16, 2016

hegner commented May 16, 2016

gartung May 16, 2016

Choose a reason for hiding this comment

gartung commented Aug 16, 2017

gartung Apr 26, 2016 •

edited

Loading

gartung commented May 2, 2016 •

edited

Loading