Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Feature: creation of relocatable tarballs from installed packages #1013

Closed
wants to merge 193 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
193 commits
Select commit Hold shift + click to select a range
e67b7eb
add create_tarball command to package build artifacts into tarballs
hegner Feb 17, 2016
84de4be
move to non-CERN specific naming conventions
hegner Feb 17, 2016
9c67751
first test version of binary install procedures
hegner Feb 17, 2016
c053647
remove debug printout
hegner Feb 17, 2016
d31ba16
add version to tarball name
hegner Feb 18, 2016
2ef4768
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
hegner Feb 18, 2016
90fcba0
Add recursion and force-overwrite to create-tarball.
brettviren Feb 19, 2016
95fe15a
Merge pull request #1 from brettviren/features/binary-packages
hegner Feb 19, 2016
860e5df
Fix dependency recursion.
brettviren Feb 19, 2016
7719192
Merge pull request #2 from brettviren/features/binary-packages
hegner Feb 19, 2016
4498737
Extend binary packaging features
hegner Feb 20, 2016
e7773ce
remove accidentally added files
hegner Feb 20, 2016
4c8c82a
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
hegner May 7, 2016
8324fcf
remove duplicate creation of install layout
hegner May 7, 2016
3210dd2
use mirrors for downloading binary tarballs
hegner May 7, 2016
5fb14db
use full version of linux distribution
hegner May 7, 2016
8446096
remove duplicated fetch; remnant from last merge which lead to duplic…
hegner May 8, 2016
80dbd10
add relocation for shebang and config files
hegner May 8, 2016
6bad462
reduce verbosity
hegner May 9, 2016
bfa5f2c
refine relocation
hegner May 9, 2016
5911b0b
fix wrong ident
hegner May 9, 2016
779b8dc
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
hegner May 12, 2016
1f9a01d
address flake8 checks
hegner May 12, 2016
a476bd7
quote pathnames in case there are spaces
gartung May 11, 2016
132993d
fix compression flag
hegner May 12, 2016
a506bf9
making this work on OSX
gartung May 11, 2016
ebd7fe9
clean up mac relocation
hegner May 12, 2016
4282850
code cleanup
hegner May 12, 2016
f9cd9a2
refactoring; documentation; testing
hegner May 15, 2016
66a812d
pep8
hegner May 15, 2016
0607152
various fixes
hegner May 17, 2016
a3783c1
use only otool and install_name_tool
gartung May 17, 2016
5f85533
fixes
gartung May 17, 2016
6e1257e
fixes #967
alalazo May 18, 2016
164da8e
Version bump to 0.9.1
tgamblin May 18, 2016
987c549
using @rpath was a bust; switched replacing olddir with newdir; add o…
gartung May 18, 2016
b48047e
revert change to cc script
gartung May 18, 2016
4d3a7dc
fix bug that kept LC_LOAD_DYLIB from being changed
gartung May 19, 2016
a285488
fix incorrect change
gartung May 23, 2016
4033646
typo
gartung May 23, 2016
1602d70
Merge branch 'llnl-develop' into binary_packages
gartung Jun 22, 2016
acaff17
fix typo from merge
gartung Jun 22, 2016
a8e2230
fixup merge
gartung Jun 22, 2016
f2bd84e
more merge fixups
gartung Jun 22, 2016
f42ff74
even more merge fixups
gartung Jun 22, 2016
20953d7
these were removed during merge
gartung Jun 23, 2016
b26f811
do not fail if tarball exists
gartung Jun 23, 2016
c67e4a7
only create tarball if it does not exist
gartung Jun 23, 2016
8c32777
only create tarball if it does not exist
gartung Jun 23, 2016
6f7bcd2
Merge remote-tracking branch 'origin/develop' into binary_packages
gartung Jun 27, 2016
7446f7c
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Jun 30, 2016
436d734
Merge branch 'binary_packages' into llnl-develop
gartung Aug 1, 2016
e9bc73c
Merge branch 'develop' of https://github.com/LLNL/spack into llnl-dev…
gartung Aug 1, 2016
90bbdca
Loop over all mirrors trying to find binary tarball
gartung Aug 4, 2016
9ce4863
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Aug 9, 2016
22b0e4b
install-policy -> fetch-binary
gartung Aug 11, 2016
4f8167b
Don't assume spack is in the path when building docs.
tgamblin Aug 15, 2016
1d73ce4
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
gartung Aug 30, 2016
29e4233
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
gartung Aug 30, 2016
cb74a9b
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Sep 22, 2016
5f2243a
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
gartung Sep 22, 2016
1cf9e76
Merge remote-tracking branch 'my/binary_packages' into binary_packages
gartung Sep 22, 2016
dc5cf9f
new opsnssl version
gartung Sep 23, 2016
772668e
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Sep 27, 2016
04dc337
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
gartung Sep 28, 2016
cfee296
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Sep 30, 2016
42a2553
add sqlite 3.12.2
gartung Sep 30, 2016
53b15b5
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Sep 30, 2016
bf3b163
Merge branch 'develop' of https://github.com/LLNL/spack into binary_p…
gartung Nov 3, 2016
bf7f1e0
changes needed after merging
gartung Nov 10, 2016
8576e76
missed
gartung Nov 11, 2016
fa01377
pass fetch_binary with the new api
gartung Nov 11, 2016
6e99a79
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Nov 15, 2016
9eb6e1a
this got removed in merge
gartung Nov 15, 2016
7da99bb
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Nov 15, 2016
6f47aa2
this is the correct path to use
gartung Nov 15, 2016
b0206ba
fix errors travis found
gartung Nov 15, 2016
47f5133
sigh
gartung Nov 15, 2016
aa378db
use upstream sqlite
gartung Nov 15, 2016
e18fddd
now avaliable in upstream
gartung Nov 15, 2016
1d475a7
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Nov 15, 2016
ac68a2d
this is really needed for the test
gartung Nov 15, 2016
6000d15
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Nov 16, 2016
41722b8
more nit picking
gartung Nov 16, 2016
e15a3f0
Merge branch 'binary_packages' of github.com:gartung/spack into binar…
gartung Nov 16, 2016
2245c00
more format fixes
gartung Nov 16, 2016
95e8b91
apple-clang does have cxx14 support
gartung Dec 5, 2016
9d94d24
Merge remote-tracking branch 'my/binary_packages' into binary_packages
gartung Dec 5, 2016
a7c0408
more test fixups
gartung Dec 14, 2016
d9fedc4
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Dec 16, 2016
9cd1543
no spaces in docstring???
gartung Dec 16, 2016
c881b54
no leading spaces???
gartung Dec 16, 2016
9412ee1
add binary_caches to index.rst???
gartung Dec 16, 2016
52a9e5d
Merge branch 'releases/v0.10.0'
tgamblin Jan 17, 2017
b2019a6
Merge tag 'v0.10.0' into binary_packages
gartung Mar 15, 2017
8830cfc
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Apr 28, 2017
9a4bccf
move fetch_binary actions out of install method
gartung May 12, 2017
9552235
Merge branch 'develop' into binary_packages
gartung May 12, 2017
a1d05c0
move the installation of binary tarballs to its own command
gartung May 12, 2017
1222851
autopep8
gartung May 12, 2017
bb117f6
fixes from testing
gartung May 17, 2017
847ba89
add to description
gartung May 17, 2017
2093e57
remove error for create_tarball and install_tarball when spack is run…
gartung May 17, 2017
eaff47b
more fixes from testing
gartung May 17, 2017
3b74ac1
Add generic variant to gmp for CXXFLAGS=-mtune=generic. This allows a…
gartung May 18, 2017
23265aa
mtune=generic is specific to gcc
gartung May 18, 2017
3148849
autopep8
gartung May 18, 2017
5a58895
recommended fix for Illegal instruction error @ http://www.linuxfroms…
gartung May 18, 2017
01f3887
Merge branch 'develop' into binary_packages
gartung May 22, 2017
b60f74d
Merge branch 'develop' into gmp-generic
gartung May 22, 2017
78e0eb5
WIP: binary_distribution: pseudocode for GPG-signed binaries
mathstuf May 22, 2017
d8e2d17
Expand on Ben Boeckel's use of python tarfile
gartung May 23, 2017
7d64020
update binary_cache doc
gartung May 23, 2017
1961e68
Extract the signature files and the tarball in stage directory.
gartung May 23, 2017
99ed0cc
remove outdated packaging.py
gartung May 23, 2017
beb878d
fix the error from spack help -a
gartung May 23, 2017
7f6c669
replace use of getstatusoutput() with which(command) followed by comm…
gartung May 23, 2017
7a269f8
use filter_file instead of os.system('sed....')
gartung May 23, 2017
e06576f
don't need symbolic links
gartung May 23, 2017
30aa688
spack does not seem to like dashes or underscores in the command names
gartung May 23, 2017
52b614c
unknown command error caused by import of relocate.py failing
gartung May 23, 2017
589f119
autopep8
gartung May 23, 2017
2109078
does spack really hate dashes in command names???
gartung May 23, 2017
80ccb59
Make installtarball and createtarball a multi level command
gartung May 23, 2017
4ef502d
autopep8
gartung May 23, 2017
0a41a51
add context.closing around tarfile.open
gartung Jun 15, 2017
7d38571
only make buildcache for link and run depdencies
gartung Jun 15, 2017
e60249b
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Jun 15, 2017
6190135
flake8
gartung Jun 16, 2017
719a471
Merge branch 'develop' into gmp-generic
gartung Jun 16, 2017
cf22340
add rpath derived from compiler used by spack
gartung Jun 19, 2017
ec856d8
don't add this for now
gartung Jun 19, 2017
4d3c062
skip adding gcc_prefix if it comes from CLT
gartung Jun 19, 2017
7b87b44
don't try to relocate libgcc_*.dylib on macOS
gartung Jun 21, 2017
3337724
Merge remote-tracking branch 'llnl/develop' into build_caches
gartung Jun 21, 2017
23d878a
keep a copy of spec.yaml out of the .spack tarball so it can be acces…
gartung Jun 21, 2017
12393eb
keep a copy of spec.yaml out of the .spack tarball so it can be acces…
gartung Jun 21, 2017
c184489
Add buildcache list subcommand. Install buildcaches based on listed s…
gartung Jun 22, 2017
d77e971
Merge branch 'build_caches' of github.com:gartung/spack into build_ca…
gartung Jun 22, 2017
04b0076
construct compiler spec to set gcc_prefix
gartung Jun 22, 2017
ff217da
put spec.yaml in build_cache dir
gartung Jun 22, 2017
3670141
find dependcies the correct way
gartung Jun 22, 2017
a7fef8e
always relocate in case compiler prefix has changed
gartung Jun 22, 2017
dc2b964
Merge remote-tracking branch 'llnl/develop' into build_caches
gartung Jun 22, 2017
38ec75a
perl seems to works more reliably
gartung Jun 22, 2017
c3f6efc
Add relative rpaths to elf objects if original rpath is in spack inst…
gartung Jun 26, 2017
3bafaee
Add ability to install by hash
gartung Jun 27, 2017
780cf4c
turn off debug print
gartung Jun 27, 2017
232a1fd
temp fix
gartung Jun 27, 2017
8b57f11
skip buildcache of externals
gartung Jun 27, 2017
478a7cc
format spec when printing
gartung Jun 27, 2017
9cfc344
Reconstruct install directory from spec.yaml
gartung Jun 27, 2017
2b0fa1f
prelocate for Mach-o objects too
gartung Jun 27, 2017
45485c9
This changes the library ID of libgcc_s.1.dylib from an absolute path…
gartung Jun 28, 2017
714c857
autopep8
gartung Jun 28, 2017
72b8d3f
make changing to relative rpaths before tarball creation optional
gartung Jun 28, 2017
90ce619
Merge branch 'develop' of https://github.com/LLNL/spack into gcc-libg…
gartung Jun 28, 2017
a69f312
make this a variant
gartung Jun 28, 2017
d875e54
Merge branch 'gcc-libgcc-rpath-resub' of github.com:gartung/spack int…
gartung Jun 28, 2017
9dd3b57
Merge branch 'gcc-libgcc-rpath-resub' into build_caches
gartung Jun 28, 2017
dd25a40
Document the need for setting extra_rpaths. Use conflicts instead of …
gartung Jun 28, 2017
b7f5fd2
Merge branch 'gcc-libgcc-rpath-resub' into build_caches
gartung Jun 28, 2017
bc5e7c9
Add two python packages for editing mach-O headers. Much faster than …
gartung Jul 11, 2017
bb85cde
Merge branch 'py-mach-o-tools' into build_caches
gartung Jul 11, 2017
4e3620f
autopep8
gartung Jul 11, 2017
cede86f
Merge branch 'gcc-libgcc-rpath-resub' into build_caches
gartung Jul 11, 2017
29155a7
autopep8
gartung Jul 11, 2017
bae8c04
Merge branch 'gmp-generic' into build_caches
gartung Jul 11, 2017
72f1ab2
remove perl ~ backup file
gartung Jul 12, 2017
968fd77
Merge branch 'build_caches' of github.com:gartung/spack into build_ca…
gartung Jul 12, 2017
e5d32d5
remove non-overriding function
gartung Jul 13, 2017
39a351c
Document the need for setting extra_rpaths. Use conflicts instead of …
gartung Jun 28, 2017
fe4eefe
Merge branch 'develop' of https://github.com/LLNL/spack into gcc-libg…
gartung Jul 13, 2017
0729d67
Merge branch 'gcc-libgcc-rpath-resub' into build_caches
gartung Jul 13, 2017
74afb10
All previous commits squashed into one.
gartung Jul 13, 2017
f3b2c9d
autopep8
gartung Jul 13, 2017
a2a66bf
Merge branch 'py-mach-o-tools' into build_caches
gartung Jul 13, 2017
3d77dd0
Merge remote-tracking branch 'origin/binary_packages' into binary_pac…
gartung Jul 13, 2017
431ae5f
don't submit these on this branch
gartung Jul 13, 2017
47485f2
not needed since testing was removed
gartung Jul 13, 2017
5f993d0
pep8 fixes
gartung Jul 14, 2017
674d7b9
add the options to use pgp2 to sign and verify the build caches
gartung Jul 14, 2017
b9ab9a3
flake8 fixes
gartung Jul 14, 2017
ed301e8
move verify parsing
gartung Jul 14, 2017
b6f1f4d
fix order of arguments
gartung Jul 14, 2017
4ac6d50
change order of arguments in correct function call
gartung Jul 14, 2017
4a81550
make buildcache install -f remove install prefix before unpacking tar…
gartung Jul 14, 2017
8a05baa
Set gpg2 signing and verifying to True by default.
gartung Jul 14, 2017
169afab
Don't include gartung.key. The trusted keys should be copied into var…
gartung Jul 14, 2017
685e458
add buildcache keys command to get keys on mirror and option to trust…
gartung Jul 14, 2017
fb46610
Ask before trusting keys downloaded from mirror.
gartung Jul 17, 2017
a610232
keep the spec file
gartung Jul 18, 2017
4316403
Merge remote-tracking branch 'llnl/develop' into binary_packages
gartung Jul 21, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions lib/spack/docs/binary_caches.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
.. _binary_caches:

Build caches
============================

.. warning:: The feature of build caches is still evolving over time.

Some sites may encourage users to set up their own test environments
before carrying out central installations, or some users prefer to set
up these environments on their own motivation. To reduce the load of
recompiling otherwise identical package specs in different installations,
created build artifacts can be put into binary tarballs, uploaded onto
your spack mirror and then downloaded and installed by others.


Creating build cache tarballs
-----------------------------

Tarballs of sofware built can be created via
``spack buildcache create ``.
It allows either to tar up a a package including all
its dependencies. The location for the tarballs
can be given via the ``--directory`` option:

.. code-block:: sh

$ spack buildcache create -d ~/caches bison
==> recursing dependencies
==> adding dependency bison@3.0.4%gcc@5.3.1=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64^m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64
==> adding dependency m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64
==> adding dependency libsigsegv@2.10%gcc@5.3.1=linux-x86_64
==> creating tarball for package bison@3.0.4%gcc@5.3.1=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64^m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64
==> /home/hegner/caches/ubuntu16_04-x86_64/gcc-5.3.1/bison/ubuntu16_04-x86_64-bison-3.0.4-n6naf2v2wt2p5tg3jdveuqufhjwlba7o.tar.gz
==> creating tarball for package libsigsegv@2.10%gcc@5.3.1=linux-x86_64
==> /home/hegner/caches/ubuntu16_04-x86_64/gcc-5.3.1/libsigsegv/ubuntu16_04-x86_64-libsigsegv-2.10-klc6t4jq2w6ochuz6xosu6vaujbwszds.tar.gz
==> creating tarball for package m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64
==> /home/hegner/caches/ubuntu16_04-x86_64/gcc-5.3.1/m4/ubuntu16_04-x86_64-m4-1.4.17-6hpdn55vhztd25vxwuamxqo7edmootwv.tar.gz


The created tarballs are put into the directory structure expected for the
spack mirror.


Installing build cache tarballs
--------------------------------

To install binary tarballs, one has to add the corresponding spack mirror
with ``spack mirror add <name> <url>``. Afterwards binaries can be installed
via:

.. code-block:: sh

$ spack buildcache install bison

The package bison and all its dependencies will be downloaded from the
specified mirror(s). It fails if a package cannot be downloaded.


Relocation
-------------------------------

Initial build and later installation do not necessarily happen at the same
location. Spack provides a very basic relocation capability and corrects for
RPATHs and non-relocatable scripts. However, many packages compile paths into
binary artificats directly. In such cases, the build instructions of this
package would need to be adjusted for better re-locatability.
1 change: 1 addition & 0 deletions lib/spack/docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ or refer to the full manual below.
repositories
command_index
package_list
binary_caches

.. toctree::
:maxdepth: 2
Expand Down
Loading