Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create, install and relocate tarballs of installed packages. #4854

Merged
merged 42 commits into from Aug 14, 2017
Merged
Show file tree
Hide file tree
Changes from 39 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
2211a4c
Adds the buildcache command to spack.
gartung Jul 21, 2017
3557fcb
convert packaging test to pytest framework
gartung Jul 24, 2017
c01923b
documentation updates
gartung Jul 24, 2017
713a3e3
pass in key if given on command line
gartung Jul 24, 2017
1fc1e0f
fix order of argumens to build_tarball
gartung Jul 24, 2017
44482ca
improve the coverage of testing without calling gpg2
gartung Jul 25, 2017
0b2d3ba
make sure gpg2 is available
gartung Jul 26, 2017
724c27d
add testing of buildcache command
gartung Jul 26, 2017
052b7cb
autopep8
gartung Jul 26, 2017
64e24cc
keep getting weird failures with the gpg stuff
gartung Jul 26, 2017
d253646
autopep8 and flake8 fixes
gartung Jul 26, 2017
514af1e
Merge branch 'develop' into build_caches
gartung Jul 26, 2017
698d6ab
mimic the output of patchelf by echoing /home/gartung/autopep8/bin:/h…
gartung Jul 27, 2017
fd40e52
factor out path string replacement for mach-O object so they can be t…
gartung Jul 27, 2017
138206f
missed a needed argument
gartung Jul 27, 2017
26c5c84
cover modify_macho_object
gartung Jul 27, 2017
27b942d
Address test failure where tarfile is signed but verify fails. Skip v…
gartung Jul 27, 2017
29e50b6
Add sha256 checksum of binary tarball to package spec.yaml. This is t…
gartung Jul 31, 2017
d781eb7
remove extraneous chdir command
gartung Aug 1, 2017
ae360f0
Use gpg2 if it is in the system path
gartung Aug 1, 2017
6abd14e
Revert "Use gpg2 if it is in the system path"
gartung Aug 2, 2017
cb9928c
add gpg directory and import mock_gpg key ig gpg2 is available
gartung Aug 2, 2017
25dafa6
autopep8
gartung Aug 2, 2017
e48c63e
use simple directory listing when the mirror is file://
gartung Aug 2, 2017
85a1a4c
remove FNAL mirror from test
gartung Aug 2, 2017
9ecf3f8
first set of changes from review
gartung Aug 7, 2017
bba77a2
don't need install_directory_name function
gartung Aug 8, 2017
f0f3b4e
remove debug print
gartung Aug 8, 2017
f004608
put back for coverage
gartung Aug 8, 2017
13b7d09
more recommended changes
gartung Aug 8, 2017
c7f1810
Improve function descriptions. Remove read-only checks since copying …
gartung Aug 10, 2017
bec0e52
Add ProcessError import
gartung Aug 10, 2017
73fdb2c
remove chmod lines as this files should be writable after copying
gartung Aug 10, 2017
b79a47a
WIP
gartung Aug 10, 2017
cb4d87c
call raise not return Exception
gartung Aug 10, 2017
9889d0e
call raise not return Exception
gartung Aug 10, 2017
e764767
remove debug print
gartung Aug 10, 2017
4006f40
increase coverage?
gartung Aug 11, 2017
a7f7153
Merge branch 'build_caches' of github.com:gartung/spack into build_ca…
gartung Aug 11, 2017
fe99867
documentation updates
gartung Aug 11, 2017
f02d2ba
Use superclass Exception
gartung Aug 11, 2017
b11372c
flake8 nitpick
gartung Aug 11, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
102 changes: 102 additions & 0 deletions lib/spack/docs/binary_caches.rst
@@ -0,0 +1,102 @@
.. _binary_caches:

Build caches
============

Some sites may encourage users to set up their own test environments
before carrying out central installations, or some users prefer to set
up these environments on their own motivation. To reduce the load of
recompiling otherwise identical package specs in different installations,
created build artifacts can be put into build cache tarballs, uploaded to
your spack mirror and then downloaded and installed by others.


Creating build cache files
--------------------------

A compressed tarball of an installed package is created. Tarballs are created
for all of its link and run dependency packages as well. Compressed tarballs are
signed with gpg and signature and tarball and put in a ".spack" file. Optionally
, the rpaths ( and ids and deps on macOS ) can be changed to paths relative to
the spack install tree before the tarball is created.

Build caches are created via:

.. code-block:: sh

$ spack buildcache create


Finding or installing build cache files
---------------------------------------

To find build caches or install build caches, a spack mirror must be configured
with

``spack mirror add <name> <url>``.

Build caches are found via:

.. code-block:: sh

$ spack buildcache list

Build caches are installed via:

.. code-block:: sh

$ spack buildcache install


Relocation
----------

Initial build and later installation do not necessarily happen at the same
location. Spack provides a relocation capability and corrects for RPATHs and
non-relocatable scripts. However, many packages compile paths into binary
artificats directly. In such cases, the build instructions of this package would
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> "artifacts"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jenkins speak for binaries.

need to be adjusted for better re-locatability.


Usage
-----
spack buildcache create <>
^^^^^^^^^^^^^^^^^^^^^^^^^^
Create tarball of installed spack package, checksum tarball and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create tarball of installed spack package

... and all dependencies. Places them in a directory. You can place the directory in a mirror and commands like "spack buildcache install" will search it for pre-compiled packages.

sign tarball if gpg2 is available.

options:

-d <path> : directory in which "build_cache" direcory is created, defaults to "."
-f : overwrite ".spack" file in "build_cache" directory if it exists
-k <key> : the key to sign package with. In the case where multiple keys exist, the package will be unsigned unless -k is used.
-r : make paths in binaries relative before creating tarball
-y : answer yes to all create unsigned "build_cache" questions
<> : list of package specs or package hashes with leading /
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is how this has been specified elsewhere in the docs.


spack buildcache list <>
^^^^^^^^^^^^^^^^^^^^^^^^
options:

<> string to be matched to matched to begining of listed concretized short
specs, eg. "spack buildcache list gcc" with print only commands to install gcc
package(s)

spack buildcache install <>
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Install "build_cache" available on spack mirror.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer: "retrieve specs available from each mirror's build cache and installs all specs which match the input spec"


options:

-f : remove install directory if it exists before unpacking tarball
-y : answer yes to all to don't verify package with gpg questions
<> : list of package specs or package hashes with leading /

spack buildcache keys
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it's worthwhile mentioning that this retrieves keys from mirrors configured in mirrors.yaml. It is also worth mentioning that this makes use of gpg2 and the user can do something like spack gpg trust key.asc if they have retrieved the key in a secure manner instead of using this function.

^^^^^^^^^^^^^^^^^^^^^
List public keys available on spack mirror.

options:

-i : trust the keys downloaded with prompt for each
-y : answer yes to all trust all keys downloaded
1 change: 1 addition & 0 deletions lib/spack/docs/index.rst
Expand Up @@ -65,6 +65,7 @@ or refer to the full manual below.
repositories
command_index
package_list
binary_caches

.. toctree::
:maxdepth: 2
Expand Down