-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Feature: creation of relocatable tarballs from installed packages #1013
Changes from 105 commits
e67b7eb
84de4be
9c67751
c053647
d31ba16
2ef4768
90fcba0
95fe15a
860e5df
7719192
4498737
e7773ce
4c8c82a
8324fcf
3210dd2
5fb14db
8446096
80dbd10
6bad462
bfa5f2c
5911b0b
779b8dc
1f9a01d
a476bd7
132993d
a506bf9
ebd7fe9
4282850
f9cd9a2
66a812d
0607152
a3783c1
5f85533
6e1257e
164da8e
987c549
b48047e
4d3a7dc
a285488
4033646
1602d70
acaff17
a8e2230
f2bd84e
f42ff74
20953d7
b26f811
c67e4a7
8c32777
6f7bcd2
7446f7c
436d734
e9bc73c
90bbdca
9ce4863
22b0e4b
4f8167b
1d73ce4
29e4233
cb74a9b
5f2243a
1cf9e76
dc5cf9f
772668e
04dc337
cfee296
42a2553
53b15b5
bf3b163
bf7f1e0
8576e76
fa01377
6e99a79
9eb6e1a
7da99bb
6f47aa2
b0206ba
47f5133
aa378db
e18fddd
1d475a7
ac68a2d
6000d15
41722b8
e15a3f0
2245c00
95e8b91
9d94d24
a7c0408
d9fedc4
9cd1543
c881b54
9412ee1
52a9e5d
b2019a6
8830cfc
9a4bccf
9552235
a1d05c0
1222851
bb117f6
847ba89
2093e57
eaff47b
3b74ac1
23265aa
3148849
5a58895
01f3887
b60f74d
78e0eb5
d8e2d17
7d64020
1961e68
99ed0cc
beb878d
7f6c669
7a269f8
e06576f
30aa688
52b614c
589f119
2109078
80ccb59
4ef502d
0a41a51
7d38571
e60249b
6190135
719a471
cf22340
ec856d8
4d3c062
7b87b44
3337724
23d878a
12393eb
c184489
d77e971
04b0076
ff217da
3670141
a7fef8e
dc2b964
38ec75a
c3f6efc
3bafaee
780cf4c
232a1fd
8b57f11
478a7cc
9cfc344
2b0fa1f
45485c9
714c857
72b8d3f
90ce619
a69f312
d875e54
9dd3b57
dd25a40
b7f5fd2
bc5e7c9
bb85cde
4e3620f
cede86f
29155a7
bae8c04
72f1ab2
968fd77
e5d32d5
39a351c
fe4eefe
0729d67
74afb10
f3b2c9d
a2a66bf
3d77dd0
431ae5f
47485f2
5f993d0
674d7b9
b9ab9a3
ed301e8
b6f1f4d
4ac6d50
4a81550
8a05baa
169afab
685e458
fb46610
a610232
4316403
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
.. _binary_caches: | ||
|
||
Binary caches | ||
============================ | ||
|
||
.. warning:: The feature of binary caches is still experimental | ||
and chosen conventions still may evolve over time. | ||
|
||
Some sites may encourage users to set up their own test environments | ||
before carrying out central installations, or some users prefer to set | ||
up these environments on their own motivation. To reduce the load of | ||
recompiling otherwise identical package specs in different installations, | ||
created build artifacts can be put into binary tarballs, uploaded onto | ||
your spack mirror and then downloaded and installed by others. | ||
|
||
|
||
Creating binary tarballs | ||
----------------------------- | ||
|
||
Tarballs of sofware built can be created via ``spack create-tarball``. | ||
It allows either to tar up a single package or a package including all | ||
its dependencies (``-r``, ``--recurse``). The location for the tarballs | ||
can be given via the ``--directory`` option: | ||
|
||
.. code-block:: sh | ||
|
||
$ spack create-tarball -d ~/caches -r bison | ||
==> recursing dependencies | ||
==> adding dependency bison@3.0.4%gcc@5.3.1=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64^m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64 | ||
==> adding dependency m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64 | ||
==> adding dependency libsigsegv@2.10%gcc@5.3.1=linux-x86_64 | ||
==> creating tarball for package bison@3.0.4%gcc@5.3.1=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64^m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64 | ||
==> /home/hegner/caches/ubuntu16_04-x86_64/gcc-5.3.1/bison/ubuntu16_04-x86_64-bison-3.0.4-n6naf2v2wt2p5tg3jdveuqufhjwlba7o.tar.gz | ||
==> creating tarball for package libsigsegv@2.10%gcc@5.3.1=linux-x86_64 | ||
==> /home/hegner/caches/ubuntu16_04-x86_64/gcc-5.3.1/libsigsegv/ubuntu16_04-x86_64-libsigsegv-2.10-klc6t4jq2w6ochuz6xosu6vaujbwszds.tar.gz | ||
==> creating tarball for package m4@1.4.17%gcc@5.3.1+sigsegv=linux-x86_64^libsigsegv@2.10%gcc@5.3.1=linux-x86_64 | ||
==> /home/hegner/caches/ubuntu16_04-x86_64/gcc-5.3.1/m4/ubuntu16_04-x86_64-m4-1.4.17-6hpdn55vhztd25vxwuamxqo7edmootwv.tar.gz | ||
|
||
|
||
The created tarballs are put into the directory structure expected for the | ||
spack mirror. | ||
|
||
|
||
Installing binary tarballs | ||
-------------------------------- | ||
|
||
To install binary tarballs, one has to add the corresponding spack mirror | ||
with ``spack mirror add <name> <url>``. Afterwards binaries can be installed | ||
via: | ||
|
||
.. code-block:: sh | ||
|
||
$ spack install-tarball bison | ||
|
||
The package bison and all its dependencies will be downloaded from the | ||
specified mirror(s). It fails if a package cannot be downloaded. | ||
|
||
|
||
Relocation | ||
------------------------------- | ||
|
||
Initial build and later installation do not necessarily happen at the same | ||
location. Spack provides a very basic relocation capability and corrects for | ||
RPATHs and non-relocatable scripts. However, many packages compile paths into | ||
binary artificats directly. In such cases, the build instructions of this | ||
package would need to be adjusted for better re-locatability. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,230 @@ | ||
# "Benedikt Hegner (CERN)" | ||
|
||
import os | ||
import platform | ||
import yaml | ||
|
||
import llnl.util.tty as tty | ||
from llnl.util.filesystem import mkdirp | ||
|
||
from spack.util.executable import which | ||
import spack.cmd | ||
import spack | ||
from spack.stage import Stage | ||
import spack.fetch_strategy as fs | ||
import spack.relocate | ||
|
||
|
||
def get_full_system_from_platform(): | ||
import platform | ||
import re | ||
system = platform.system() | ||
if system == "Linux": | ||
pf = platform.linux_distribution(full_distribution_name=0)[0] | ||
version = platform.linux_distribution(full_distribution_name=0)[1] | ||
if pf != 'Ubuntu': | ||
# For non-Ubuntu major version number is enough | ||
# to understand compatibility | ||
version = version.split('.')[0] | ||
elif system == "Darwin": | ||
pf = "macos10" | ||
version = platform.mac_ver()[0].split(".")[1] | ||
else: | ||
raise "System %s not supported" % system | ||
sys_type = pf + version + '-' + platform.machine() | ||
sys_type = re.sub(r'[^\w-]', '_', sys_type) | ||
return sys_type.lower() | ||
|
||
|
||
def prepare(): | ||
""" | ||
Install patchelf as pre-requisite to the | ||
required relocation of binary packages | ||
""" | ||
if platform.system() == 'Darwin': | ||
return | ||
dir = os.getcwd() | ||
patchelf_spec = spack.cmd.parse_specs("patchelf", concretize=True)[0] | ||
if not spack.store.layout.check_installed(patchelf_spec): | ||
patchelf = spack.repo.get(patchelf_spec) | ||
patchelf.do_install() | ||
os.chdir(dir) | ||
|
||
|
||
def buildinfo_file_name(spec): | ||
""" | ||
Filename of the binary package meta-data file | ||
""" | ||
return os.path.join(spec.prefix, ".spack", "binary_distribution") | ||
|
||
|
||
def read_buildinfo_file(package): | ||
""" | ||
Read buildinfo file | ||
""" | ||
filename = buildinfo_file_name(package) | ||
with open(filename, 'r') as inputfile: | ||
content = inputfile.read() | ||
buildinfo = yaml.load(content) | ||
return buildinfo | ||
|
||
|
||
def write_buildinfo_file(spec): | ||
""" | ||
Create a cache file containing information | ||
required for the relocation | ||
""" | ||
text_to_relocate = [] | ||
binary_to_relocate = [] | ||
blacklist = (".spack", "man") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What if they put binaries in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We removed it because it was just time wasted in there. Scanning plenty of files. I will remove that exclusion. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If it saves tons of time it might be worth it 😄. I actually haven't seen anyone put binaries in |
||
for root, dirs, files in os.walk(spec.prefix, topdown=True): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If it's not ridiculously expensive, we should consider just making this file get generated and installed to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd prefer doing that once we went through the first iterations and know the format is really stable. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sounds reasonable |
||
dirs[:] = [d for d in dirs if d not in blacklist] | ||
for filename in files: | ||
path_name = os.path.join(root, filename) | ||
filetype = spack.relocate.get_filetype(path_name) | ||
if spack.relocate.needs_binary_relocation(filetype): | ||
rel_path_name = os.path.relpath(path_name, spec.prefix) | ||
binary_to_relocate.append(rel_path_name) | ||
elif spack.relocate.needs_text_relocation(filetype): | ||
rel_path_name = os.path.relpath(path_name, spec.prefix) | ||
text_to_relocate.append(rel_path_name) | ||
|
||
# Create buildinfo data and write it to disk | ||
buildinfo = {} | ||
buildinfo['buildpath'] = spack.store.layout.root | ||
buildinfo['relocate_textfiles'] = text_to_relocate | ||
buildinfo['relocate_binaries'] = binary_to_relocate | ||
filename = buildinfo_file_name(spec) | ||
with open(filename, 'w') as outfile: | ||
outfile.write(yaml.dump(buildinfo, default_flow_style=True)) | ||
|
||
|
||
def tarball_directory_name(spec): | ||
""" | ||
Return name of the tarball directory according to the convention | ||
<os>-<architecture>/<compiler>/<package>/ | ||
""" | ||
return "%s/%s/%s" % (get_full_system_from_platform(), | ||
str(spec.compiler).replace("@", "-"), | ||
spec.name) | ||
|
||
|
||
def tarball_name(spec): | ||
""" | ||
Return the name of the tarfile according to the convention | ||
<os>-<architecture>-<package>-<dag_hash>.tar.gz | ||
""" | ||
return "%s-%s-%s-%s.tar.gz" % (get_full_system_from_platform(), | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should use
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is for convenience as this is how we order/group the binary archives in the end. From single package perspective your comment makes sense though. |
||
spec.name, | ||
spec.version, | ||
spec.dag_hash()) | ||
|
||
|
||
def tarball_path_name(spec): | ||
""" | ||
Return the full path+name for a given spec according to the convention | ||
<tarball_directory_name>/<tarball_name> | ||
""" | ||
return os.path.join(tarball_directory_name(spec), | ||
tarball_name(spec)) | ||
|
||
|
||
def build_tarball(spec, outdir, force=False): | ||
""" | ||
Build a tarball from given spec and put it into the directory structure | ||
used at the mirror (following <tarball_directory_name>). | ||
""" | ||
tarfile_dir = os.path.join(outdir, tarball_directory_name(spec)) | ||
tarfile = os.path.join(outdir, tarball_path_name(spec)) | ||
if os.path.exists(tarfile): | ||
if force: | ||
os.remove(tarfile) | ||
else: | ||
tty.warn("file exists, use -f to force overwrite: %s" % tarfile) | ||
return | ||
if not os.path.exists(tarfile_dir): | ||
mkdirp(tarfile_dir) | ||
|
||
tar = which('tar', required=True) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a reason we're using the command line and not the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No. It may not have been available at the inception of this branch. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I believe @mathstuf is referring to https://docs.python.org/2/library/tarfile.html There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For the record, we use tar from the command line in other places across Spack. Are there any advantages to using There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I did not author that part so I was guessing. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually what appears to be missing is the path to the tarfile being created. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bug is in create_tarball.py. Need to set the default output directory to $CWD there. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Or it is elsewhere. Tracing now. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @mathstuf python tarfile is much slower than command line tar. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually tarfile is fine. I was creating a tarball of everything installed. |
||
dirname = os.path.dirname(spec.prefix) | ||
basename = os.path.basename(spec.prefix) | ||
|
||
# handle meta-data | ||
cp = which("cp", required=True) | ||
spec_file = os.path.join(spec.prefix, ".spack", "spec.yaml") | ||
target_spec_file = tarfile + ".yaml" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yap. This is there to facilitate the mirroring and querying. You don't want to extract the data from every package "manually" again. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok this would be useful, then, for partial matching if we implement that. This can be used to check if a spec is "good enough" to satisfy a build. |
||
cp(spec_file, target_spec_file) | ||
|
||
# create info for later relocation and create tar | ||
write_buildinfo_file(spec) | ||
tar("--directory=%s" % dirname, "-czf", tarfile, basename) | ||
tty.msg(tarfile) | ||
|
||
|
||
def download_tarball(package): | ||
""" | ||
Download binary tarball for given package into stage area | ||
Return True if successful | ||
""" | ||
mirrors = spack.config.get_config('mirrors') | ||
if len(mirrors) == 0: | ||
tty.die("Please add a spack mirror to allow " + | ||
"download of pre-compiled packages.") | ||
tarball = tarball_path_name(package.spec) | ||
for key in mirrors: | ||
url = mirrors[key] + "/" + tarball | ||
# print url | ||
# stage the tarball into standard place | ||
stage = Stage(url, name=package.stage.path) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @gartung @hegner: This will construct a Homebrew's bottle scheme is pretty simple, and it makes it pretty easy to put binary hashes into formulae. Spack has a slightly more difficult situation because a) there can be a lot of different builds for the same package and b) you can't get the symbolic I think a) and b) can eventually be mitigated if binary archive sites had a way to provide indices of available hashes, so that Spack could download an index of available binaries and find a "close enough match". For a first cut at this, exact matching of the DAG hash is probably fine, especially if you expect users typically not to pull straight from There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @tgamblin - seems you are mixing quite a few concepts into your comment and I am not sure I get all the points you are trying to make. I'd believe that is something for the telcon. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok -- we can discuss more on the call. I think it's worthwhile to have a more detailed discussion about this. I'm not sure you want to actually fetch the hashes directly from the mirror. You need to have a way to guarantee that the hash itself came from a secure channel. If you trust the server to provide both a reliable checksum and a reliable package, you're vulnerable to a compromised server. The hashes/other verification mechanism need to come along some secure channel. The approach I mentioned of signing packages like Debian seems promising for this as it's easy to transfer a public key once and verify packages with it from then on. btw -- LLNL has some S3 space where we can start to put up a mirror. CERN may have something similar -- we can talk about that on the call as well. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Correct - trustworthy is the point. More for the telcon. What concerns mirrors - we could set up something for a start. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @tgamblin I added this kludge because the previous method of fetching the tarball from a mirror was broken after merging in llnl/develop. I have been using my linux desktop as the mirror for testing but it only works onsite at Fermilab. The hashes are there so we can match the binary built to the spec used to build it. |
||
try: | ||
stage.fetch() | ||
return True | ||
except fs.FetchError: | ||
next | ||
return False | ||
|
||
|
||
def extract_tarball(package): | ||
""" | ||
extract binary tarball for given package into install area | ||
""" | ||
tarball = tarball_name(package.spec) | ||
tar = which("tar") | ||
local_tarball = package.stage.path + "/" + tarball | ||
mkdirp('%s' % package.prefix) | ||
tar("--strip-components=1", | ||
"-C%s" % package.prefix, | ||
"-xf", | ||
local_tarball) | ||
|
||
|
||
def relocate_package(package): | ||
""" | ||
Relocate the given package | ||
""" | ||
buildinfo = read_buildinfo_file(package) | ||
new_path = spack.store.layout.root | ||
old_path = buildinfo['buildpath'] | ||
if old_path == new_path: | ||
return True # No need to relocate | ||
|
||
tty.warn("Using experimental feature for relocating package from", | ||
"%s to %s." % (old_path, new_path)) | ||
|
||
# as we may need patchelf, find out where it is | ||
patchelf_executable = '' | ||
if platform.system() != 'Darwin': | ||
patchelf_spec = spack.cmd.parse_specs("patchelf", concretize=True)[0] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is cool; I'll port it over to the GPG branch. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can thank @hegner There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So it actually isn't working here; it keeps throwing There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry. I never tested this on anything but RHEL7 clones and macOS. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm a little bit behind in following the developments here. It fails in the concretization of that spec. Any compiler would be fine if there is one for your platform. Question would be why you get debian6 on one side and fedora25 on another. Does a command line install of patchelf work? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It also seems that the cmd infrastructure just swallows exceptions? I see the exceptions in the test, but the command just silently does nothing :/ . There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It isn't patchelf, but There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since you have it now installed, can you check whether There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This returns the right thing: |
||
patchelf = spack.repo.get(patchelf_spec) | ||
patchelf_executable = os.path.join(patchelf.prefix, "bin", "patchelf") | ||
|
||
# now do the actual relocation | ||
for filename in buildinfo['relocate_binaries']: | ||
path_name = os.path.join(package.prefix, filename) | ||
spack.relocate.relocate_binary(path_name, | ||
old_path, | ||
new_path, | ||
patchelf_executable) | ||
for filename in buildinfo['relocate_textfiles']: | ||
path_name = os.path.join(package.prefix, filename) | ||
spack.relocate.relocate_text(path_name, old_path, new_path) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
############################################################################## | ||
# Copyright (c) 2013, Lawrence Livermore National Security, LLC. | ||
# Produced at the Lawrence Livermore National Laboratory. | ||
# | ||
# This file is part of Spack. | ||
# Written by Todd Gamblin, tgamblin@llnl.gov, All rights reserved. | ||
# LLNL-CODE-647188 | ||
# | ||
# For details, see https://github.com/llnl/spack | ||
# Please also see the LICENSE file for our notice and the LGPL. | ||
# | ||
# This program is free software; you can redistribute it and/or modify | ||
# it under the terms of the GNU General Public License (as published by | ||
# the Free Software Foundation) version 2.1 dated February 1999. | ||
# | ||
# This program is distributed in the hope that it will be useful, but | ||
# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF | ||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and | ||
# conditions of the GNU General Public License for more details. | ||
# | ||
# You should have received a copy of the GNU Lesser General Public License | ||
# along with this program; if not, write to the Free Software Foundation, | ||
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA | ||
############################################################################## | ||
import argparse | ||
|
||
import llnl.util.tty as tty | ||
|
||
import spack | ||
import spack.cmd | ||
from spack.binary_distribution import build_tarball | ||
|
||
|
||
description = "Create binary cache tarballs for given packages" | ||
section = "packaging" | ||
level = "long" | ||
|
||
|
||
def setup_parser(subparser): | ||
subparser.add_argument('-r', '--recurse', action='store_true', | ||
help="also make tarballs for dependencies.") | ||
subparser.add_argument('-f', '--force', action='store_true', | ||
help="overwrite tarball if it exists.") | ||
subparser.add_argument('-d', '--directory', default=".", | ||
help="directory in which to save the tarballs.") | ||
|
||
subparser.add_argument( | ||
'packages', nargs=argparse.REMAINDER, | ||
help="specs of packages to package") | ||
|
||
|
||
def create_tarball(parser, args): | ||
if not args.packages: | ||
tty.die("binary cache tarball creation requires at least one package argument") | ||
|
||
pkgs = set(args.packages) | ||
specs = set() | ||
for pkg in pkgs: | ||
for spec in spack.cmd.parse_specs(pkg, concretize=True): | ||
specs.add(spec) | ||
if args.recurse: | ||
tty.msg('recursing dependencies') | ||
for d, node in spec.traverse(order='pre', depth=True): | ||
tty.msg('adding dependency %s' % node) | ||
specs.add(node) | ||
for spec in specs: | ||
tty.msg('creating binary cache tarball for package %s ' % spec) | ||
build_tarball(spec, args.directory, args.force) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Spec('patchelf').concretized()
might be simpler here, but there's a better way nowThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tgamblin - thanks, didn't know. What do you mean with "there's a better way now"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding comments as I go 😄 -- I mean you don't need to concretize this.
installed_db.query()
, described below, should match against specs that are installed, even if they don't exactly match whatpatchelf
currently concretizes to.