Is there a command/phase just prior to make install

[markcmiller86]

I would like to know if there is a way to put a package into the state just prior to the actual install command. So, the package has been fetched, staged, patched and configured/cmake'd or whatever but it is not been installed? Is there a way to tell spack to put a package into this state?

[adamjstewart]

spack build

[markcmiller86]

Setting aside for the moment there is no mention of a command spack build here

[scratlantis:lib/spack/spack] miller86% spack build zlib
==> Error: Package zlib has no build phase, or its build phase is not separated from install
[scratlantis:lib/spack/spack] miller86% 

[adamjstewart]

Not every package supports it, specifically packages that use the Package base class don't support it. AutotoolsPackage and CMakePackage do.

[markcmiller86]

I need it to work for any package though. So, package dependent behavior won't work.

What about this...can I somehow override Spack's idea of what make or maybe change default flags it is passing to make? If I can, I can allow Spack to do the spack install thing but by causing make to behave like make -n avoid the penalty of the actual build until I actually need to build it.

[scheibelp]

"install" is a phase shared by all packages but there is not currently a way to stop when that phase is reached. Is there a reason you want everything up to (but not including) install but more than what you get when you stage the package?

[markcmiller86]

Yes. I am using Spack as a tool to query Software Quality Engineering aspects of a package. I would like to know if a package supports make targets such as install, check or test and clean. Best way to do that is get the package ready to build and then query the Makefile database using something like...

target_list=$(make -f - target_list << EOF
.PHONY: target_list
target_list:
        @\$(MAKE) -pRrq : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if (\$\$1 !~ "^[#.]") {print \$\$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^\$@\$\$'
$(cat Makefile)
EOF)
if [[ -n "$(echo "$target_list" | grep ^install$)" ]]; then
    echo "have install target"
fi
if [[ -n "$(echo "$target_list" | grep ^check$)" ]]; then
    echo "have check target"
fi
if [[ -n "$(echo "$target_list" | grep ^test$)" ]]; then
    echo "have test target"
fi
if [[ -n "$(echo "$target_list" | grep ^clean$)" ]]; then
    echo "have clean target"
fi

I don't wanna try greping the package.py files themselves even though there may be telltail signs in them about what I am looking for.

My idea about somehow either overriding Spack's notion of make with say make --dry-run or overriding flags Spack is passing to make to insert --dry-run would avoid the cost of an actual install to perform these queries.

That said, its not a showstopper for me. I can probably suffer the actual install. And, some of what I need to do in the way of SQE queries does in fact require me to do the install at some point. I am just trying to keep things as much decoupled as possible.

[tgamblin]

@markcmiller86:

I would like to know if there is a way to put a package into the state just prior to the actual install command. So, the package has been fetched, staged, patched and configured/cmake'd or whatever but it is not been installed?

It sounds like you actually want something that runs before spack build -- i.e., it's configured and Makefiles have been generated but the build hasn't happened. I think the easiest way to do that would be to add a method to the package like this:

class SomePackage(SomePackageSubclass):
    @run_before('build')
    def do_something(self):
        # blahblah

That'll work for any package that supports a build phase. You could also write a similar method that runs before 'install'. To get it to run on any package, you'll probably need to inject it into the PackageBase class sometime before the install begins in spack install.

Like @scheibelp said, note that install is somewhat overloaded. We stole the original Spack package interface from homebrew, so install in the base Package class does everything. That was for simplicity. It's the only method that's really required to be implemented by a package author. If you start before install in a regular old Package, nothing has happened yet, so that is probably not what you want. CMakePackage, AutotoolsPackage, and MakefilePackage all have a build phase that happens after configure and before install... I think that is what you want.

Some observations:

1. I think most of the packages that support a proper make install target are already one of CMakePackage, AutotoolsPackage, or MakefilePackage, and they use the install() method from their superclass (which just calls make install). So humans have already done the work for you.
2. Many of the packages that implement Package look like they have an ad-hoc install method written in the Spack code, and they mostly use Makefiles or something weird. Running before install might not be so bad in those cases.
3. If the Package subclasses do have a configure script it's usually some homegrown thing (like for TAU).

It looks like one of your issues here is that zlib is still a Package and not an AutotoolsPackage. @adamjstewart has a comment in there that says zlib breaks when using AutotoolsPackage. Is that still an issue @adamjstewart?

[tgamblin]

@markcmiller86:

My idea about somehow either overriding Spack's notion of make with say make --dry-run or overriding flags Spack is passing to make to insert --dry-run would avoid the cost of an actual install to perform these queries.

In case it wasn't clear above, Spack's notion of make is generally the build phase.

[tgamblin]

@markcmiller86: another thing you might consider looking at is the implementation of spack create, which already does a bunch of spelunking around in downloaded package archives in order to figure out how to generate a template package with the right build system superclass. I think that might do some of what you want.

[adamjstewart]

Setting aside for the moment there is no mention of a command spack build here

Yeah, commands don't get added to the command index unless they appear in the documentation. Feel free to write a quick section on spack configure and spack build if you have the time.

What about this...can I somehow override Spack's idea of what make or maybe change default flags it is passing to make? If I can, I can allow Spack to do the spack install thing but by causing make to behave like make -n avoid the penalty of the actual build until I actually need to build it.

It wouldn't be hard to hack this. Check in lib/spack/spack/build_environment.py for where make() is defined.

Immediately after:

m.make = MakeExecutable('make', jobs)

add:

m.make.add_default_arg('-n')

Of course, this means that if you need to build a stack of software with dependencies, none of the dependencies will actually build either.

To get it to run on any package, you'll probably need to inject it into the PackageBase class sometime before the install begins in spack install.

This won't actually work. From my experience, if you define a @run_before('foo') in a base class and then define/redefine phases in a sub-class, all of your run_before/run_after decorators disappear. You would have to add it to the Package base class.

It looks like one of your issues here is that zlib is still a Package and not an AutotoolsPackage. @adamjstewart has a comment in there that says zlib breaks when using AutotoolsPackage. Is that still an issue @adamjstewart?

Yes, although the package comes with a configure script, it does not use Autotools. I contacted the developer about this and he confirmed. I think the reason I couldn't get it to build was that AutotoolsPackage runs /absolute/path/to/configure instead of ./configure. But regardless, we really shouldn't use AutotoolsPackage unless the package actually uses Autotools.

You might also be interested in the discussion in #3642. In an ideal world, packages like zlib that don't use AutotoolsPackage would define their own custom phases much like perl does. Unfortunately, right now, spack build doesn't work even if your package has a manual build phase. If we can fix that, you can convert packages that use the Package to include individual phases.

[alalazo]

@markcmiller86 Closing this discussion as stale. Feel free to reopen if you think the topic needs some further clarification.