Re-work shared vs. static builds (Universal Variants)

[citibeth]

@davydden @adamjstewart @alalazo @tgamblin

This is a continuation of #2380. It's in a new thread because the proposal now goes far beyond documentation. The idea would be to Create a dummy package (call it spack for now) with a multi-valued variant (#2386; call it library-type for now taking enum values shared and static). Then make the base class Package depends_on('spack') (#2466) by default. This will have the following effects:

1. Packages should look at spec['spack'].library_type to figure out whether they should either
   a) build shared libs (or static libs with -fPIC-or-equivalent)
   b) build static libs

2. Users can set up all of Spack for either shared or static mode my making an entry for spack/build_type in packages.yaml. Alternately, they can write ^spack@build-type=xxx directly on the command line.

This proposal solves a serious usability problem of all our other shared vs static proposals: that users will have to put +shared or +static all over their builds and specs --- but ONLY for packages that build binaries. It would have worked out fine for most users, but been a royal PITA for Cray users.

We can do other "universal variants" this way as well. It is much cleaner than trying to forward variants from the top of the DAG. (Or we can attach less-than-universal variants to dummy packages that are only depended on by SOME packages.)

[alalazo]

I think this is really going somewhere. The only thing that doesn't convince me out of the box is the implementation with fake dependencies in a base class, as we may find something cleaner. But having globals for certain properties definitely makes sense.

We may even provide a spack globals command to set or query them from cli like:

spack globals set library-type=static

Does anybody see use-cases that would be negatively affected by this proposal? Currently I can think only of artifact cases, that would never happen in a real production environment.

[citibeth]

In general, I think we should avoid building additional machinery if etisting stuff can be used to do the job.

…

On Dec 6, 2016 7:02 AM, "Massimiliano Culpo" ***@***.***> wrote: I think this is really going somewhere. The only thing that doesn't convince me out of the box is the implementation with fake dependencies in a base class, as we may find something cleaner. But having globals for certain properties definitely makes sense. We may even provide a spack globals command to set or query them from cli like: spack globals set library-type=static — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2492 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd61YxxUtuVmSWe3iiAztFb65xNQYks5rFU7HgaJpZM4LFUY7> .

[alalazo]

In general, I think we should avoid building additional machinery if etisting stuff can be used to do the job.

That's debatable, but my point is that the implementation would be a stretch of the dependency model, at least logically. Anyhow I consider this a minor issue at the moment.

[tgamblin]

My only objection to this is that the relationship is actually more complicated than what's allowed by the dummy dependency. Suppose we have library_type and it can be one of:

• ('static')
• ('shared')
• ('static', 'shared')

By using a dummy dependency for "global" properties like this, you force builds with any difference in global settings to be completely separate. So the more things you add to the spack dep, the more likely it is you'll have to rebuild everything for small changes. Example: if you have hdf5^spack+static,shared, it can't be depended on by app^spack+static, even though it would be binary compatible. You'd have to rebuild an entire new stack where every package is ^spack+static.

I like the fact that the global enforces consistency, and we've discussed doing something like this for compilers in #3296 with virtual dependencies. vdeps may be a better model for this, but I haven't though that through -- I don't think static/shared libs quite match vdep semantics.

I think that a potentially better solution may be offered by the new concretizer (coming real soon now™). Once we have concretization expressed as SAT, I think it will be fairly easy to add additional rules to the solve for things like "packages that have +static also need dependencies to have +static if possible". That's just an example -- I think we'd need to hash out the rules and conventions, but at least they'd be expressible in a not-so-ad-hoc way.

[citibeth]

I would say that our thinking has evolved since this Issue was posted. In particular, #2644 provides similar functionality and is much simpler (and also implemented). At this point, we are waiting on #2386 as a prerequisite to clean up shared vs. static builds.

Might universal variants be used for something else, or should this issue be closed? My gut feeling is that bottom-up forwarding of variants feels wrong. In the world of build systems, this practice is frowned upon; (i.e. features enabled/disabled based on what ./configure can find on the system, rather than explicit flags enabling/disabling them).

However... it is worth noting that some packages (i.e. a random assortment) use this kind of bottom-up forwarding of variants in an ad-hoc manner. We should at least keep around the observation that things can be forwarded bottom-up or top-down.

Should this Issue be closed?