Push towards documented consensus on shared packages

[becker33]

As discussed in the telcon this week: https://github.com/LLNL/spack/wiki/Telcon%3A-2017-08-31

Also as discussed on the telcon, I'm going to use "common variants" to refer to variants that happen to be present on many packages, and "global variants" or "universal variants" to refer to a variant that is technically ensured to be present for all packages (or some major subset).

We need to document our community standards for common variants. There are reasons that we don't use universal variants for shared/static libraries, but community standards make Spack more usable. Unless/until we put those community standards into the documentation and start rejecting pull requests that don't adhere to those standards, we can't get the full benefits of standardization.

I have two concrete proposals for what the standard for shared/static variants should be. These will also interact with the (already standardized) pic variant. The goal is to hash out any corner cases of my proposals, choose one of them (or theoretically some third proposal), and document and implement it. We will then request changes to any incoming packages that fail to adhere to the standard.

Standard 1. This is what we (mostly) currently do.
All packages that can build either static or shared libraries have a variant shared. This variant defaults to True. When True, we build static and shared libraries. When false, we build only static libraries.

Pros: Already in use for most packages
Cons: Does not allow us to only build shared. Requires us to depend on ~shared or +pic versions of all dependencies when building ~static. Developers think of building "static only" more than "no shared"

Standard 2. Use multi-valued variants
All packages that can provide either static or shared libraries have a variant libraries, with possible values static and shared, non-exclusively. We provide whichever libraries are requested in this variant.

Pros: Allows any combination of shared and static libraries.
Cons: Requires changes to many packages. Requires manual variant forwarding (until #391 is reintroduced) for the libraries variant.

I'm leaning slightly towards option 2. I'd like input on possible corner cases for each of them and how they may improve or disrupt existing workflows.

@citibeth @markcmiller86 @tgamblin @alalazo @adamjstewart I know you have all had previous feelings on this topic.

[davydden]

If we allow building both shared and static simultaneously, can we get in trouble when within the same DAG there are packages which use both?

If not, I like multivalued variant approach.

[scheibelp]

I think we were also talking about the possibility of a single-valued variant with the possible values shared, static, and static_and_shared. The idea would be that static implies building only the static libraries.

[citibeth]

OK, here goes how I've thought it through... We need a single 3-valued variant (call it link-type): static, shared, pic. PIC means "static with PIC code," whereas static means "static without PIC flag." The assumption is users will generally just sent a single global *all:link-type*, and then use it for everything they build. Most users will use *link-type=shared*. Cray users will use *link-type=static*. Users who want to build everything static but make their stuff available for use in shared libraries might set *link-type=pic*. Now and then, someone will choose to adjust the *link-type* for an individual library. Of course, not all packages can build all these three types; in case that's not possible, a package will need to build a suitable substitute. But WHICH substitute? The key here is that the value of *link-type* tells Spack *how you intend to link*, it does not necessarily say how that particular library is constructed. Each package therefore needs to build SOMETHING that is compatible with your intended link style. For example, if you selected *link-type=shared* and a package can only do *static* or *pic*, then it needs to build a *pic* library for you (which is not shared), because that is compatible with shared linkage. If a package is ONLY able to build *static*, then it is incompatible with *link-type=shared*, and exclusion features should be used in Spack to throw the appropriate error at concretization time. So there's a 3x3 matrix of what library link-types are compatible with what requested link-types. That matrix is: requested=shared: compatible with shared, pic requested=pic: compatible with pic requested=static: compatible with static, pic Note that MAYBE *shared* should be compatible with *requested=static* as a last resort. On systems that support shared libraries, this would be a good substitute. However, on systems like Cray, it would not. Therefore, *shared* should not be compatible with *requested=static*. If someone is willing to let shared libraries "slip into" the build, they can set *link-type=shared* individually for the appropriate libraries. Oh yes... if you want to build with two different values for *link-type*, you should concretize and build the package twice. No need to try to shoe-horn in more than one *link-type* in a single build. Of course some libraries WILL do that for you, it's just a feature we don't need to use in Spack. Sometimes you will be unnecessarily building a library twice; that's not a big problem either.

[davydden]

I think we all agree that we shall use a single variant which encodes all options. The main thing to decide is wether we allow it to be multiple choices (shared+static) or a single choice (one of shared, static, and possibly pic).

@scheibelp :

a single-valued variant with the possible values shared, static, and static_and_shared.

i am not in favour of this. If one wants to mix shared and static in one build, that's what multivalued variants for. Having static_and_shared and shared will require having all depends_on() doubled. Let's not do that.

@citibeth :

No need to try to
shoe-horn in more than one link-type in a single build.

I am ok with that. We can always revise those things if we find a specific DAG of packages which needs to have both shared and static build of one of its nodes.

single 3-valued variant (call it link-type): static, shared,
pic

I like what @citibeth suggests, specifically approach it from

how you intend to link

perspective.

In this case, maybe we only need two values: shared and static, meaning that shared will result in a package build which can be used when building shared libraries. If a package can build shared libs, then that what it should do for + shared . Otherwise it should do instead static with pic to satisfy + shared variant. Whereas static is static libraries with or without pic (that should not matter downstream).
Alternative is to name shared value dynamic.

[citibeth]

There are two issues here: how are link-type variants encoded in specs and variants, and how will our packages make use of those variants? *How Variants are Encoded?* Let's suppose for now that we want packages to build more than one link type; do we use single or multiple variants? We need to think through how this affects things. We currently have three approaches: 1. Separate "traditional" on/off variants, one each for shared, static, pic. 2. One single-valued variant that can be either 'shared', 'static' or 'pic'. 3. One multi-valued variant that equals a subset of {shared, static, pic}. Suppose I have a package that `depends_on('foo+static')`. How will the choices above affect concretization? (1) I will be able to use foo+static, foo+static+shared, foo+static+pic or foo+static+shared+pic (2) Will I be able to use `foo link-type={static,shared}`? Or `foo link-type={static,shared,pic}`? If so, then this is semantically equivalent to (1), and we should choose whichever one we like better. If not, then (1) should be preferred over (2). (3) I will be able to use only `foo link-type=static`. The assumption is that other link-types would been built under other specs, if we want those link-types. *How Many Link-Types In A Build?* Once we've decided how to encode link types in specs and variants, we need to decide, fundamentally, whether we want a Spack build to produce more than one link type. Pros and cons of this: *Option 1: Allow Spack to build more than one link-type at once:* Pros: Some underlying builds can natively produce more than one link type. If the user wants more than one link type build, then using this capability will reduce unnecessary re-building. Some underlying builds ALWAYS build more than one link type, and this option would make use of that. Cons: we would have to go through gymnastics with every package to get the multiple link types; and the gymnastics code could come to dominate our codebase. For example, CMake natively builds only one link-type. CMakePackage would need to run the underlying CMake and build multiple times. That gymnastics would be hidden from end-user view (but would need to be maintained). For Makefile-based packages, that code would all have to be in the main recipe, complicating everyone's life. And supposing a package does NOT allow more than one link-type per spec? (As some surely won't just because of how long it will take to convert everything). *Option 2:* *Spack builds only one link-type per spec*. The big advantage here is that many/most packages do NOT know how to build more than one link-type, and we won't have to force them to change. This option, while it might take a little more build-time here and there, is vastly simpler to implement and manage. I strongly believe we should do Option 2. In the end, it's easier to ignore the "extra" capability of some builds to build more than one link-type, than to "add in" that capability to the majority of builds that don't do it. ==============

@scheibelp <https://github.com/scheibelp> : a single-valued variant with the possible values shared, static, and static_and_shared. i am not in favour of this. If one wants to mix shared and static in one build, that's what multivalued variants for. Having static_and_shared and shared will require having all depends_on() doubled. Let's not do that.

I agree with @scheielp here; see above for why.

@citibeth <https://github.com/citibeth> : No need to try to shoe-horn in more than one *link-type* in a single build. I am ok with that. We can always revise those things if we find a specific DAG of packages which needs to have both shared and static build of one of its nodes.

That *is* an issue I hadn't thought of. How realistic do we think it is? In general, I was assuming one would just set `all: link-type=...` in `packages.yaml` and forget about it from there on.

In this case, maybe we only need two values: shared and static,

In a past conference call, I surmised that we need `pic` too. Some people want/need that control.

[davydden]

a single-valued variant with the possible values shared, static, and
static_and_shared.

i am not in favour of this. If one wants to mix shared and static in one
build, that's what multivalued variants for. Having static_and_shared and
shared will require having all depends_on() doubled. Let's not do that.

I agree with @scheielp here; see above for why.

I think you misunderstood my point. If there is single valued variant which can take one of {shared,static,shared_and_static} then the same choice can be encoded in a multi-valued variant {shared,static} where you need to choose at least one. But maintaining the former is a PITA as you would do

depends_on(foo+shared, when=....)
depends_on(foo+shared_and_static, when=...)

for each and every case instead of just

depends_on(foo link=shared, when=....)

which can be satisfied by both foo link=shared or foo link=shared,static.

p.s. I am not insisting that we need to adopt link={shared,static}, I am ok having link={shared,static,pic}.

[citibeth]

On Sat, Sep 16, 2017 at 1:25 PM, Denis Davydov ***@***.***> wrote: a single-valued variant with the possible values shared, static, and static_and_shared. i am not in favour of this. If one wants to mix shared and static in one build, that's what multivalued variants for. Having static_and_shared and shared will require having all depends_on() doubled. Let's not do that. I agree with @scheielp here; see above for why. I think you misunderstood my point. If there is single valued variant which can take one of {shared,static,shared_and_static} then the same choice can be encoded in a multi-valued variant {shared,static} where you need to choose at least one.

Since we have multi-valued variants, there's no need for the former.

But maintaining the former is a PITA as you would do depends_on(foo+shared, when=....) depends_on(foo+shared_and_static, when=...) I agree, we can't be encouraging this kind of variant forwarding in

packages, which results in inflexibility and combinatorial explosion. It's hard enough as is to discourage people from this pattern (we should set defaults we want in `packages.yaml`, not directly in the `package.py` file). *for each and every case* instead of just

depends_on(foo+shared, when=....) which can be satisfied by both foo^link=shared or foo^link=shared,static. p.s. I am not saying we need to adopt link={shared,static}, I am ok having link={shared,static,pic}.

I don't think that packages that can build more than one link-type is practical or desirable anyway. Which makes this part of the discussion moot.

[scheibelp]

If one wants to mix shared and static in one build, that's what multivalued variants for. Having static_and_shared and shared will require having all depends_on() doubled. Let's not do that.

Two points:

• If for example a package must always build static libs, and may additionally build shared libs, then it would support {static, static_and_shared}. That's probably not a general issue though and most packages can stick with just {static, shared}
• Also I think it's possible to add some logic to automatically propagate static/shared through dependencies so that it is not strictly required for example to say depends_on(x link=shared, when="link=shared") and depends_on(x link=static, when="link=static")

[citibeth]

On Thu, Sep 21, 2017 at 5:19 PM, scheibelp ***@***.***> wrote: If one wants to mix shared and static in one build, that's what multivalued variants for. Having static_and_shared and shared will require having all depends_on() doubled. Let's not do that.

Do you mean multi-valued variants (which are semantically almost equivalent to a bunch of boolean variants)? Or enum-valued variants?

Two points: - If for example a package must always build static libs, and may additionally build shared libs, then it would support {static, static_and_shared}. That's probably not a general issue though and most packages can stick with just {static, shared} What is the advantage of building more than one kind of library in a

single Spack build?

- Also I think it's possible to add some logic to automatically propagate static/shared through dependencies so that it is not strictly required for example to say depends_on(x link=shared, when="link=shared") and depends_on(x link=static, when="link=static") I agree, we absolutely cannot have ad-hoc variant forwarding sprinkled

throughout our code. But what's the use case for this? I'm assuming users will just set `all: link-type=xxxxxx` in their `packages.yaml` file and be done with it. Does anyone want to do anything more fine-grained than that? If so, what?

[davydden]

@scheibelp :

If for example a package must always build static libs, and may additionally build shared libs, then it would support {static, static_and_shared}.

i think you can encode this situation into a multivalued variant link={shared,static} and add a conflict statement saying that it can not have link=shared, it must be either link=shared,static or link=static. So for the sake of uniformity I would still not bother with {static, static_and_shared} approach.

[citibeth]

We still have not had a discussion, or agreed upon, whether Spack packages should even be ALLOWED to build more than one link-type at once. See email of Sep 15 detailing this issue. Until that is resolved, I think it's premature to talk about how we would encode it, IF we do allow Spack packages to build more than one link-type at a time.

…

On Thu, Sep 21, 2017 at 5:34 PM, Denis Davydov ***@***.***> wrote: @scheibelp <https://github.com/scheibelp> : If for example a package must always build static libs, and may additionally build shared libs, then it would support {static, static_and_shared}. i think you can encode this situation into a multivalued variant link={shared,static} and add a conflict statement saying that it can not have link=shared, it must be either link=shared,static or link=static. So for the sake of uniformity I would still not bother with {static, static_and_shared} approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5269 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd6p61O3qo4abChfTAuBa1bool3ppks5sktZMgaJpZM4PKtvk> .

[citibeth]

Basically, I see only a small number of ways people will use this functionality: 1. Most people will set `all: link-type=shared` in `packages.yaml`. Packages will all build shared if they can; or PIC if they cannot. Since this is how most people will do things, Spack will ship with it as the default. 2. Cray users will set `all: link-type=static` in `packages.yaml`. Packages will all build static, or PIC. For packages that cannot, users will either vent their frustration, or individually set those packges to `link-type=shared` in `packages.yaml`. 3. sysadmins might be tasked with building shared and static versions of some things. Becuase some of their users like static builds, for whatever reason. They will maintain two Spack environments/contextx/configurations: one with `all: link-type=shared`, and one with `all: link-type=static`. The two environments will produce two entirely separate sets of packages. Each user they support will use either the shared packages, or the static packages. *Can anyone provide any additional use cases for this functionality?* I'd like to know what they are before we tie ourselves in knots providing complicated flexibility or optimization that nobody wants or needs.

[markcmiller86]

Regarding @becker33 definition of common variants...

• variants that happen to exist on many packages
• should probably ensure they have same meanings, defaults, spellings, etc.
• I dunno if ensure is satisfied by a manual PR checklist or by some automated tool. The latter would be better, of course.
• would maybe have to decide when a variant is frequent enough to be deemed common and then require this additional level of management
• Is there no hope of having a Spack-proper mechanism for defining to common variants and then ways for any package to subscribe to them? I mean something like common_variants.py file somewhere in Spack proper and then common_variant('examples') for package.py file to declare?

Regarding @becker33

Standard 1. This is what we (mostly) currently do.
All packages that can build either static or shared libraries have a variant shared. This variant defaults to True. When True, we build static and shared libraries. When false, we build only static libraries.

Just curious but how is this enforced now? In particular do all current instances indeed build both static and shared when +shared and only static with ~shared? And, what happens with pic in the latter cases? Is a separate +pic required to get static libs that can nonetheless be used in a shared link?

@becker33... What is the rationale for the con of not being able to build only shared? Is this is a disk-space issue? Time to build issue? Conflagration issue with concretization or something? Turns out, the flip side of this...not being able to build just static I am more familiar with. And, I typically arrive at this need because compilers tend to favor linking shared and then cannot due to some lib (often some darn system lib I have no control over) having been compiled without pic.

I think I like @citibeth single valued variant, link-type with choices shared, static, pic but I am not sure I understand the rationale for the name link-type? I mean I tend to think of the link step only when I am building an executable (e.g. application) and so do not immediately connote link-type with creation of a library (shared or static). If I understand it though, setting link-type say to pic means I can build all static libs but nonetheless use those libs in an eventual shared object if I need to. It seems like this variant sort of slices (think python array slicing) the spack-managed install point into objects that match one of these three criteria and feels right to me.

But, don't we already have essentially universal (@becker33 verbiage) support for pic using the picflag setting on a package? If so, why collapse that into the proposed link-type variant?

I also think a multi-valued variant with values shared and static (what about pic here?) feels better than a single-valued variant with shared, static and shared_and_static (and, again, what about pic here?)

[citibeth]

- Is there no hope of having a Spack-proper mechanism for defining to *common* variants and then ways for any package to *subscribe* to them? I mean something like common_variants.py file somewhere in Spack proper and then common_variant('examples') for package.py file to declare?

Right now, common variants are enforced purely as a matter of convention. They can be set using the `all:` entry in `packages.yaml`. If I set `all: foo=bar` in `packages.yaml`, then the variant `foo` will be set to `bar` for every package that accepts a `foo` variant. This is quick-and-dirty, but a WHOLE lot easier than any variant forwarding proposal we've come up with in the last 18 months. And quite useful too, especially when combined with Spack contexts, environments, configurations, etc. We still need a way to set common variants on the command line: i.e. an equivalent to `all:foo=bar` that can be written on the command line. In general, anything you can do in `packages.yaml` should be available on the command line (unfortunately, it is not). Someone should find a way to fix that, rather than continuing to work around its lack.

Regarding @becker33 <https://github.com/becker33> Standard 1. This is what we (mostly) currently do. All packages that can build either static or shared libraries have a variant shared. This variant defaults to True. When True, we build static and shared libraries. When false, we build only static libraries. Unfortunately, it's not so simple or clean. Currently, we have boolean

variants called `shared`, `static` and `pic`. There is no consistency. The best that can be said of them is: (1) Everything builds either `shared` or `pic` by default, making life easy for people who like shared libraries (i.e most users). (2) If you say `+shared` (and the package accepts it), you will get a shared library. If you say `+static` (and the package accepts it), you will get a static or static-PIC library. If you say `+static+pic` (and the package accepts it), you will get a static-PIC library. `+pic` without `+static` isn't defined; the `+pic` might just be ignored then (all shared libs are PIC anyway). (3) In some cases, you may get additional libraries you didn't ask for. Spack packages currently build AT LEAST what you asked for, but they might build more.

Just curious but how is this *enforced* now?

It's not.

In particular do all current instances indeed build *both* static and shared when +shared and *only* static with ~shared?

No, there is no consistency here. They will build shared if they get `+shared`. There are no guarantees of what they will NOT build if they get `~shared`; they might build the shared library anyway. There are practical reasons for this. Some upstream builds aren't very flexible about what they can and cannot build. Some builds simply build a shared and a static library, and there's nothing you can do to change that. Some are more flexible, but can only build one type at a time.

And, what happens with pic in the latter cases? Is a separate +pic required to get static libs that can nonetheless be used in a shared link?

But, don't we already have essentially *universal* (@becker33

<https://github.com/becker33> verbiage) support for pic using the picflag setting on a package? If so, why collapse that into the proposed link-type variant?

Currently, yes; you need separate `+pic` if you want to link with shared libraries. That's one of the things we want to fix: the current `pic` and `static` boolean variants have 4 settings between them, but only three make sense (`+pic~static` makes no sense). Collapsing into the `link-type` variant would eliminate the possibility of nonsensical variant combos. Collapsing also simplifies the whole mess. Currently, there are 8 ways to set 3 variants, and 2 of them make no sense. And there is no consistency about what is or is not built, for any particular setting of those 3 variants. The proposal is to pare it down to just 3 possible settings, which more simply and clearly expresses what 99% of users want. It's easier to define what should happen in each of those 3 cases (see below), and it's easier to implement as well (see below).

@becker33 <https://github.com/becker33>... What is the rationale for the *con* of not being able to build *only* shared?

I think you mean the con on Option 1? (Correct me if I'm wrong; I'm confused with the multiple negatives in these sentences). Suppose we make a multi-valued variant `link-type = subset of {static, shared, pic}`. In that case, there are 8 different ways `link-type` could be set, specifying up to 3 different versions of a library that any single Spack recipe would have to build. The problem is, upstream builds aren't infinitely flexible. So let's think of how we would implement such a directive. * I believe CMake projects allow you to specify ONE of those three options. Now, instead of just doing a straight CMake build, `CMakeProject` would have to run the upstream build up to three times, and then merge the resulting trees into one. We currently have a few recipes that run the upstream build more than once for various reasons, and it's never fun. We have no way to merge multiple CMake builds of the same thing into one resulting Spack tree. And that gets us into dangerous territory, in which the end product of Spack is different from what the upstream build produced (and maybe wrong). Furthermore, this could break stuff like `spack setup`. * Not all packages are CMake-based. We'd have to do the same stuff for `AutotoolsPackage`, of course. But that's just the beginning. Many packages have weird hand-written Makefiles. Each one of them would need to be updated to run the upstream build up to 3 times (except for the hand-written builds that can generate multiple link-types at once). Each one of these would be as much work as getting `CMakePackage` and `AutotoolsPackage` to build multiple link-types. This would be a LOT of work. And to what end? * Some packages aren't flexible about what they build. If we specify that a build CANNOT contain anything not requested, then we'd also have to update our recipes to deleted unwanted stuff that happens to get built anyway. Again, mucking with what the upstream build produced; which is complicated and error-prone. Due to these practical constraints, I believe our best bet is: 1. Single-valued variant link-type = static, shared OR pic. 2. Spack guarantees that something compatible the requested `link-type` (see below) is built; other link-types not requested may also be built. 3. If a package cannot build something compatible with the requested `link-type`, it fails at concretization time. 4. No `link-type` forwarding between packages (or else we'd be forwarding every single dependency)

Turns out, the flip side of this...not being able to build *just* static I am more familiar with. And, I typically arrive at this need because compilers tend to favor linking shared and then cannot due to some lib (often some darn system lib I have no control over) having been compiled without pic.

On most computers, if you request static and get shared, it's no big deal. But the main reason we want people to be able to request static is for Crays, where you HAVE to build everything static. Therefore, a request for `link-type=static` CANNOT be satisfied with a shared libarary. Cray users who HAVE to build static can't use packages that can ONLY build shared. That's too bad, but at least Spack (hopefully) isn't contributing to it.

I think I like @citibeth <https://github.com/citibeth> single valued variant, link-type with choices shared, static, pic but I am not sure I understand the rationale for the name *link-type*?

A common proposal is `build-type`. Unfortunately, there are two orthogonal concepts one might call `build-type`: 1. static / shared / pic 2. debug / release / relase-with-debugging-info I propose that we call (1) link-type and (2) build-type. Anyone with a better name for (1), please suggest it.

If I understand it though, setting link-type say to pic means I can build all static libs but nonetheless use those libs in an eventual shared object if I need to.

`link-type=pic` means the library MUST be compatible with shared libraries, and SHOULD be static if possible. Therefore, a Spack package with `link-type=pic` should produce one of the following (in order of decreasing preference): 1. static library with -PIC flag. 2. shared library Similarly, `link-type=shared` means the library MUST be compatible with shared libraries, and SHOULD be shared if possible. That would be either: 1. shared library 2. static library with -PIC flag. Finally, `link-type=static` means the library MUST be compatible with static linking. That would be either: 1. shared library (no PIC flag) 2. static library with -PIC flag.

I also think a multi-valued variant with values shared and static (what about pic here?) feels better than a single-valued variant with shared, static and shared_and_static (and, again, what about pic here?)

I think that *trying to make packages reliably build more than one link-type at a time is asking for a world of pain* (see above). If anyone disagrees with me, please focus on how the packages would implement this functionality, rather than how we would specify the functionality in variants. That's where the real work is.

[scheibelp]

But what's the use case for [complex logic to propagate link-types]? I'm assuming users will just set all: link-type=xxxxxx in their packages.yaml file and be done with it. Does anyone want to do anything more fine-grained than that? If so, what?

I think the primary use case for more-fine-grained setting of link types is that build/run dependencies don't need to match the link type of their parent.

I think that trying to make packages reliably build more than one link-type at a time is asking for a world of pain (see above).

we would have to go through gymnastics with every package to get the multiple link types; and the gymnastics code could come to dominate our codebase. For example, CMake natively builds only one link-type. CMakePackage would need to run the underlying CMake and build multiple times.

Even if the standardized variants are general enough to allow building both shared and static, packages should not have to support that. For example: link-type could be a multi-valued variant in general and any package could override it to be a single-valued variant; in particular CMakePackage could do that once for all CMake-based packages.

[scheibelp]

Given the combinations of pic/shared/static (and moreover which ones are valid) I think that this might be better represented with two single-valued variants:

shared={none, pic, no-pic}
static={none, pic, no-pic}

[scheibelp]

[one option is that] Spack guarantees that something compatible with the requested link-type (see below) is built; other link-types not requested may also be built.

Can a dependent have trouble building against a library if both the shared and static libs are present? If so IMO it is worthwhile to avoid building "extra" libraries that aren't obvious from the spec.

[citibeth]

But what's the use case for [complex logic to propagate link-types]? I'm > assuming users will just set all: link-type=xxxxxx in their packages.yaml file > and be done with it. Does anyone want to do anything more fine-grained than > that? If so, what? I think the primary use case for more-fine-grained setting of link types is that build/run dependencies don't need to match the link type of their parent.

Every case of variant forwarding I've seen so far has been, IMHO, a mistake. Usually it's people who want to set up defaults on how their packages will build; defaults that would be better encoded in `packages.yaml` or bundle packages. Every case I've seen reduces flexibility, and creates cases where valid specs will no longer concretize. So far, these cases have all been on packages that are used by only a small number of people, so no one runs into the buggy cases. For this reason, I am *extremely* skeptical of solutions that advocate variant forwarding. Anyway... can you provide any real-world use cases where this fine-grained control is desired? And can you make a compelling argument that this should be *encoded in the package.py file*, rather than set by the user who wants this mixed system in their *packages.yaml* file? I think that *trying to make packages reliably build more than one

link-type at a time is asking for a world of pain* (see above). we would have to go through gymnastics with every package to get the multiple link types; and the gymnastics code could come to dominate our codebase. For example, CMake natively builds only one link-type. CMakePackage would need to run the underlying CMake and build multiple times.

Even if the standardized variants are general enough to allow building both

shared and static, packages should not have to support that. For example: link-type could be a multi-valued variant in general and any package could override it to be a single-valued variant; in particular CMakePackage could do that once for all CMake-based packages.

I think the key detail here is whether the variant means that Spack should build *(A)* a *library of that link-type*, or whether Spack should build *(B)* a *library compatible with that link-type*. We need more discussion of the pros and cons of these two approaches. We need to think through typical user scenarios. The multi-valued variant proposal is semantically equivalent (or similar) to having 3 boolean variants, which we could call *shared*, *static* and *pic*. Kind of like what we have today, except the *pic* variant would be a separate link-type, rather than something entwined with *static*. ***The first problem is, packages won't be able to build *static* and *pic* together, because the resulting libraries would need to reside in files of the same name. Therefore, my bet is that NOBODY is able to build these two link-types simultaneously. Glossing over that issue... let's think more about option *(A)* above. I want to do static linkage, so I set *all:+static* in my *packages.yaml* file. There's some package *foo* that only does static linkage, no shared. It goes ahead anyway and builds *foo+shared*, ignoring the *+static* variant, since it doesn't use the static variant. Then bad things happen when you actually try to link. OK, this problem can be fixed. Either (a) we use multi-valued variants and they happen to have the concretization semantics we want (not sure if they do), or (b) we declare that *all packages that use any kind of link-type must declare all three variants; and then use lock-outs to make certain combinations invalid*. With that fix, let's suppose I want to build with shared libs; so I put *all:+shared* in my *packages.yaml* file. Package *bar* doesn't do shared. With the fix above, *bar+shared* will cause a concretization error. Which means I will have to manually specify *bar~shared+pic* in my *packages.yaml* for every package that doesn't *actually* build shared libraries. This will be a PITA. I don't see a way to make option *(A)* above "just work". Users will always have to be aware of which link-types the packages they use can generate; but frankly, I don't think anyone cares that LAPACK is a static-only library. Option *(B)* sets ups a system in which users can simply set their preferred link-type and then forget about it. IF users want to forward variants, then option *(B)* sets up a system that would seem most logical for that task.

Given the combinations of pic/shared/static (and moreover which ones are valid) I think that this might be better represented with two single-valued variants: shared={none, pic, no-pic} static={none, pic, no-pic}

`shared=no-pic` makes no sense. All shared libs are PIC. Assuming this issue is fixed, you are proposing:

shared=True/False static={none, pic, no-pic}

This would solve the problem *** above; but it leads to an even more complex, fine-grained system that every Spack user will have to be familiar with. Again... what is the purpose of all this fine-grained control. Most people never want to think about shared vs. static. Most people just want things to build shared if possible, and pic otherwise.

On Thu, Sep 21, 2017 at 10:28 PM, scheibelp ***@***.***> wrote: [one option is that] Spack guarantees that something compatible with the requested link-type (see below) is built; other link-types not requested may also be built. Can a dependent have trouble building against a library if both the shared and static libs are present? If so IMO it is worthwhile to avoid building "extra" libraries that aren't obvious from the spec.

Some upstream packages build "extra" libraries. Whatever issues its dependents have using those libraries have already been resolved. So IF there ever were a problem, there's nothing for Spack to fix. However... if we take an upstream package that natively builds just one link-type, and modify it to build multiple link-types, then we could conceivably create a problem for upstream packages that assumed they would find only one link-type. The moral of the story is, the less we mess with what the upstream packages do, the better. We shouldn't be adding or deleting files that the upstream build created.

[scheibelp]

However... if we take an upstream package that natively builds just one link-type, and modify it to build multiple link-types, then we could conceivably create a problem for upstream packages that assumed they would find only one link-type.

My goal is not to force packages to support functionality that is implied by variant settings or combinations of them. My goal is to force the variants to accurately represent what the package builds. If a package is building extra libraries, I want it to be reflected in the variants; if it always wants to build the shared libraries for example then I don't think it should be possible set shared=False in the spec.

(still thinking on the other parts)

[alalazo]

I am joining the party late, and I see that there's much fun going on 😄

Shared vs. static

I quite like @becker33 second proposal (have a common variant libraries - or libs as people will probably type this a lot - with two values shared and static). As @scheibelp noted variants can be overridden, so we can choose on a fine scale (i.e. package by package) what should be a single-valued variant and what should be a multi-valued variant. For weird cases we can use a validator function, like it is done in mvapich2 for process managers.

Position independent objects

The "picness" of the objects instead should probably be modeled with a universal boolean variant. And by universal here I really mean a single switch that can be turned on or off for all the packages at the same time. Said otherwise that should be a single global property attached to the entire DAG, not multiple and equally valued properties attached to each node. And it should enter into the hashing mechanism. I don't think we currently have support for that.

I think this property should matter only for the libs=static part - see below for details

Role of platforms

Reading the discussion I must admit my ignorance on Cray: is it impossible there to build a shared library (like it was for instance on BG/P)? If that's the case we may want to give platforms the power to inject constraints on specs. For instance a fictitious BG/P platform should be able to inject:

conflicts('libs=shared', msg='Shared libraries are not supported on BG/P')

on all specs that need it. In case, this is also a feature we currently don't have.

A few random thoughts on previous comments

Given the combinations of pic/shared/static (and moreover which ones are valid) I think that this might be better represented with two single-valued variants:

shared={none, pic, no-pic}
static={none, pic, no-pic}

Am I right saying that shared=no-pic doesn't actually exist? I ask because I lack a formal knowledge of how an OS works, but in my naive view if the code is not position independent it cannot be assigned addresses at load-time -> it cannot be shared. The case shared=none && static=none also makes no sense to me. That's what leads me to think that the model above is a better fit.

We need more discussion of the pros and cons of these two approaches. We need to think through typical user scenarios.

I would be for the opposite approach: let's go to the trenches! Out of the metaphor I think we should:

1. select a few representative / problematic / complex packages
2. distill a few sensible solutions from this discussion
3. sketch all the solutions out on the packages at point 1. in different branches

I'll start proposing a few nominees: trilinos, petsc and dealii.

[davydden]

I quite like @becker33 second proposal (have a common variant libraries - or libs as people will probably type this a lot - with two values shared and static)

👍

so we can choose on a fine scale (i.e. package by package) what should be a single-valued variant and what should be a multi-valued variant

👍

I would be for the opposite approach: let's go to the trenches!

fair point. petsc would probably be the easiest as it has fewer deps, whereas dealii itself depends on petsc and trilinos and from that perspective is more complex.

[adamjstewart]

I haven't been following this discussion closely, but I'm fine with any solution as long as it is consistent across every package. We should be handling this at the level of AutotoolsPackage and CMakePackage if possible. That will remove a lot of complexity from individual packages.

[citibeth]

On Fri, Sep 22, 2017 at 5:09 PM, Massimiliano Culpo < ***@***.***> wrote: I am joining the party late, and I see that there's much fun going on 😄 Shared vs. static I quite like @becker33 <https://github.com/becker33> second proposal (have a *common variant* libraries - or libs as people will probably type this a lot - with two values shared and static).

Are you suggesting we add new functionality to Spack variants, or do we already have what we need? Currently, you can set `all:` in `packages.yaml`, and any variant set there will be set on any package that takes it. I admit this functionality is rather rudimentary, but maybe it's sufficient. Also note that plenty of packages don't have a link-type. So I don't think we should enforce that a particular variant appears on EVERY package. As @scheibelp <https://github.com/scheibelp> noted variants can be

overridden, so we can choose on a fine scale (i.e. package by package) what should be a single-valued variant and what should be a multi-valued variant. For weird cases we can use a validator function, like it is done in mvapich2 for process managers.

The fundamental choice we need to make here is whether the variant specifies (a) *how the package is built*, or (b) *what kinds of linking it's compatible with*. I'm leaning toward (b) because I think it will make user's lives easier. But (a) also has merits.

Position independent objects The "picness" of the objects instead should probably be modeled with a *universal boolean variant*. And by universal here I really mean a single switch that can be turned on or off for all the packages at the same time.

Do you mean like the `all:` functionality in `packages.yaml`? Do you think that *shared/static* should also be modeled this way?

Said otherwise that should be a single global property attached to the entire DAG,

What about DAGs that don't involve any compiled code?

I think this property should matter only for the libs=static part - see below for details

At one of the telcons, someone said they wanted static-PIC as an available option. Otherwise, I'd just suggest we use *shared* or *static.*

Role of platforms Reading the discussion I must admit my ignorance on Cray: is it impossible there to build a shared library (like it was for instance on BG/P)?

I believe it's possible to use shared libs; but if you do, your program won't run in parallel like it needs to. The login nodes are different from the processing nodes. A few random thoughts on previous comments

Given the combinations of pic/shared/static (and moreover which ones are valid) I think that this might be better represented with two single-valued variants: shared={none, pic, no-pic} static={none, pic, no-pic} Am I right saying that shared=no-pic doesn't actually exist?

Correct. What's wrong with `link-type={static, shared, pic}`. There are only three ways to build something.

We need more discussion of the pros and cons of these two approaches. We need to think through typical user scenarios. I would be for the opposite approach: let's go to the trenches! Out of the metaphor I think we should:

That's what I meant.

1. select a few representative / problematic / complex packages 2. distill a few sensible solutions from this discussion 3. sketch all the solutions out on the packages at point 1. in different branches I'll start proposing a few nominee: trilinos, petsc and dealii.

I think this is a good way forward. I'm hoping for two general categories of options to be tried out here: (a) link-type (or something like it) specifies how the library was built. i.e. shared means shared. (b) link-type specifies what the library is compatible with. i.e. shared means shared; unless that was not possible, in which case it means static+pic

[davydden]

At one of the telcons, someone said they wanted static-PIC as an available
option. Otherwise, I'd just suggest we use shared or static.

until someone in this discussion raises concerns and explains why we really need a separate static build without pic, let's not overcomplicate the discussion and assume that we have static always built with pic. That, by the way, means that the only global setting is:

all:
  link: static

for Cray and alike. Whereas for standard systems when Spack builds shared libs, they will work with both shared and static with pic dependencies. So we don't need to enforce anything here nor forward variants in anyway. I think it would be a good balance between simplicity+robustness and complexity+features.

[citibeth]

Because in theory at least, non pic static code has cheaper function calls. That is important to some people in HPC.

…

On Sep 23, 2017 1:39 AM, "Denis Davydov" ***@***.***> wrote: At one of the telcons, someone said they wanted static-PIC as an available option. Otherwise, I'd just suggest we use *shared* or *static.* until someone in this discussion raises concerns and explains why we really need a separate static build without pic, let's not overcomplicate the discussion and assume that we have static always built with pic. That, by the way, means that the only global setting is: all: link: static for Cray and alike because when Spack is building shared libs, they will work with both shared and static with pic. So we don't need to enforce anything here nor forward variants anyway. I think it would be a good balance between simplicity+robustness and complexity+features. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5269 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1cd7q2cqNfVBClWbxQcb01L0pCJ7s3ks5slJmcgaJpZM4PKtvk> .

[davydden]

Thanks, I was not aware of that.

…

23 сент. 2017 г., в 13:29, Elizabeth Fischer ***@***.***> написал(а): Because in theory at least, non pic static code has cheaper function calls. That is important to some people in HPC.

[bartlettroscoe]

NOTE: CMake by default can only build shared or static libraries, not both. Yes it is true you could configure two different build directories (one for shared and one for static) but that seems like a waste and you are asking for trouble installing these into the same installation directory.

Newer CMake support mulit-configuration builds but that requires usage of build generators and a lot of CMake products are not fully complaint with that yet.

[alalazo]

Discussion revamped in #23474, where having 2 boolean variants is proposed +shared+static for a few libraries.