Use virtual packages to model how software is used (vs. how it is built or deployed)

[alalazo]

Wrapping up a few thoughts exchanged in a meeting about how to handle packages that bundle many components. Examples of such packages are intel-parallel-studio, boost, cray-libsci and llvm.

Requirements

A few facts about the packages we want to model follows:

• There are both binary/proprietary packages (e.g. intel-parallel-studio, cray-libsci) and packages that can be built from sources (e.g. boost or llvm)
• Dependents of these packages might need to use only a subset of the features that are provided (e.g. a package might need to link only Boost Graph, or use just the MKL portion of Intel Parallel Studio)
• There are conflicting features that can't be used together (e.g. we can't link threaded MKL together with non-threaded MKL, though both are provided by the same intel-parallel-studio package)
• There are other features that instead can be used together (e.g. different dependencies may ask for separate components of Boost, for instance Boost Graph and Boost Hana)
• There is in general a need to distinguish in our model how a software is built/deployed vs. how it is used

Related PRs or Issues

• Boost: boost : list of libraries not updated #1486 Variants might be conditional on the version of the package #9740 Make boost minimal and composable #22303
• Intel: Add linking choices for intel-oneapi-mkl #22621
• Selection of virtual providers: Finer selection of virtual providers #15569 Finer selection of virtual providers in a DAG #15443

Proposed solution

A solution proposed to deal with the requirements above is to extend the capabilities of virtual packages by:

• Giving a more programmatic interface to all of them (currently they are mostly declared implicitly)
• Permit the use of the variant and conflicts directives for virtual packages
• Unify the constraints on virtual packages like we do for normal packages
• Recording each requirement on virtual dependencies as an edge property in the concretized DAG
• Permit to declare virtual packages either in their own package.py file (for interfaces that have many different implementations) or alternatively in the same package.py where the only provider is declared (for packages that bundle components that can be picked singularly)
• Permit to have virtual packages with the same name of a regular package

At a high level this approach will go in the direction of separating how a software is used (virtual packages) from how a software is built or deployed (regular packages).

Use Case: Intel Parallel Studio

This sketched example wants to show how we can deal with conflicting features that can't be used together. Given this metadata in the package files:

# var/spack/repos/builtin/packages/intel-parallel-studio/package.py
class IntelParallelStudio(Package):

    # Each version points to a separate binary tarball, for provenance
    version('cluster.2020.4', ...)
    version('professional.2020.4', ...)
    version('composer.2020.4', ...)

    # No variants declared since we are always using the same binary package

    # Declaring that something is provided in the same directive implies that 
    # a request for one of the things implies being a provider of the others
    provides('mkl', 'lapack', 'blas')
    provides('mkl threads=openmp', 'lapack threads=openmp', 'blas threads=openmp')
    provides('mkl threads=none'  , 'lapack threads=none'  , 'blas threads=none'  )

    # Methods to return linking information
    def mkl_libs(self):
        pass

    def lapack_libs(self):
        pass
    

class Mkl(VirtualPackage):
     
    # vdeps constraints are unified during concretization to catch
    # conflicts among different usages of a package in the dag
    variant('threads', default='none', values=('openmp', 'none'))

# var/spack/repos/builtin/packages/client/package.py
class Client(Package):

    depends_on('adep')

    # Not sure it's useful, but we may still add a 'requires' directive that
    # behaves like depends_on but explicitly targets only vdeps (while
    # depends_on targets vdeps first and falls back to regular packages)
    requires('mkl threads=openmp')

# var/spack/repos/builtin/packages/client/package.py
class Adep(Package):
     
    # The semantics for depends_on is to search for vdep 
    #first and real package later.
    depends_on('lapack')

by concretizing the spec:

$ spack solve client

we would get a DAG like:

client
|        \ (B)
| (A)   adep
|          | (C)
intel-parallel-studio

with the following properties:

• On edge (A) required: ['mkl threads=openmp'] and provided: ['mkl threads=openmp']
• On edge (C) required: ['lapack'] and provided: ['lapack threads=openmp']

The differences between provided and required in the edge attributes stems from additional constraints that account for non-local facts and for the properties of the provider. The provided attribute can never be less strict than the corresponding required. In this example the fact that intel-parallel-studio declared to provide three virtuals together plus the requirement from client to use the threaded version of mkl implies that adep is provided the threaded version of lapack.

Use Case: Boost

This example is to show how we can deal with packages that bundle many components:

# var/spack/repos/builtin/packages/boost/package.py
class Boost(Package):

    version('1.75.0', ...)
    version('1.74.0', ...)

    # Variant affecting how the package is built
    variant('cxxstd', default='98', values=('98', '11', '14', '17', '2a'), multi=False, ...)
    variant('debug', default=False, ...)

    # Components that are provided
    provides('boost components=json ', when='@1.75.0:')
    provides('boost components=static_string ', when='@1.73.0:')

    # Uber installation of Boost with everything installed
    def install(self, spec, prefix):
        pass


class Boost(VirtualPackage):
    # For the Boost virtual package version is not relevant, we'll refer to that of the real package
    # This means that we allow to omit the version directive in virtual packages

    # Components allowed in Boost
    variant('components', values=any_combination_of(
        'json', 'static_string', ...
    ), ... )

# var/spack/repos/builtin/packages/client/package.py
class Client(Package):
    # Asking for both components means we need to use boost version 1.75.0 or greater
    requires('boost components=json,static_string')

[alalazo]

@tgamblin @scheibelp @becker33 @trws Above there's a slightly extended write up of my notes from yesterday. Feel free to edit it to add information or evolve the proposal.

[alalazo]

Another thought. If we go with this model, and we curate the definition of which libraries (or in general which resources) are needed to use a component we can minimize deployments by creating artifacts for an end-user that strip all the unneeded components. Think for instance a minimal size final image for spack containerize.

[scheibelp]

Regarding:

    # Declaring that something is provided in the same directive implies that 
    # a request for one of the things implies being a provider of the others
    provides('mkl', 'lapack', 'blas')
    provides('mkl threads=openmp', 'lapack threads=openmp', 'blas threads=openmp')
    provides('mkl threads=none'  , 'lapack threads=none'  , 'blas threads=none'  )

Does this mean that a dependent which uses blas threads=openmp needs to use lapack threads=openmp?

In general, if this specific implementation provides mkl threads=openmp, then what about packages which provide mkl without this? How is a virtual package constructed from providers with different feature sets?

[scheibelp]

My understanding based on further discussion is that the answer is yes: provides(x, y) means that any dependent using the package for X also uses it for Y. These examples aren't necessarily based on the true needs of intel-parallel-studio (which might be able to provide just blas and another package could provide lapack). This particular detail is more relevant to the use cases in "finer selection of virtual providers" than to #22621.

[scheibelp]

We also discussed the concept of a "use-variant" as an alternative (variants which describe how a package is used rather than how it is built), for intel-parallel-studio that would look like:

class IntelParallelStudio(Package):

    # Each version points to a separate binary tarball, for provenance
    version('cluster.2020.4', ...)
    ...

    use_variant(threads, values=[openmp, none])

# var/spack/repos/builtin/packages/adep/package.py
class Adep(Package):
     
    # This contrasts with the virtuals example: if you want a feature from a
    # specific implementation, you have to name the implementation rather than
    # the virtual
    depends_on('intel-parallel-studio threads=openmp')

[scheibelp]

Something similar to use_variant could be implemented without any changes to core Spack, although this approach would require some extra effort on the package-writer's part:

A package like intel-oneapi-mkl includes multiple versions of libraries that conflict, e.g. threaded and sequential; if any package uses the threaded versions of the libraries, then all packages should. This could be achieved by splitting intel-oneapi-mkl into intel-oneapi-mkl-base and intel-oneapi-mkl-options:

• intel-oneapi-mkl-base installs all the libraries together and has no variants
• intel-oneapi-mkl-options doesn't actually install anything but depends on intel-oneapi-mkl-base. It has variants that control which libraries are used by dependents (so for example it has a threading variant that could be none or openmp)

In a sense, the variants on intel-oneapi-mkl-options function exactly like the use_variant.

[alalazo]

I'll leave this note as a reminder for a later discussion, but I think we may need:

1. Conditions on the use variants
2. Be able to express that something applies when the package provides a certain virtual

In the case of the example above:

class IntelParallelStudio(Package):

    # Each version points to a separate binary tarball, for provenance
    version('cluster.2020.4', ...)
    ...

    provides('mpi')
    provides('mkl')

    use_variant(threads, values=[openmp, none], when=<some-way-to-say-it-provides-mkl>)

The idea is that a dependent that doesn't use the mkl portion of IntelParallelStudio shouldn't have the threads variant appearing on the connecting edge.

[alalazo]

Adding a note on the .libs query mechanism following discussion with @scheibelp yesterday. If we proceed with the approach to extend the virtual packages to let them have variants etc. the libs method needs to have information on the requestor. This information may be passed to the package with the spec.root attribute (or something very similar to it).

As an example, consider:

    A
  /    \
 /      \
B         C
  \ json  |
   \      | json,hana
    \     |
       boost

where B needs the json component of boost while C needs both json and hana. By default:

spec['boost'].libs

should return only json if the caller is B, json and hana if the caller is C and nothing if the caller is A. We may need to have a mechanism to override this default in certain cases, so we can think of repurposing the query parameters for that:

spec['boost:components=hana,json,static_string'].libs

[cessenat]

I Think this is what I need too for the use case of fftw-api where I wrongly expected it to transmit the variant to the concrete package:

depends_on('fftw-api@3:~mpi')

$ spack spec mycode ^fftw
Concretized
--------------------------------
mycode@1.2.4%gcc@10.3.0~mpi arch=linux-ubuntu21.04-x86_64_v2
    ^fftw@3.3.9%gcc@10.3.0+mpi~openmp~pfft_patches precision=double,float arch=linux-ubuntu21.04-x86_64_v2
        ^openmpi@4.1.1%gcc@10.3.0~atomics~cuda~cxx~cxx_exceptions+gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~pmi~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=none schedulers=none arch=linux-ubuntu21.04-x86_64_v2
... bla bla bla ...

spack debug report

• Spack: 0.16.2-4553-b96d836a72
• Python: 3.9.5
• Platform: linux-ubuntu21.04-x86_64_v2
• Concretizer: clingo

So, I'm very interested.