From d4c25afc94adc4973105eca4f567316d90c26447 Mon Sep 17 00:00:00 2001 From: Harmen Stoppels Date: Tue, 27 May 2025 23:32:52 +0200 Subject: [PATCH 1/2] jules wip --- tutorial_advanced_packaging.rst | 195 ++++++++--------- tutorial_basics.rst | 77 ++++--- tutorial_binary_cache.rst | 151 +++++++------ tutorial_buildsystems.rst | 231 +++++++++++--------- tutorial_configuration.rst | 113 +++++----- tutorial_developer_workflows.rst | 326 ++++++++++++++++------------ tutorial_environments.rst | 101 +++++---- tutorial_modules.rst | 249 +++++++++++---------- tutorial_packaging.rst | 360 ++++++++++++++++--------------- 9 files changed, 982 insertions(+), 821 deletions(-) diff --git a/tutorial_advanced_packaging.rst b/tutorial_advanced_packaging.rst index 72e2957dc9..4a9fbbb399 100644 --- a/tutorial_advanced_packaging.rst +++ b/tutorial_advanced_packaging.rst @@ -11,17 +11,17 @@ Advanced Topics in Packaging ============================ -Spack tries to automatically configure packages with information from -dependencies such that all you need to do is to list the dependencies -(i.e., with the ``depends_on`` directive) and the build system (for example -by deriving from :code:`CmakePackage`). - -However, there are many special cases. Often you need to retrieve details -about dependencies to set package-specific configuration options, or to -define package-specific environment variables used by the package's build +Spack tries to automatically configure software packages with information from +their Spack package dependencies. Ideally, you only need to list the +dependencies (using the ``depends_on`` directive) and specify the build system +(for example, by deriving from :code:`CmakePackage`). + +However, there are many special cases. Often, you need to retrieve details +about dependencies to set package-specific configuration options or define +package-specific environment variables used by the software's build system. This tutorial covers how to retrieve build information from -dependencies, and how you can automatically provide important information to -dependents in your package. +dependencies and how you can automatically provide important information to +dependent packages from your package. ---------------------- Setup for the Tutorial @@ -32,8 +32,8 @@ Setup for the Tutorial We do not recommend doing this section of the tutorial in a production Spack instance. -The tutorial uses custom package definitions with missing sections that -will be filled in during the tutorial. These package definitions are stored +This tutorial uses custom package definitions with missing sections that +will be filled in during the exercises. These package definitions are stored in a separate package repository, which can be enabled with: .. code-block:: console @@ -41,16 +41,16 @@ in a separate package repository, which can be enabled with: $ spack repo add --scope=site var/spack/repos/tutorial This section of the tutorial may also require a newer version of -gcc. If you have not already installed gcc @7.2.0 and added it to your +GCC. If you have not already installed GCC @7.2.0 and added it to your configuration, you can do so with: .. code-block:: console - $ spack install gcc@7.2.0 %gcc@5.4.0 + $ spack install gcc@7.2.0 %gcc@5.4.0 # This might take a while $ spack compiler add --scope=site `spack location -i gcc@7.2.0 %gcc@5.4.0` -If you are using the tutorial docker image, all dependency packages -will have been installed. Otherwise, to install these packages you can use +If you are using the tutorial Docker image, all dependency packages +will have been pre-installed. Otherwise, to install these packages, you can use the following commands: .. code-block:: console @@ -65,9 +65,9 @@ the rest of the tutorial. .. note:: Several of these packages depend on an MPI implementation. You can use - OpenMPI if you install it from scratch, but this is slow (>10 min.). + OpenMPI if you install it from scratch, but this is slow (>10 minutes). A binary cache of MPICH may be provided, in which case you can force - the package to use it and install quickly. All tutorial examples with + packages to use it and install quickly. All tutorial examples with packages that depend on MPICH include the spec syntax for building with it. .. _adv_pkg_tutorial_start: @@ -76,45 +76,47 @@ the rest of the tutorial. Modifying a Package's Build Environment --------------------------------------- -Spack sets up several environment variables like ``PATH`` by default to aid in -building a package, but many packages make use of environment variables which -convey specific information about their dependencies (e.g., ``MPICC``). -This section covers how to update your Spack packages so that package-specific -environment variables are defined at build-time. +Spack sets up several environment variables, like ``PATH``, by default to aid +in building a software package. However, many software packages make use of +environment variables that convey specific information about their +dependencies (e.g., ``MPICC`` for an MPI library). This section covers how to +update your Spack packages so that such package-specific environment +variables are defined at build time. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Set environment variables in dependent packages at build-time ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Dependencies can set environment variables that are required when their -dependents build. For example, when a package depends on a python extension -like py-numpy, Spack's ``python`` package will add it to ``PYTHONPATH`` -so it is available at build time; this is required because the default setup -that spack does is not sufficient for python to import modules. - -To provide environment setup for a dependent, a package can implement the -:py:func:`setup_dependent_build_environment +dependent packages build. For example, when a package depends on a Python +extension like ``py-numpy``, Spack's ``python`` package will add the extension's +path to ``PYTHONPATH`` so it is available at build time. This is required +because the default setup that Spack performs is not always sufficient for +Python to import modules from dependencies. + +To provide environment setup for a dependent package, a Spack package can +implement the :py:func:`setup_dependent_build_environment ` -and or :py:func:`setup_dependent_run_environment -` functions. -These functions take as a parameter a :py:class:`EnvironmentModifications -` object, which includes -convenience methods to update the environment. For example, an MPI -implementation can set ``MPICC`` for build-time use for packages that -depend on it: +and/or :py:func:`setup_dependent_run_environment +` methods. +These methods take an :py:class:`EnvironmentModifications +` object as a parameter, +which includes convenience methods to update the environment. For example, an +MPI implementation package can set ``MPICC`` for build-time use for packages +that depend on it: .. code-block:: python def setup_dependent_build_environment(self, env, dependent_spec): env.set('MPICC', join_path(self.prefix.bin, 'mpicc')) -In this case packages that depend on ``mpi`` will have ``MPICC`` defined in -their environment when they build. This section is focused on modifying the -build-time environment represented by ``env``, but it's worth noting that -modifications to the run-time environment, made through the +In this case, packages that depend on this ``mpi`` package will have ``MPICC`` +defined in their environment when they build. This section is focused on +modifying the build-time environment, represented by ``env``. However, it's +worth noting that modifications to the runtime environment, made through the :py:func:`setup_dependent_run_environment -` function's -``env`` parameter, are included in Spack's automatically-generated +` method's +``env`` parameter, are included in Spack's automatically generated module files. We can practice by editing the ``mpich`` package to set the ``MPICC`` @@ -122,7 +124,7 @@ environment variable in the build-time environment of dependent packages. .. code-block:: console - root@advanced-packaging-tutorial:/# spack edit mpich + $ spack edit mpich Once you're finished, the method should look like this: @@ -145,7 +147,7 @@ At this point we can, for instance, install ``netlib-scalapack`` with .. code-block:: console - root@advanced-packaging-tutorial:/# spack install netlib-scalapack ^mpich + $ spack install netlib-scalapack ^mpich ... ==> Created stage in /usr/local/var/spack/stage/netlib-scalapack-2.0.2-km7tsbgoyyywonyejkjoojskhc5knz3z ==> No patches needed for netlib-scalapack @@ -158,7 +160,7 @@ At this point we can, for instance, install ``netlib-scalapack`` with [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2-km7tsbgoyyywonyejkjoojskhc5knz3z -and double check the environment logs to verify that every variable was +You can double-check the environment logs to verify that each variable was set to the correct value. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -166,12 +168,10 @@ Set environment variables in your own package ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Packages can modify their own build-time environment by implementing the -:py:func:`setup_build_environment -` +method and their own runtime environment by implementing the +:py:func:`setup_run_environment ` +method. For the ``qt`` Spack package, this looks like: .. code-block:: python @@ -215,9 +215,9 @@ Let's see how it works by completing the ``elpa`` package: .. code-block:: console - root@advanced-packaging-tutorial:/# spack edit elpa + $ spack edit elpa -In the end your method should look like: +In the end, your method should look like: .. code-block:: python @@ -232,21 +232,22 @@ In the end your method should look like: spack_env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags) spack_env.append_flags('LIBS', spec['lapack'].libs.link_flags) -At this point it's possible to proceed with the installation of ``elpa ^mpich`` +At this point, it's possible to proceed with the installation of ``elpa ^mpich``: ------------------------------ Retrieving Library Information ------------------------------ Although Spack attempts to help packages locate their dependency libraries -automatically (e.g. by setting ``PKG_CONFIG_PATH`` and ``CMAKE_PREFIX_PATH``), +automatically (e.g., by setting ``PKG_CONFIG_PATH`` and ``CMAKE_PREFIX_PATH``), a package may have unique configuration options that are required to locate libraries. When a package needs information about dependency libraries, the -general approach in Spack is to query the dependencies for the locations of -their libraries and set configuration options accordingly. By default most -Spack packages know how to automatically locate their libraries. This section -covers how to retrieve library information from dependencies and how to locate -libraries when the default logic doesn't work. +general approach in Spack is to query its dependencies for the locations of +their libraries and set configuration options accordingly. By default, most +Spack packages know how to automatically locate their own libraries for +dependents. This section covers how to retrieve library information from +dependencies and how to define how libraries are located when the default +logic doesn't work. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Accessing dependency libraries @@ -277,9 +278,9 @@ the ``cmake_args`` section: .. code-block:: console - root@advanced-packaging-tutorial:/# spack edit armadillo + $ spack edit armadillo -If you followed the instructions in the package, when you are finished your +If you followed the instructions in the package, when you are finished, the ``cmake_args`` method should look like: .. code-block:: python @@ -302,15 +303,15 @@ If you followed the instructions in the package, when you are finished your ] As you can see, getting the list of libraries that your dependencies provide -is as easy as accessing the their ``libs`` attribute. Furthermore, the interface +is as easy as accessing their ``libs`` attribute. Furthermore, the interface remains the same whether you are querying regular or virtual dependencies. -At this point you can complete the installation of ``armadillo`` using ``openblas`` +At this point, you can complete the installation of ``armadillo`` using ``openblas`` as a LAPACK provider (``armadillo ^openblas ^mpich``): .. code-block:: console - root@advanced-packaging-tutorial:/# spack install armadillo ^openblas ^mpich + $ spack install armadillo ^openblas ^mpich ==> pkg-config is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkg-config-0.29.2-ae2hwm7q57byfbxtymts55xppqwk7ecj ... ==> superlu is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/superlu-5.2.1-q2mbtw2wo4kpzis2e2n227ip2fquxrno @@ -327,21 +328,21 @@ as a LAPACK provider (``armadillo ^openblas ^mpich``): Fetch: 0.01s. Build: 3.96s. Total: 3.98s. [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4 -Hopefully the installation went fine and the code we added expanded to the right list -of semicolon separated libraries (you are encouraged to open ``armadillo``'s -build logs to double check). +Hopefully, the installation went fine and the code we added expanded to the correct list +of semicolon-separated libraries (you are encouraged to open ``armadillo``'s +build logs to double-check). ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Providing libraries to dependents ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Spack provides a default implementation for ``libs`` which often works -out of the box. A user can write a package definition without having to -implement a ``libs`` property and dependents can retrieve its libraries +Spack provides a default implementation for the ``libs`` property which often +works out of the box. A user can write a package definition without having to +implement a custom ``libs`` property, and dependents can retrieve its libraries as shown in the above section. However, the default implementation assumes that -libraries follow the naming scheme ``lib.so`` (or e.g. -``lib.a`` for static libraries). Packages which don't -follow this naming scheme must implement this function themselves, e.g. +libraries follow the naming scheme ``lib.so`` (or e.g., +``lib.a`` for static libraries). Packages that don't +follow this naming scheme must implement this property themselves, e.g., ``opencv``: .. code-block:: python @@ -355,12 +356,12 @@ follow this naming scheme must implement this function themselves, e.g. This issue is common for packages which implement an interface (i.e. virtual package providers in Spack). If we try to build another version of -``armadillo`` tied to ``netlib-lapack`` (``armadillo ^netlib-lapack ^mpich``) +``armadillo`` tied to ``netlib-lapack`` (``armadillo ^netlib-lapack ^mpich``), we'll notice that this time the installation won't complete: .. code-block:: console - root@advanced-packaging-tutorial:/# spack install armadillo ^netlib-lapack ^mpich + $ spack install armadillo ^netlib-lapack ^mpich ==> pkg-config is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkg-config-0.29.2-ae2hwm7q57byfbxtymts55xppqwk7ecj ... ==> openmpi is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f @@ -391,7 +392,7 @@ customized library search logic. Let's edit it: .. code-block:: console - root@advanced-packaging-tutorial:/# spack edit netlib-lapack + $ spack edit netlib-lapack and follow the instructions in the ``# TUTORIAL:`` comment as before. What we need to implement is: @@ -400,7 +401,7 @@ What we need to implement is: @property def lapack_libs(self): - shared = True if '+shared' in self.spec else False + shared = '+shared' in self.spec return find_libraries( 'liblapack', root=self.prefix, shared=shared, recursive=True ) @@ -416,7 +417,7 @@ install ``armadillo ^netlib-lapack ^mpich``: .. code-block:: console - root@advanced-packaging-tutorial:/# spack install armadillo ^netlib-lapack ^mpich + $ spack install armadillo ^netlib-lapack ^mpich ... ==> Building armadillo [CMakePackage] @@ -429,7 +430,7 @@ install ``armadillo ^netlib-lapack ^mpich``: Since each implementation of a virtual package is responsible for locating the libraries associated with the interfaces it provides, dependents do not need -to include special-case logic for different implementations and for example +to include special-case logic for different implementations and, for example, need only ask for :code:`spec['blas'].libs`. ---------------------- @@ -442,8 +443,8 @@ Attach attributes to other packages Build tools usually also provide a set of executables that can be used when another package is being installed. Spack gives you the opportunity -to monkey-patch dependent modules and attach attributes to them. This -helps make the packager experience as similar as possible to what would +to monkey-patch dependent package modules and attach attributes to them. This +helps make the packager's experience as similar as possible to what would have been the manual installation of the same package. An example here is the ``automake`` package, which overrides @@ -458,7 +459,7 @@ An example here is the ``automake`` package, which overrides for name in executables: setattr(module, name, self._make_executable(name)) -so that every other package that depends on it can use directly ``aclocal`` +so that every other package that depends on it can directly use ``aclocal`` and ``automake`` with the usual function call syntax of :py:class:`Executable `: .. code-block:: python @@ -466,19 +467,20 @@ and ``automake`` with the usual function call syntax of :py:class:`Executable libsigsegv is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz ==> m4 is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-r5envx3kqctwwflhd4qax4ahqtt6x43a ... @@ -498,15 +500,15 @@ Let's look at an example and try to install ``netcdf ^mpich``: /usr/local/var/spack/stage/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj/netcdf-4.4.1.1/spack-build-out.txt We can see from the error that ``netcdf`` needs to know how to link the *high-level interface* -of ``hdf5``, and thus passes the extra parameter ``hl`` after the request to retrieve it. -Clearly the implementation in the ``hdf5`` package is not complete, and we need to fix it: +of ``hdf5``, and thus passes the extra parameter ``hl`` in the query to retrieve it. +Clearly, the implementation in the ``hdf5`` package is not complete, and we need to fix it: .. code-block:: console - root@advanced-packaging-tutorial:/# spack edit hdf5 + $ spack edit hdf5 If you followed the instructions correctly, the code added to the -``lib`` property should be similar to: +``libs`` property should be similar to: .. code-block:: python :emphasize-lines: 1 @@ -516,7 +518,7 @@ If you followed the instructions correctly, the code added to the libraries = query2libraries[key] shared = '+shared' in self.spec return find_libraries( - libraries, root=self.prefix, shared=shared, recurse=True + libraries, root=self.prefix, shared=shared, recursive=True ) where we highlighted the line retrieving the extra parameters. Now we can successfully @@ -524,7 +526,7 @@ complete the installation of ``netcdf ^mpich``: .. code-block:: console - root@advanced-packaging-tutorial:/# spack install netcdf ^mpich + $ spack install netcdf ^mpich ==> libsigsegv is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz ==> m4 is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-r5envx3kqctwwflhd4qax4ahqtt6x43a ... @@ -540,3 +542,4 @@ complete the installation of ``netcdf ^mpich``: ==> Successfully installed netcdf Fetch: 0.01s. Build: 24.61s. Total: 24.62s. [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj + diff --git a/tutorial_basics.rst b/tutorial_basics.rst index 061ad3929c..790f5a7b97 100644 --- a/tutorial_basics.rst +++ b/tutorial_basics.rst @@ -27,8 +27,8 @@ output is all from an Ubuntu 22.04 Docker image. Installing Spack ---------------- -Spack works out of the box. Simply clone Spack to get going. We will -clone Spack and immediately check out the most recent release, v0.23. +Spack works out of the box. Simply clone Spack to get started. We will +clone Spack and immediately check out a recent stable release, v0.23 (note: the specific version is used to ensure consistency of output in this tutorial). .. literalinclude:: outputs/basics/clone.out :language: console @@ -82,10 +82,10 @@ Let's go ahead and install ``gmake``, You will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``. The ``glibc`` and ``gcc-runtime`` packages are automatically tracked by Spack to manage consistency requirements -among compiler runtimes. These do not represent separate installs from -source, but represent aspects of the compiler Spack used for the -install. For the rest of this section, we will ignore these components -and focus on the rest of the install. +among compiler runtimes. They don't represent separate software builds from +source but are records of the system's compiler runtime components that Spack +uses. For the rest of this section, we will ignore these internally tracked +components and focus on the packages explicitly installed. Spack can install software either from source or from a binary cache. Packages in the binary cache are signed with GPG for @@ -104,8 +104,8 @@ tutorial from a binary cache using the same ``spack install`` command. By default this will install the binary cached version if it exists and fall back on installing from source if it does not. -Spack's spec syntax is the interface by which we can request specific -configurations of the package. The ``%`` sigil is used to specify +Spack's "spec" syntax is the interface by which we can request specific +configurations of a package. The ``%`` sigil is used to specify compilers. .. literalinclude:: outputs/basics/zlib-clang.out @@ -113,7 +113,8 @@ compilers. Note that this installation is located separately from the previous one. We will discuss this in more detail later, but this is part of what -allows Spack to support arbitrarily versioned software. +allows Spack to support software with many version and configuration combinations +simultaneously. You can check for particular versions before requesting them. We will use the ``spack versions`` command to see the available versions, and then @@ -135,7 +136,7 @@ The spec syntax also includes compiler flags. Spack accepts ``cppflags``, ``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` parameters. The values of these fields must be quoted on the command line if they include spaces. These values are injected -into the compile line automatically by the Spack compiler wrappers. +into the compilation commands automatically by the Spack compiler wrappers. .. literalinclude:: outputs/basics/zlib-O3.out :language: console @@ -158,7 +159,7 @@ installation directories for every combinatorial version. As we move into more complicated packages with software dependencies, we can see that Spack reuses existing packages to satisfy a dependency. By default, Spack tries hard to reuse existing installations as dependencies, either from a local -store or from configured remote buildcaches. This minimizes unwanted rebuilds +store or from configured remote binary caches. This minimizes unwanted rebuilds of common dependencies, in particular if you update Spack frequently. .. literalinclude:: outputs/basics/tcl.out @@ -193,7 +194,7 @@ even if it also appears as a dependency. Let's move on to slightly more complicated packages. HDF5 is a good example of a more complicated package, with an MPI dependency. If -we install it "out of the box," it will build with OpenMPI. +we install it with default settings, it will typically build with OpenMPI if no other MPI provider is specified. .. literalinclude:: outputs/basics/hdf5.out :language: console @@ -209,17 +210,19 @@ Here we can install HDF5 without MPI support. :language: console We might also want to install HDF5 with a different MPI -implementation. While MPI is not a package itself, packages can depend on -abstract interfaces like MPI. Spack handles these through "virtual -dependencies." A package, such as HDF5, can depend on the MPI -interface. Other packages (``openmpi``, ``mpich``, ``mvapich2``, etc.) -provide the MPI interface. Any of these providers can be requested for -an MPI dependency. For example, we can build HDF5 with MPI support -provided by MPICH by specifying a dependency on ``mpich``. Spack also -supports versioning of virtual dependencies. A package can depend on the -MPI interface at version 3, and provider packages specify what version of -the interface *they* provide. The partial spec ``^mpi@3`` can be satisfied -by any of several providers. +implementation. While ``mpi`` itself is a virtual package representing an +interface, other packages can depend on such abstract interfaces. Spack +handles these through "virtual dependencies." A package, such as HDF5, can +depend on the ``mpi`` virtual package (the interface). Actual MPI implementation +packages (like ``openmpi``, ``mpich``, ``mvapich2``, etc.) "provide" this +virtual package. Any of these providers can be requested to satisfy an MPI +dependency. For example, we can build HDF5 with MPI support provided by +MPICH by specifying a dependency on ``mpich`` (e.g., ``hdf5 ^mpich``). +Spack also supports versioning of virtual dependencies. A package can depend +on the MPI interface at version 3 (e.g., ``hdf5 ^mpi@3``), and provider +packages specify what version of the interface *they* provide. The partial +spec ``^mpi@3`` can be satisfied by any of several MPI implementation packages +that provide MPI version 3. .. literalinclude:: outputs/basics/hdf5-hl-mpi.out :language: console @@ -246,7 +249,7 @@ Now let's look at an even more complicated package. :language: console Now we're starting to see the power of Spack. Trilinos in its default -configuration has 23 top level dependencies, many of which have +configuration has 23 direct dependencies, many of which have dependencies of their own. Installing more complex packages can take days or weeks even for an experienced user. Although we've done a binary installation for the tutorial, a source installation of @@ -312,8 +315,10 @@ We can uninstall packages by spec using the same syntax as install. We can also uninstall packages by referring only to their hash. -We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to -remove packages that are required by another installed package. +We can use either the ``--force`` (or ``-f``) flag or the ``--dependents`` (or ``-R``) +flag to remove packages that are required by another installed package. +Use ``--force`` to remove just the specified package, leaving dependents broken. +Use ``--dependents`` to remove the specified package and all of its dependents. .. literalinclude:: outputs/basics/uninstall-needed.out :language: console @@ -322,8 +327,9 @@ remove packages that are required by another installed package. :language: console Spack will not uninstall packages that are not sufficiently -specified. The ``-a`` (all) flag can be used to uninstall multiple -packages at once. +specified (i.e., if the spec is ambiguous and matches multiple installed packages). +The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching +an ambiguous spec. .. literalinclude:: outputs/basics/uninstall-ambiguous.out :language: console @@ -365,16 +371,16 @@ Customizing Compilers --------------------- Spack manages a list of available compilers on the system, detected -automatically from the user's ``PATH`` variable. The ``spack -compilers`` command is an alias for the command ``spack compiler list``. +automatically from the user's ``PATH`` variable. The ``spack compilers`` +command is an alias for ``spack compiler list``. .. literalinclude:: outputs/basics/compilers.out :language: console -The compilers are maintained in a YAML file. Later in the tutorial you -will learn how to configure compilers by hand for special cases. Spack -also has tools to add compilers, and compilers built with Spack can be -added to the configuration. +The compilers are maintained in a YAML file (``compilers.yaml``). Later in the +tutorial, you will learn how to configure compilers manually for special +cases. Spack also has tools to add compilers, and compilers built with Spack +can be added to the Spack compiler configuration. .. literalinclude:: outputs/basics/install-gcc-12.1.0.out :language: console @@ -384,7 +390,8 @@ added to the configuration. We can add GCC to Spack as an available compiler using the ``spack compiler add`` command. This will allow future packages to build with -``gcc@12.3.0``. To avoid having to copy and paste GCC's path, we can use +this compiler (e.g., ``gcc@12.1.0`` as shown in the example output). +To avoid having to copy and paste GCC's path, we can use ``spack location -i`` to get the installation prefix. .. literalinclude:: outputs/basics/compiler-add-location.out diff --git a/tutorial_binary_cache.rst b/tutorial_binary_cache.rst index f24ec6cba2..68155bf1ba 100644 --- a/tutorial_binary_cache.rst +++ b/tutorial_binary_cache.rst @@ -11,14 +11,15 @@ Binary Caches Tutorial ================================== -In this section of the tutorial you will learn how to share Spack built binaries +In this section of the tutorial, you will learn how to share Spack-built binaries across machines and users using build caches. We will explore a few concepts that apply to all types of build caches, but the -focus is primarily on **OCI container registries** like Docker Hub or Github Packages +focus is primarily on **OCI container registries** (like Docker Hub or GitHub Packages) as a storage backend for binary caches. Spack supports a range of storage backends, -like an ordinary filesystem, S3, and Google Cloud Storage, but OCI build caches -have a few interesting properties that make them worth exploring more in depth. +such as an ordinary filesystem, Amazon S3, and Google Cloud Storage, but OCI +build caches have a few interesting properties that make them worth exploring more +in-depth. Before we configure a build cache, let's install the ``julia`` package, which is an interesting example because it has some non-trivial dependencies like ``llvm``, @@ -50,20 +51,24 @@ Setting up an OCI build cache on GitHub Packages For this tutorial we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use. -First go to ``_ to generate a Personal access -token with ``write:packages`` permissions. Copy this token. +First, go to ``_ to generate a Personal Access +Token (classic) with ``write:packages`` permissions. Copy this token. -Next, we will add this token to the mirror config section of the Spack environment: +Next, we will add this token to the mirrors configuration for the Spack environment. +Replace `` with your GitHub username and +`` with your GitHub username or an organization +where you have permissions to create packages. The build cache name, +`buildcache-${USER}-${HOSTNAME}`, is a suggestion; you can choose your own. .. code-block:: console - $ export MY_OCI_TOKEN= + $ export MY_OCI_TOKEN= $ spack -e . mirror add \ - --oci-username \ + --oci-username \ --oci-password-variable MY_OCI_TOKEN \ --unsigned \ my-mirror \ - oci://ghcr.io//buildcache-${USER}-${HOSTNAME} + oci://ghcr.io//buildcache-${USER}-${HOSTNAME} .. note :: @@ -82,53 +87,59 @@ Your ``spack.yaml`` file should now contain the following: - julia mirrors: my-mirror: - url: oci://ghcr.io//buildcache-- + # The URL should match what you used in `spack mirror add` + url: oci://ghcr.io//buildcache-${USER}-${HOSTNAME} access_pair: - id: + # This should be your GitHub username + id: secret_variable: MY_OCI_TOKEN signed: false -Let's push ``julia`` and its dependencies to the build cache +Let's push ``julia`` and its dependencies to the build cache. The output will +reflect the OCI URL you configured. .. code-block:: console $ spack -e . buildcache push my-mirror -which outputs +which outputs something like: .. code-block:: text - ==> Selected 66 specs to push to oci://ghcr.io//buildcache-- + ==> Selected 66 specs to push to oci://ghcr.io//buildcache-${USER}-${HOSTNAME} ==> Checking for existing specs in the buildcache - ==> 66 specs need to be pushed to ghcr.io//buildcache-- + ==> 66 specs need to be pushed to ghcr.io//buildcache-${USER}-${HOSTNAME} ==> Uploaded sha256:d8d9a5f1fa443e27deea66e0994c7c53e2a4a618372b01a43499008ff6b5badb (0.83s, 0.11 MB/s) ... ==> Uploading manifests ==> Uploaded sha256:cdd443ede8f2ae2a8025f5c46a4da85c4ff003b82e68cbfc4536492fc01de053 (0.64s, 0.02 MB/s) ... - ==> Pushed zstd@1.5.6/ew3aaos to ghcr.io//buildcache--:zstd-1.5.6-ew3aaosbmf3ts2ylqgi4c6enfmf3m5dr.spack + ==> Pushed zstd@1.5.6/ew3aaos to ghcr.io//buildcache-${USER}-${HOSTNAME}:zstd-1.5.6-ew3aaosbmf3ts2ylqgi4c6enfmf3m5dr.spack ... - ==> Pushed julia@1.9.3/dfzhutf to ghcr.io//buildcache--:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack + ==> Pushed julia@1.9.3/dfzhutf to ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack -The location of the pushed package +The location of the pushed package, when referred to as a Docker-style image, will be: .. code-block:: text - ghcr.io//buildcache--:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack + ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack looks very similar to a container image --- we will get to that in a bit. .. note :: - Binaries pushed to GitHub packages are ``private`` by default, which means you need a token - to download them. You can change the visibility to ``public`` by going to GitHub Packages - from your GitHub account, selecting the ``buildcache`` package, go to ``package settings``, - and change the visibility to ``public`` in the ``Danger Zone`` section. This page can also - be directly accessed by going to + Binaries pushed to GitHub Packages are ``private`` by default, which means you (or others) + need a token to download them. You can change the visibility to ``public`` by navigating + to your package on GitHub Packages (usually under your profile or organization page, + then "Packages"). Select your build cache (e.g., `buildcache-${USER}-${HOSTNAME}`), go to + its "Package settings", and change the visibility to ``public`` in the "Danger Zone" section. + The direct URL is typically: .. code-block:: text - https://github.com/users//packages/container/buildcache/settings + https://github.com/users//packages/container//settings + # or for an organization: + # https://github.com/orgs//packages/container//settings ------------------------------- @@ -149,19 +160,21 @@ in the ``spack.yaml`` file: - julia mirrors:: # <- note the double colon my-mirror: - url: oci://ghcr.io//buildcache-- + # Ensure this URL matches the one you configured + url: oci://ghcr.io//buildcache-${USER}-${HOSTNAME} access_pair: - id: - secret_variable: MY_OCI_TOKEN + # This should be your GitHub username + id: + secret_variable: MY_OCI_TOKEN # Or remove access_pair if the cache is public signed: false -An "overwrite install" should be enough to show that the build cache is used: +An "overwrite install" should be enough to show that the build cache is used (output will vary based on your specific configuration): .. code-block:: console $ spack -e . install --overwrite julia - ==> Fetching https://ghcr.io/v2//buildcache--/blobs/sha256:34f4aa98d0a2c370c30fbea169a92dd36978fc124ef76b0a6575d190330fda51 - ==> Fetching https://ghcr.io/v2//buildcache--/blobs/sha256:3c6809073fcea76083838f603509f10bd006c4d20f49f9644c66e3e9e730da7a + ==> Fetching https://ghcr.io/v2//buildcache-${USER}-${HOSTNAME}/blobs/sha256:34f4aa98d0a2c370c30fbea169a92dd36978fc124ef76b0a6575d190330fda51 + ==> Fetching https://ghcr.io/v2//buildcache-${USER}-${HOSTNAME}/blobs/sha256:3c6809073fcea76083838f603509f10bd006c4d20f49f9644c66e3e9e730da7a ==> Extracting julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl from binary cache [+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl @@ -194,9 +207,10 @@ to avoid a separate step. .. note:: As of Spack 0.22, build caches can be used across different Linux distros. The concretizer - will reuse specs that have a host compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``). - For packages compiled with ``gcc`` (and a few others), users do not have to install compilers - first, as the build cache contains the compiler runtime libraries as a separate package. + will reuse specs that have a host-compatible ``libc`` dependency (e.g., from ``glibc`` or ``musl``). + For packages compiled with ``gcc`` (and a few other compilers), users do not have to install these + compilers first on the target machine, as the build cache can provide the necessary compiler + runtime libraries as a separate package dependency. After an index is created, it's possible to list the available packages in the build cache: @@ -216,31 +230,33 @@ concretizer can use it to avoid source builds, and ``spack install`` will fetch However, we can also use this build cache to share binaries directly as runnable container images. We can already attempt to run the image associated with the ``julia`` package that we have -pushed earlier: +pushed earlier (substitute your actual image path): .. code-block:: console - $ docker run ghcr.io//buildcache--:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack julia + $ docker run ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack julia exec /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl/bin/julia: no such file or directory -but immediately we see it fails. The reason is that one crucial part is missing, and that is a -``glibc``, which Spack always treats as an external package. +But immediately we see it fails. The reason is that one crucial part is missing: a +compatible C standard library (like ``glibc``), which Spack usually treats as an +external package provided by the host system. Container images need this to be part +of the image itself. -To fix this, we force push to the registry again, but this time we specify a base image with a -recent version of ``glibc``, for example from ``ubuntu:24.04``: +To fix this, we force-push to the registry again, but this time we specify a base image that +provides a compatible C standard library, for example, ``ubuntu:24.04`` (which includes ``glibc``): .. code-block:: console $ spack -e . buildcache push --force --base-image ubuntu:24.04 my-mirror ... - ==> Pushed julia@1.9.3/dfzhutf to ghcr.io//buildcache:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack + ==> Pushed julia@1.9.3/dfzhutf to ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack -Now let's pull this image again and run it: +Now let's pull this image again and run it (substitute your actual image path): .. code-block:: console - $ docker pull ghcr.io//buildcache--:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack - $ docker run -it --rm ghcr.io//buildcache--:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack + $ docker pull ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack + $ docker run -it --rm ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack root@f53920f8695a:/# julia julia> 1 + 1 2 @@ -248,9 +264,11 @@ Now let's pull this image again and run it: This time it works! The minimal ``ubuntu:24.04`` image provides us not only with ``glibc``, but also other utilities like a shell. -Notice that you can use any base image of choice, like ``fedora`` or ``rockylinux``. The only -constraint is that it has a ``libc`` compatible with the external in the Spack built the binaries. -Spack does not validate this. +Notice that you can use any base image of your choice, like ``fedora`` or ``rockylinux``. The main +constraint is that it has a C standard library (e.g., ``libc.so.6`` from ``glibc``) that is +binary-compatible with the one Spack assumed when building the binaries (often the one from the +build host or a ``target=""` spec). Spack does not currently validate this +compatibility automatically when pushing with a base image. -------------------------------------- Spack environments as container images @@ -279,20 +297,21 @@ we could both edit and run Julia code. $ spack -e . install --add vim -This time we push to the OCI registry, but also pass ``--tag julia-and-vim`` to instruct Spack -to create an image for the environment as a whole, with a human-readable tag: +This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim`` to instruct +Spack to create an additional image tag for the environment as a whole, with a more +human-readable name: .. code-block:: console $ spack -e . buildcache push --base-image ubuntu:24.04 --tag julia-and-vim my-mirror - ==> Tagged ghcr.io//buildcache:julia-and-vim + ==> Tagged ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-and-vim -Now let's run a container from this image: +Now let's run a container from this new image tag (substitute your actual image path): .. code-block:: console - $ docker run -it --rm ghcr.io//buildcache--:julia-and-vim + $ docker run -it --rm ghcr.io//buildcache-${USER}-${HOSTNAME}:julia-and-vim root@f53920f8695a:/# vim ~/example.jl # create a new file with some Julia code root@f53920f8695a:/# julia ~/example.jl # and run it @@ -323,10 +342,10 @@ has a few downsides: * When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning that all dependencies that did install successfully are lost. Troubleshooting the build - typically means starting from scratch in ``docker run`` or on the host system. -* In certain CI environments, it is not possible to use ``docker build``. For example, the - CI script itself may already run in a docker container, and running ``docker build`` *safely* - inside a container is tricky. + typically means starting from scratch either within a ``docker run`` session or on the host system. +* In certain CI environments, it is not possible to use ``docker build`` directly. For example, the + CI script itself may already run in a Docker container, and running ``docker build`` *safely* + inside another container (Docker-in-Docker) is tricky. The takeaway is that Spack decouples the steps that ``docker build`` combines: build isolation, running the build, and creating an image. You can run @@ -340,8 +359,8 @@ Relocation Spack is different from many package managers in that it lets users choose where to install packages. This makes Spack very flexible, as users can install packages in their home directory and do not need root privileges. The downside is that sharing binaries is more complicated, -as binaries may contain hard-coded, absolute paths to machine specific locations, which have -to be adjusted when binaries are installed on a different machine. +as binaries may contain hard-coded, absolute paths to machine-specific locations, which have +to be adjusted when these binaries are installed on a different machine or in a different path. Fortunately Spack handles this automatically upon install from a binary cache. But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: @@ -354,8 +373,8 @@ only modify strings in-place, and if the new path is longer than the old one, we overwrite the next string in the table. To maximize the chances of successful relocation, you should build your binaries in a -relative long path. Fortunately Spack can automatically pad paths to make them longer, -using the following command: +relatively long path. Fortunately, Spack can automatically pad paths to make them longer, +using the following command for your environment: .. code-block:: console @@ -365,11 +384,11 @@ using the following command: Using build caches in CI ------------------------ -Build caches are a great way to speed up CI pipelines. Both GitHub Actions and Gitlab CI +Build caches are a great way to speed up CI pipelines. Both GitHub Actions and GitLab CI support container registries, and this tutorial should give you a good starting point to leverage them. -Spack also provides a basic GitHub Action to already provide you with a binary cache: +Spack also provides a basic GitHub Action that already provides you with a binary cache: .. code-block:: yaml @@ -390,6 +409,6 @@ Summary In this tutorial we have created a build cache on top of an OCI registry, which can be used -* to ``spack install julia vim`` on machines without source builds -* to automatically create container images for individual packages while pushing to the cache -* to create container images for multiple packages at once +* to run ``spack install julia vim`` on machines and have Spack fetch pre-built binaries instead of building from source. +* to automatically create container images for individual packages when pushing to the cache. +* to create container images for entire Spack environments (multiple packages) at once. diff --git a/tutorial_buildsystems.rst b/tutorial_buildsystems.rst index ae87cd871f..7634f69288 100644 --- a/tutorial_buildsystems.rst +++ b/tutorial_buildsystems.rst @@ -11,16 +11,17 @@ Spack Package Build Systems =========================== -You may begin to notice after writing a couple of package template files a -pattern emerge for some packages. For example, you may find yourself writing -an :code:`install()` method that invokes: :code:`configure`, :code:`cmake`, -:code:`make`, :code:`make install`. You may also find yourself writing -:code:`"prefix=" + prefix` as an argument to :code:`configure` or :code:`cmake`. +You may begin to notice after writing a couple of package template files that +a pattern emerges for some packages. For example, you may find yourself writing +an :code:`install()` method that invokes :code:`configure`, :code:`make`, and +:code:`make install` (or their :code:`cmake` equivalents). You may also find +yourself writing :code:`"--prefix=" + self.prefix` as an argument to +:code:`configure` or :code:`cmake`. Rather than having you repeat these lines for all packages, Spack has classes that can take care of these patterns. In addition, -these package files allow for finer grained control of these build systems. -In this section, we will describe each build system and give examples on -how these can be manipulated to install a package. +these base classes allow for finer-grained control of these build systems. +In this section, we will describe each build system base class and give examples +on how these can be used to simplify packaging. ----------------------- Package Class Hierarchy @@ -44,26 +45,30 @@ Package Class Hierarchy PackageBase -> PythonPackage [dir=back] } -The above diagram gives a high level view of the class hierarchy and how each -package relates. Each subclass inherits from the :code:`PackageBaseClass` -super class. The bulk of the work is done in this super class which includes -fetching, extracting to a staging directory and installing. Each subclass +The above diagram gives a high-level view of the class hierarchy and how each +package class relates. Each build system specific class (e.g., :code:`AutotoolsPackage`) +inherits from the :code:`PackageBase` superclass (via the intermediate :code:`Package` +class in most cases, though :code:`Package` itself is also a direct subclass of +:code:`PackageBase` for packages with unique build steps). The bulk of the common +work, such as fetching, extracting to a staging directory, and managing the +installation process, is handled by :code:`PackageBase`. Each subclass then adds additional build-system-specific functionality. In the following -sections, we will go over examples of how to utilize each subclass and to see +sections, we will go over examples of how to utilize each subclass and see how powerful these abstractions are when packaging. ----------------- Package ----------------- -We've already seen examples of a :code:`Package` class in our walkthrough for writing -package files, so we won't be spending much time with them here. Briefly, -the Package class allows for arbitrary control over the build process, whereas -subclasses rely on certain patterns (e.g. :code:`configure` :code:`make` -:code:`make install`) to be useful. :code:`Package` classes are particularly useful -for packages that have a non-conventional way of being built since the packager -can utilize some of Spack's helper functions to customize the building and -installing of a package. +We've already seen examples of using the generic :code:`Package` class in our +walkthrough for writing package files, so we won't be spending much time with +it here. Briefly, the :code:`Package` class allows for arbitrary control over +the build process, whereas its subclasses (like :code:`AutotoolsPackage`, +:code:`CMakePackage`, etc.) rely on common build patterns (e.g., :code:`configure`, +:code:`make`, :code:`make install`) to be useful. The generic :code:`Package` +class is particularly useful for software that has a non-conventional build +process, as it allows the packager to use Spack's helper functions to customize +the building and installing of a package fully. ------------------- Autotools @@ -83,11 +88,11 @@ consist of the following: You'll see that this looks similar to what we wrote in our packaging tutorial. -The :code:`Autotools` subclass aims to simplify writing package files and provides -convenience methods to manipulate each of the different phases for a :code:`Autotools` -build system. +The :code:`AutotoolsPackage` subclass aims to simplify writing package files for +Autotools-based software and provides convenience methods to manipulate each of +the different phases for an :code:`Autotools` build system. -:code:`Autotools` packages consist of four phases: +:code:`AutotoolsPackage` builds consist of four main phases, each corresponding to a method that can be overridden: 1. :code:`autoreconf()` 2. :code:`configure()` @@ -95,8 +100,8 @@ build system. 4. :code:`install()` -Each of these phases have sensible defaults. Let's take a quick look at some -the internals of the :code:`Autotools` class: +Each of these phases has sensible defaults. Let's take a quick look at some of +the internals of the :code:`AutotoolsPackage` class: .. code-block:: console @@ -106,8 +111,9 @@ the internals of the :code:`Autotools` class: This will open the :code:`AutotoolsPackage` file in your text editor. .. note:: - The examples showing code for these classes is abridged to avoid having - long examples. We only show what is relevant to the packager. + The examples showing code for these classes are abridged to avoid having + long examples. We only show what is relevant to the packager for overriding + or understanding behavior. .. literalinclude:: _spack_root/lib/spack/spack/build_systems/autotools.py @@ -125,7 +131,7 @@ then we can append to our :code:`build_targets` property: build_targets = ["foo"] -Which is similar to invoking make in our Package +Which is similar to invoking :code:`make("foo")` in our :code:`Package` class's :code:`install()` method. .. code-block:: python @@ -134,16 +140,16 @@ Which is similar to invoking make in our Package This is useful if we have packages that ignore environment variables and need a command-line argument. -Another thing to take note of is in the :code:`configure()` method. -Here we see that the :code:`prefix` argument is already included since it is a -common pattern amongst packages using :code:`Autotools`. We then only have to -override :code:`configure_args()`, which will then return it's output to -to :code:`configure()`. Then, :code:`configure()` will append the common -arguments +Another thing to take note of is the :code:`configure()` method in :code:`AutotoolsPackage`. +It automatically includes the :code:`--prefix=self.prefix` argument, a common +pattern for packages using :code:`Autotools`. Therefore, you typically only need to +override the :code:`configure_args()` method to return a list of additional +arguments. The :code:`configure()` method appends these to the standard arguments. -Packagers also have the option to run :code:`autoreconf` in case a package -needs to update the build system and generate a new :code:`configure`. Though, -for the most part this will be unnecessary. +Packagers also have the option to run :code:`autoreconf -fi` by overriding the +:code:`autoreconf()` method if a package needs to update its build system files +(e.g., :code:`configure` script, :code:`Makefile.in` templates). However, for +most packages, this will be unnecessary. Let's look at the :code:`mpileaks` package.py file that we worked on earlier: @@ -151,9 +157,9 @@ Let's look at the :code:`mpileaks` package.py file that we worked on earlier: $ spack edit mpileaks -Notice that mpileaks is a :code:`Package` class but uses the :code:`Autotools` -build system. Although this package is acceptable let's make this into an -:code:`AutotoolsPackage` class and simplify it further. +Notice that :code:`mpileaks` was originally written as a generic :code:`Package` +but uses the :code:`Autotools` build system. Although this is acceptable, +let's convert it to an :code:`AutotoolsPackage` to simplify it further. .. literalinclude:: tutorial/examples/Autotools/0.package.py :language: python @@ -172,10 +178,10 @@ to be overridden is :code:`configure_args()`. :emphasize-lines: 25,26,27,28,29,30,31,32 :linenos: -Since Spack takes care of setting the prefix for us we can exclude that as -an argument to :code:`configure`. Our packages look simpler, and the packager -does not need to worry about whether they have properly included :code:`configure` -and :code:`make`. +Since Spack's :code:`AutotoolsPackage` takes care of setting the prefix for us, +we can exclude that from :code:`configure_args()`. Our package file looks simpler, +and the packager does not need to worry about the standard invocation of +:code:`configure` and :code:`make`. This version of the :code:`mpileaks` package installs the same as the previous, but the :code:`AutotoolsPackage` class lets us do it with a cleaner looking @@ -190,7 +196,7 @@ to edit a :code:`Makefile` to set up platform and compiler specific variables. These packages are handled by the :code:`Makefile` subclass which provides convenience methods to help write these types of packages. -A :code:`MakefilePackage` class has three phases that can be overridden. These include: +A :code:`MakefilePackage` build has three main phases that can be overridden by the packager: 1. :code:`edit()` 2. :code:`build()` @@ -272,8 +278,8 @@ compiler: $ spack stage bowtie .. note:: - As usual make sure you have shell support activated with spack: - :code:`source /path/to/spack_root/spack/share/spack/setup-env.sh` + As usual make sure you have shell support activated with Spack: + :code:`source /path/to/spack/share/spack/setup-env.sh` .. code-block:: console @@ -290,19 +296,19 @@ compiler: LIBS = $(LDFLAGS) -lz HEADERS = $(wildcard *.h) -To fix this, we need to use the :code:`edit()` method to write our custom -:code:`Makefile`. +To fix this, we need to use the :code:`edit()` method to modify the :code:`Makefile`. .. literalinclude:: tutorial/examples/Makefile/2.package.py :language: python :emphasize-lines: 23,24,25 :linenos: -Here we use a :code:`FileFilter` object to edit our :code:`Makefile`. It takes -in a regular expression and then replaces :code:`CC` and :code:`CXX` to whatever -Spack sets :code:`CC` and :code:`CXX` environment variables to. This allows us to -build :code:`Bowtie` with whatever compiler we specify through Spack's -:code:`spec` syntax. +Here we use a :code:`FileFilter` object (a Spack utility) to edit our :code:`Makefile`. +It takes a regular expression to find lines (e.g., assignments to `CC` and `CXX`) +and replaces them with values derived from Spack's build environment (e.g., +`self.compiler.cc` and `self.compiler.cxx`, which point to Spack's compiler +wrappers). This allows us to build :code:`Bowtie` with whatever compiler we +specify through Spack's spec syntax. Let's change the build and install phases of our package: @@ -311,10 +317,11 @@ Let's change the build and install phases of our package: :emphasize-lines: 28,29,30,31,32,35,36 :linenos: -Here demonstrate another strategy that we can use to manipulate our package -We can provide command-line arguments to :code:`make()`. Since :code:`Bowtie` -can use :code:`tbb` we can either add :code:`NO_TBB=1` as a argument to prevent -:code:`tbb` support or we can just invoke :code:`make` with no arguments. +Here we demonstrate another strategy that we can use to manipulate our package's +build. We can provide command-line arguments to :code:`make()`. Since :code:`Bowtie` +can use :code:`tbb`, we can either add :code:`NO_TBB=1` as an argument to prevent +:code:`tbb` support, or we can invoke :code:`make` with no arguments if TBB is desired +and found by its build system. :code:`Bowtie` requires our :code:`install_target` to provide a path to the install directory. We can do this by providing :code:`prefix=` as a command @@ -361,9 +368,10 @@ the :code:`Makefile`. Some packages include a configuration file that sets certain compiler variables, platform specific variables, and the location of dependencies or libraries. -If the file is simple and only requires a couple of changes, we can overwrite +If the file is simple and only requires a couple of changes, we can replace those entries with our :code:`FileFilter` object. If the configuration involves -complex changes, we can write a new configuration file from scratch. +complex changes, we can write a new configuration file from scratch within the +:code:`edit()` method. Let's look at an example of this in the :code:`elk` package: @@ -453,9 +461,9 @@ Let's look at an example of this in the :code:`elk` package: for key in config: inc.write('{0} = {1}\n'.format(key, config[key])) -:code:`config` is just a dictionary that we can add key-value pairs to. By the -end of the :code:`edit()` method we write the contents of our dictionary to -:code:`make.inc`. +:code:`config` is just a Python dictionary that we populate with key-value pairs. +By the end of the :code:`edit()` method, we write the contents of this dictionary +to the :code:`make.inc` file, which the package's :code:`Makefile` then includes. --------------- CMake @@ -480,14 +488,14 @@ As you can see from the example above, it's very similar to invoking the variable names and options differ. Most options in CMake are prefixed with a :code:`'-D'` flag to indicate a configuration setting. -In the :code:`CMakePackage` class we can override the following phases: +In the :code:`CMakePackage` class, we can override the following build phases: 1. :code:`cmake()` 2. :code:`build()` 3. :code:`install()` -The :code:`CMakePackage` class also provides sensible defaults so we only need to -override :code:`cmake_args()`. +The :code:`CMakePackage` class also provides sensible defaults, so often we only +need to override :code:`cmake_args()` to pass package-specific options. Let's look at these defaults in the :code:`CMakePackage` class in the :code:`_std_args()` method: @@ -507,7 +515,7 @@ Unix-Makefile_ generators as well as Ninja_ generators. .. _Unix-Makefile: https://cmake.org/cmake/help/v3.4/generator/Unix%20Makefiles.html .. _Ninja: https://cmake.org/cmake/help/v3.4/generator/Ninja.html -If no generator is specified Spack will default to :code:`Unix Makefiles`. +If no generator is specified, Spack will default to :code:`Unix Makefiles`. Next we setup the build type. In :code:`CMake` you can specify the build type that you want. Options include: @@ -520,12 +528,13 @@ that you want. Options include: With these options you can specify whether you want your executable to have the debug version only, release version or the release with debug information. -Release executables tend to be more optimized than Debug. In Spack, we set -the default as Release unless otherwise specified through a variant. +Release executables tend to be more optimized than Debug versions. In Spack, the +default build type is :code:`RelWithDebInfo` (Release with Debug Information) +unless otherwise specified through a variant (e.g., `+debug` or `build_type=Debug`). Spack then automatically sets up the :code:`-DCMAKE_INSTALL_PREFIX` path, -appends the build type (:code:`RelWithDebInfo` default), and then specifies a verbose -:code:`Makefile`. +appends the build type (defaulting to :code:`RelWithDebInfo`), and enables a verbose +:code:`Makefile` output by default. Next we add the :code:`rpaths` to :code:`-DCMAKE_INSTALL_RPATH:STRING`. @@ -580,11 +589,11 @@ Again we fill in the details: :linenos: :emphasize-lines: 9,13,14,18,19,20,21,22,23 -As mentioned earlier, Spack will use sensible defaults to prevent repeated code -and to make writing :code:`CMake` package files simpler. +As mentioned earlier, Spack's :code:`CMakePackage` uses sensible defaults to +reduce boilerplate and simplify writing package files for :code:`CMake`-based software. -In callpath, we want to add options to :code:`CALLPATH_WALKER` as well as add -compiler flags. We add the following options like so: +In :code:`callpath`, we might want to control options like :code:`CALLPATH_WALKER` +or add specific compiler flags. We can return these options from :code:`cmake_args()`: .. literalinclude:: tutorial/examples/Cmake/2.package.py :language: python @@ -639,18 +648,18 @@ so we override the :code:`install()` method to do it for us: PythonPackage -------------- -Python extensions and modules are built differently from source than most applications. -These modules are usually installed using the following line: +Python extensions and modules are built differently than most compiled applications. +These modules are often installed using commands like: .. code-block:: console $ pip install . -We can write package files for Python packages using the :code:`Package` class, -but the class brings with it a lot of methods that are useless for Python packages. -Instead, Spack has a :code:`PythonPackage` subclass that allows packagers -of Python modules to be able to invoke :code:`pip`. +We could write package files for Python packages using the generic :code:`Package` +class, but it has many general-purpose methods not specific to Python builds. +Instead, Spack provides a :code:`PythonPackage` subclass that simplifies packaging +Python modules, typically by invoking :code:`python setup.py install` or :code:`pip install .`. We will write a package file for Pandas_: @@ -681,15 +690,17 @@ And we are left with the following template: :language: python :linenos: -As you can see this is not any different than any package template that we have -written. We have the choice of providing build options or using the sensible -defaults +As you can see, this is not any different from other package templates we have +seen. We have the choice of providing build options or using the sensible +defaults provided by :code:`PythonPackage`. -Luckily for us, there is no need to provide build args. +Luckily for :code:`py-pandas`, there is no need to provide specific build arguments; +the defaults work well. -Next we need to find the dependencies of a package. Dependencies are usually -listed in :code:`setup.py`. You can find the dependencies by searching for -:code:`install_requires` keyword in that file. Here it is for :code:`Pandas`: +Next, we need to find the dependencies of the Python package. These are usually +listed in its :code:`setup.py` file (or increasingly, :code:`pyproject.toml`). +You can often find the runtime dependencies by looking for an :code:`install_requires` +keyword in :code:`setup.py`. Here's a snippet for an older version of :code:`Pandas`: .. code-block:: python @@ -714,9 +725,10 @@ You can find a more comprehensive list at the Pandas documentation_. .. _documentation: https://pandas.pydata.org/pandas-docs/stable/install.html -By reading the documentation and :code:`setup.py` we found that :code:`Pandas` -depends on :code:`python-dateutil`, :code:`pytz`, and :code:`numpy`, :code:`numexpr`, -and finally :code:`bottleneck`. +By reading the documentation and :code:`setup.py` (or :code:`pyproject.toml`), we +would find that :code:`py-pandas` depends on Spack packages like :code:`py-python-dateutil`, +:code:`py-pytz`, :code:`py-numpy`, :code:`py-numexpr`, and :code:`py-bottleneck`. +(Note: Spack prepends `py-` to most Python package names for clarity). Here is the completed :code:`Pandas` script: @@ -724,11 +736,14 @@ Here is the completed :code:`Pandas` script: :language: python :linenos: -It is quite important to declare all the dependencies of a Python package. -Spack can "activate" Python packages to prevent the user from having to -load each dependency module explicitly. If a dependency is missed, Spack will -be unable to properly activate the package and it will cause an issue. To -learn more about extensions go to `spack extensions `_. +It is quite important to declare all the dependencies of a Python package in its +Spack :code:`package.py` file. Spack can "activate" Python packages, which makes +their modules available in the user's Python environment without needing to load +each dependency module explicitly. If a Spack dependency is missed, Spack will +be unable to properly activate the package with all its underlying requirements, +which can cause runtime issues. To learn more about extensions and Python +packaging in Spack, see the relevant sections in the Spack documentation, +including information on the `spack extensions `_ command. From this example, you can see that building Python modules is made easy through the :code:`PythonPackage` class. @@ -737,9 +752,9 @@ through the :code:`PythonPackage` class. Other Build Systems ------------------- -Although we won't get in depth with any of the other build systems that Spack -supports, it is worth mentioning that Spack does provide subclasses -for the following build systems: +Although we won't go in-depth with all of the other build systems that Spack +supports, it is worth mentioning that Spack provides other specialized base classes, +including: 1. :code:`IntelPackage` 2. :code:`SconsPackage` @@ -749,10 +764,12 @@ for the following build systems: 6. :code:`QMakePackage` -Each of these classes have their own abstractions to help assist in writing -package files. For whatever doesn't fit nicely into the other build-systems, -you can use the :code:`Package` class. +Each of these classes has its own abstractions to help assist in writing +package files for software using those build systems. For packages that don't fit +nicely into any of these specialized classes, you can always use the generic +:code:`Package` class for maximum flexibility. -Hopefully by now you can see how we aim to make packaging simple and -robust through these classes. If you want to learn more about these build -systems, check out `Implementing the installation procedure `_ in the Packaging Guide. +Hopefully, by now you can see how Spack aims to make packaging simple and +robust through these build system base classes. If you want to learn more about +these build systems, check out the `Implementing the installation procedure `_ +section in the Spack Packaging Guide. diff --git a/tutorial_configuration.rst b/tutorial_configuration.rst index 26bea9608f..747dcaff88 100644 --- a/tutorial_configuration.rst +++ b/tutorial_configuration.rst @@ -26,18 +26,18 @@ A partial list of some key configuration sections is provided below. * - config - General settings (install location, number of build jobs, etc) * - concretizer - - Specializaiton of the concretizer behavior (reuse, unification, etc) + - Specialization of the concretizer behavior (reuse, unification, etc.) * - compilers - Define the compilers that Spack can use (required and system specific) * - mirrors - - Locations where spack can look for stashed source or binary distributions + - Locations where Spack can look for stashed source or binary distributions * - packages - Specific settings and rules for packages * - modules - Naming, location and additional configuration of Spack generated modules The full list of sections can be viewed with ``spack config list``. -For further education we encourage you to explore the spack +For further education, we encourage you to explore the Spack `documentation on configuration files `_. The principle goals of this section of the tutorial are: @@ -46,14 +46,14 @@ The principle goals of this section of the tutorial are: 2. Demonstrate how to manipulate configurations 3. Show how to configure system assets with spack (compilers and packages) -As such we will primarily focus on the ``compilers`` -and ``packages`` configuration sections in this portion of the tutorial. +As such, we will primarily focus on the ``compilers`` +and ``packages`` configuration sections in this portion of the tutorial. We will explain this by first covering how to manipulate configurations from the command line and then show how this impacts the configuration file hierarchy. We will then move into compiler and package configurations to help -you develop skills for getting the builds you want on your system. Finally, -we will give some brief attention to more generalized spack configurations +you develop skills for getting the builds you want on your system. Finally, +we will give some brief attention to more generalized Spack configurations in the ``config`` section. For all of these features, we will demonstrate how we build up a full @@ -67,9 +67,9 @@ output is all from a server running Ubuntu version 22.04. Configuration from the command line ----------------------------------- -You can run ``spack config blame [section]`` at any point in time to see what -your current configuration is. If you omit the section then spack will dump all -the configurations settings to your screen. Let's go ahead and run this for the +You can run ``spack config blame
`` at any point in time to see what +your current configuration is for that `
`. If you omit the section, then Spack +will show all configuration settings. Let's go ahead and run this for the ``concretizer`` section. .. code-block:: console @@ -89,19 +89,20 @@ through the command line. $ spack config add concretizer:reuse:false If we rerun ``spack config blame concretizer`` we can see that the change was -applied. +applied. .. code-block:: console $ spack config blame concretizer -Notice that the reference file on for this option is now different. -This indicates the scope where the configuration was set in, and we will -discuss how spack chooses the default scope shortly. +Notice that the reference file for this option is now different. +This indicates the scope where the configuration was set, and we will +discuss how Spack chooses the default scope shortly. For now, it is important to note that the ``spack config`` command accepts an -optional ``--scope`` flag so we can be more precise in the configuration process. -This will make more sense after the next section which provides -the definition of spack's configuration scopes and their hierarchy. +optional ``--scope`` flag (e.g., ``--scope user``, ``--scope site``) so we can +be more precise in the configuration process. This will make more sense after +the next section, which provides the definition of Spack's configuration scopes +and their hierarchy. .. _configs-tutorial-scopes: @@ -123,7 +124,7 @@ Environment In environment base directory (in ``spack.yaml``) Custom Custom directory, specified with ``--config-scope`` User ``~/.spack/`` Site ``$SPACK_ROOT/etc/spack/`` -System ``/etc/spack/`` +System ``/etc/spack`` (or ``/etc/spack.d`` for multiple files) Defaults ``$SPACK_ROOT/etc/spack/defaults/`` ============ =================================================== @@ -156,24 +157,23 @@ other configuration scopes. Platform-specific scopes ^^^^^^^^^^^^^^^^^^^^^^^^ -Some facilities manage multiple platforms from a single shared -file system. In order to handle this, each of the configuration -scopes listed above has two *sub-scopes*: platform-specific and -platform-independent. For example, compiler settings can be stored -in the following locations: - -#. ``$ENVIRONMENT_ROOT/spack.yaml`` -#. ``~/.spack//compilers.yaml`` -#. ``~/.spack/compilers.yaml`` -#. ``$SPACK_ROOT/etc/spack//compilers.yaml`` -#. ``$SPACK_ROOT/etc/spack/compilers.yaml`` -#. ``/etc/spack//compilers.yaml`` -#. ``/etc/spack/compilers.yaml`` -#. ``$SPACK_ROOT/etc/defaults//compilers.yaml`` -#. ``$SPACK_ROOT/etc/defaults/compilers.yaml`` - -These files are listed in decreasing order of precedence, so files in -``~/.spack/`` will override settings in ``~/.spack``. +Some facilities manage multiple platforms (e.g., different OS versions or CPU architectures) +from a single shared file system. Spack allows platform-specific configurations +within most scopes. For a given configuration section (like `compilers`), +Spack will look for a platform-specific file (e.g., `compilers.yaml` inside a +directory named after the platform, like `~/.spack/cray-cnl7-haswell/compilers.yaml`) +before looking for a platform-independent file in the same scope (e.g., `~/.spack/compilers.yaml`). +This means that within a particular scope (like `user`), the platform-specific configuration +takes precedence. + +For example, for the `user` scope (`~/.spack`), Spack would consider: +1. `~/.spack//compilers.yaml` (platform-specific, highest precedence within user scope) +2. `~/.spack/compilers.yaml` (platform-independent) + +This platform-specific lookup occurs within each of the configurable scopes (User, Site, System, Defaults). +Environment configurations (`spack.yaml`) are typically platform-agnostic as defined, but Spack +can be made aware of platform-specific environments if needed (e.g., by using different environment files +or by including platform-specific logic within package files). ----------- YAML Format @@ -294,8 +294,8 @@ active environment): paths: cc: /usr/bin/gcc-10 cxx: /usr/bin/g++-10 - f77: usr/bin/gfortran-10 - fc: usr/bin/gfortran-10 + f77: /usr/bin/gfortran-10 + fc: /usr/bin/gfortran-10 flags: {} operating_system: ubuntu22.04 target: x86_64 @@ -356,23 +356,23 @@ sections we leave unchanged. These sections specify when Spack can use different compilers, and are primarily useful for configuration files that will be used across multiple systems. -We can verify that our new compiler works by invoking it now: +We can verify that our new compiler works by invoking it now (Spack will build `zlib` if a binary isn't found for this exact compiler spec): .. code-block:: console - $ spack install --no-cache zlib %clang@14.0.0-gfortran + $ spack install zlib %clang@14.0.0-gfortran ... This new compiler also works on Fortran codes. We'll show it by -compiling a small package using as a build dependency ``cmake%gcc@11.4.0`` -since it is already available in our binary cache: +compiling a small package, using a pre-existing `cmake` built with `gcc@11.4.0` +(if available in a binary cache or already installed) to speed up the process: .. code-block:: console - $ spack install --reuse cmake %gcc@11.4.0 + $ spack install cmake %gcc@11.4.0 # Ensure cmake is available ... - $ spack install --no-cache --reuse json-fortran %clang@=14.0.0-gfortran ^cmake%gcc@11.4.0 + $ spack install json-fortran %clang@=14.0.0-gfortran ^cmake%gcc@11.4.0 ... @@ -710,7 +710,8 @@ with MPI again: mpi: [mpich, openmpi] curl: externals: - - spec: curl@7.81.0 %gcc@11.4.0 + # This should match the version identified on the system, e.g., 7.81.0 + - spec: curl@7.81.0 %gcc@11.4.0 prefix: /usr buildable: false mpich: @@ -838,9 +839,10 @@ from this file system with the following ``config.yaml``: `Basic Settings `_ for details. -On systems with compilers that absolutely *require* environment variables -like ``LD_LIBRARY_PATH``, it is possible to prevent Spack from cleaning -the build environment with the ``dirty`` setting: +On systems with compilers that absolutely *require* certain environment variables +(like ``LD_LIBRARY_PATH``, though this is generally discouraged for builds), +it is possible to prevent Spack from cleaning most of the build environment +with the ``dirty`` setting: .. code-block:: yaml @@ -859,7 +861,7 @@ node with 16 cores, this will look like: .. code-block:: console - $ spack install --no-cache --verbose --overwrite --yes-to-all zlib + $ spack install --verbose --overwrite --yes-to-all zlib ==> Installing zlib ==> Executing phase: 'install' ==> './configure' '--prefix=/home/user/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/zlib-1.2.12-fntvsj6xevbz5gyq7kfa4xg7oxnaolxs' @@ -888,11 +890,12 @@ number of cores our build uses, set ``build_jobs`` like so: build_jobs: 2 -If we uninstall and reinstall zlib-ng, we see that it now uses only 2 cores: +If we uninstall and reinstall `zlib`, we see that it now uses only 2 cores: .. code-block:: console - $ spack install --no-cache --verbose --overwrite --yes-to-all zlib-ng + $ spack uninstall --yes-to-all zlib # First uninstall the previous version + $ spack install --verbose --overwrite --yes-to-all zlib ==> Installing zlib ==> Executing phase: 'install' ==> './configure' '--prefix=/home/user/spack/opt/spack/linux-ubuntu22.04... @@ -907,10 +910,12 @@ If we uninstall and reinstall zlib-ng, we see that it now uses only 2 cores: Obviously, if you want to build everything in serial for whatever reason, you would set ``build_jobs`` to 1. -Last we'll unset ``concretizer:reuse:false`` since we'll want to -enable concretizer reuse for the rest of this tutorial. +Lastly, we'll remove the `concretizer:reuse:false` setting from our user scope +configuration, as we'll want to enable concretizer reuse for the rest of this +tutorial. This command removes the setting from the default configuration scope +(which is `user` if not specified and if a user-level config file exists for it). -.. code-block:: yaml +.. code-block:: console $ spack config rm concretizer:reuse diff --git a/tutorial_developer_workflows.rst b/tutorial_developer_workflows.rst index ff8c6d9e23..2f25e65451 100644 --- a/tutorial_developer_workflows.rst +++ b/tutorial_developer_workflows.rst @@ -13,7 +13,7 @@ Developer Workflows Tutorial This tutorial will guide you through the process of using the ``spack develop`` command to develop software from local source code within a -spack environment. With this command spack will manage your +Spack environment. With this command, Spack will manage your dependencies while you focus on testing changes to your library and/or application. @@ -27,18 +27,18 @@ mirror or the internet before building and installing your package. As developers, we want to build from local source, which we will constantly change, build, and test. -Let's imagine for a second we're working on ``scr``. ``scr`` is a +Let's imagine for this tutorial we're working on ``scr``. ``scr`` is a library used to implement scalable checkpointing in application codes. It supports writing/reading checkpoints quickly and efficiently using MPI and high-bandwidth file I/O. We'd like to test changes to -scr within an actual application so we'll test with ``macsio``, a +``scr`` within an actual application, so we'll test with ``macsio``, a proxy application written to mimic typical HPC I/O workloads. We've chosen ``scr`` and ``macsio`` because together they are quick to build. -We'll start by making an environment for our development. We need to +We'll start by making an environment for our development. We need to build ``macsio`` with ``scr`` support, and we'd like everything to be -built without fortran support for the time being. Let's set up that +built without Fortran support for the time being. Let's set up that development workflow. .. literalinclude:: outputs/dev/setup-scr.out @@ -157,45 +157,56 @@ be the 3.1.0 release that we want to write a patch for: .. literalinclude:: outputs/dev/develop-1.out :language: console -The spack develop command marks the package as being a "development" -package in the spack.yaml. This adds a special ``dev_path=`` attribute -to the spec for the package, so spack remembers where the source code -for this package is located. The develop command also downloads/checks -out the source code for the package. By default, the source code is -downloaded into a subdirectory of the environment. You can change the -location of this source directory by modifying the ``path:`` attribute -of the develop configuration in the environment. - -There are a few gotchas with the spack develop command - -* You often specify the package version manually when specifying a - package as a dev package. Spack needs to know the version of the dev - package so it can supply the correct flags for the package's build - system. If a version is not supplied then spack will take the maximum version - defined in the package where where `infinity versions `_ like ``develop`` and ``main`` - have a higher value than the numeric versions. -* You should ensure a spec for the package you are developing appears in the DAG of at least one of the roots of the environment with the same version that you are developing. - ``spack add `` with the matching version you want to develop is a way to ensure - the develop spec is satisfied.the ``spack.yaml`` environments file. This is because - develop specs are not concretization constraints but rather a criteria for adding - the ``dev_path=`` variant to existing spec. -* You'll need to re-concretize the environment so that the version - number and the ``dev_path=`` attributes are properly added to the - cached spec in ``spack.lock``. +The ``spack develop`` command marks the package as being a "development" +package in the ``spack.yaml`` file of the current environment. This adds a +special ``dev_path=`` attribute to the spec for the package in +the environment's ``spack.lock`` file once concretized, so Spack remembers +where the local source code for this package is located. The ``spack develop`` +command also downloads or checks out the source code for the package if it's +not already present at the specified path (or default path). By default, if +no path is specified, the source code is placed into a subdirectory within the +environment (e.g., ``./spack-src/scr``). You can specify a custom location for +this source directory using the ``--path`` option with ``spack develop``, or by +modifying the ``path:`` attribute of the develop configuration in the ``spack.yaml`` +file. + +There are a few important considerations (or "gotchas") when using the ``spack develop`` command: + +* You often need to specify the package version manually when marking a + package for development (e.g., ``spack develop scr@3.1.0``). Spack needs to + know the version of the development package to correctly apply dependencies + and flags from its ``package.py`` file. If a version is not supplied, Spack + will try to infer it, often defaulting to the highest version defined in the + package file, where `infinity versions `_ + (like ``@develop`` or ``@main`` if defined in the package) are considered higher + than numeric versions. +* You must ensure that a spec for the package you are developing (with the + correct version) exists in the environment's list of root specs (e.g., added via + ``spack add @``) *before* running ``spack develop``. + The ``spack develop`` command doesn't add the package to the environment's + roots; it modifies an *existing* spec in the environment by associating it + with a local source path. If the package/version isn't already a root spec, + the ``dev_path`` attribute won't be associated correctly. +* After running ``spack develop`` or changing which packages are in development + mode, you **must** re-concretize the environment (e.g., ``spack concretize -f`` + or ``spack install`` which triggers concretization). This ensures that the + ``dev_path=`` attribute is correctly recorded in the ``spack.lock`` file and + used for subsequent builds. .. literalinclude:: outputs/dev/develop-conc.out :language: console -Now that we have this done, we tell spack to rebuild both ``scr`` and +Now that we have this done, we tell Spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``. .. literalinclude:: outputs/dev/develop-2.out :language: console This rebuilds ``scr`` from the subdirectory we specified. If your -package uses cmake, spack will build the package in a build directory -that matches the hash for your package. From here you can change into -the appropriate directory and perform your own build/test cycles. +package uses CMake, Spack will perform an out-of-source build, creating a +build directory typically inside ``/.spack-build-/`` or similar, +depending on the Spack version and package recipe. You can change into +the appropriate build directory and perform your own build/test cycles manually if needed. Now, we can develop our code. For the sake of this demo, we're just going to intentionally introduce an error. Let's edit a file and @@ -204,46 +215,48 @@ remove the first semi-colon we find. .. literalinclude:: outputs/dev/edit-1.out :language: console -Once you have a development package, ``spack install`` also works much -like "make". Since spack knows the source code directory of the -package, it checks the filetimes on the source directory to see if -we've made recent changes. If the file times are newer, it will -rebuild ``scr`` and any other package that depends on ``scr``. +Once you have a development package, subsequent ``spack install`` calls work somewhat +like ``make``. Since Spack knows the source code directory of the +development package, it checks the timestamps of the files in that directory. +If the source files are newer than the last build time, Spack will +rebuild that package and any other packages in the environment that depend on it. .. literalinclude:: outputs/dev/develop-3.out :language: console -Here, the build failed as expected. We can look at the output for the -build in ``scr/spack-build-out.txt`` to find out why, or we can -launch a shell directly with the appropriate environment variables to -figure out what went wrong by using ``spack build-env scr@2.0 -- -bash``. If that's too much to remember, then sourcing -``scr/spack-build-env.txt`` will also set all the appropriate -environment variables so we can diagnose the build ourselves. Now -let's fix it and rebuild directly. +Here, the build failed as expected. We can look at the build log using +``spack build-log scr`` (Spack will show the path to the log file). +Alternatively, to debug interactively, we can launch a shell directly within +the correct build environment using ``spack build-env scr -- bash``. +If you prefer to source the environment script manually, you can dump it first +with ``spack build-env --dump scr > my_scr_build_env.sh`` and then ``source ./my_scr_build_env.sh``. +Now let's fix the error in the source code and rebuild using ``spack install``. .. literalinclude:: outputs/dev/develop-4.out :language: console -You'll notice here that spack rebuilt both ``scr`` and ``macsio``, as +You'll notice here that Spack rebuilt both ``scr`` and ``macsio``, as expected. -Taking advantage of iterative builds with spack requires cooperation -from your build system. When spack performs a rebuild on a -development package, it reruns all the build stages for your package -without cleaning the source and build directories to a pristine -state. If your build system can take advantage of the previously -compiled object files then you'll end up with an iterative build. - -- If your package just uses make, you also should get iterative builds - for free when running ``spack develop``. -- If your package uses cmake with the typical ``cmake`` / ``build`` / - ``install`` build stages, you'll get iterative builds for free with - spack because cmake doesn’t modify the filetime on the - ``CMakeCache.txt`` file if your cmake flags haven't changed. -- If your package uses autoconf, then rerunning the typical - ``autoreconf`` stage typically modifies the filetime of - ``config.h``, which can trigger a cascade of rebuilding. +Taking advantage of iterative builds with Spack requires cooperation +from your package's build system. When Spack performs a rebuild on a +development package, it typically reruns all the build phases defined in the +Spack package file (e.g., `cmake`, `build`, `install` for `CMakePackage`) +without cleaning the source and build directories to a pristine state. If your +build system can detect unchanged files and avoid recompiling them (e.g., +via correct `Makefile` dependencies), then you'll achieve an incremental/iterative +build. + +- If your package just uses `make` and has a well-structured `Makefile`, + you should get iterative builds for free when using ``spack develop``. +- If your package uses CMake and follows typical practices, you'll generally + get iterative builds because CMake and the generated build files (e.g., Makefiles + or Ninja files) are designed for this. Spack doesn't interfere with CMake's + own change detection unless CMake flags themselves change. +- If your package uses Autotools, rerunning the `autoreconf` or `configure` + stages (if Spack deems it necessary due to changes or if the package recipe + forces it) might modify files like `config.h` or Makefiles, which can + sometimes trigger more extensive rebuilds than desired. Multiple packages can also be marked as develop. If we were co-developing ``macsio``, we could run @@ -252,10 +265,11 @@ co-developing ``macsio``, we could run :language: console Using development workflows also lets us ship our whole development -process to another developer on the team. They can simply take our -spack.yaml, create a new environment, and use this to replicate our -build process. For example, we'll make another development environment -here. +process to another developer on the team. They can simply take our +``spack.yaml`` file, create a new environment from it, and use ``spack develop`` +(potentially with ``--path`` pointing to their local checkouts if they differ) +to replicate the build process. For example, we'll make another development +environment here. .. literalinclude:: outputs/dev/otherdevel.out :language: console @@ -263,8 +277,11 @@ here. Here, ``spack develop`` with no arguments will check out or download the source code and place it in the appropriate places. -When we're done developing, we simply tell spack that it no longer -needs to keep a development version of the package. +When we're done developing a particular package locally, we can tell Spack +that it no longer needs to use the local development version by using +``spack develop --uninstall ``. This removes the ``dev_path`` +attribute. Remember to re-concretize and reinstall if you want Spack to use a +standard (non-dev) version of the package. .. literalinclude:: outputs/dev/wrapup.out :language: console @@ -273,13 +290,13 @@ needs to keep a development version of the package. Workflow Summary ------------------- -Use the ``spack develop`` command with an environment to make a -reproducible build environment for your development workflow. Spack -will set up all the dependencies for you and link all your packages -together. Within a development environment, ``spack install`` works -similar to ``make`` in that it will check file times to rebuild the -minimum number of spack packages necessary to reflect the changes to -your build. +Use the ``spack develop`` command within a Spack environment to create a +reproducible build setup for your development workflow. Spack +will manage all the dependencies and link your packages +together. Within such an environment, subsequent ``spack install`` commands +work similarly to ``make`` in that they will check file timestamps to rebuild +the minimum number of Spack packages necessary to reflect changes you've made +to your local source code. ------------------- Optional: Tips and Tricks @@ -298,88 +315,127 @@ Source Code Management ---------- ``spack develop`` allows users to manipulate the source code locations -The default behavior is to let spack manage its location and cloning operations, -but software developers often want more control over these. +The default behavior is to let Spack manage its location (typically within the +environment directory, e.g., ``./spack-src/``) and cloning operations. +However, software developers often want more control. -The source directory can be set with the ``--path`` argument when calling ``spack develop``. -If this directory already exists then ``spack develop`` will not attempt to fetch the code -for you. This allows developers to pre-clone the software or use preferred paths as they wish. +The source directory can be explicitly set with the ``--path `` +argument when calling ``spack develop``. If this directory already exists and +contains source code, Spack will use it and will not attempt to fetch or clone +the code. This allows developers to pre-clone their software (e.g., to a specific +branch or fork) or use preferred project directory structures. .. code-block:: console - # pre-clone the source code and then point spack develop to it - # note that we can clone into any repo/branch combination desired - $ git clone https://github.com/llnl/scr.git $SPACK_ENV/code - # note that with `--path` the code directory and package name can be different - $ spack develop --path $SPACK_ENV/code scr@3.1.0 + # Example: pre-clone the source code and then point spack develop to it + # We'll use a directory relative to our environment for this example. + # Assume your environment is in 'my-dev-env'. + $ mkdir -p my-dev-env/local-code + $ git clone https://github.com/llnl/scr.git my-dev-env/local-code/scr-custom-checkout + # Activate your environment if not already active + $ spack env activate ./my-dev-env + # Add scr to the environment if not already present + $ spack add scr@3.1.0 + # Now, tell Spack to use the local checkout for scr@3.1.0 + $ spack develop --path local-code/scr-custom-checkout scr@3.1.0 $ spack concretize -f Navigation and the Build Environment ---------- -Diving into the build environment was introduced previously in the packaging section with the -``spack build-env scr -- bash`` command. This is a helpful function because it allows you -to run commands inside the build environment. In the packages section of the tutorial -this was combined with ``spack cd`` to produce a manual build outside of Spack's automated -Process. -This command is particularly useful in developer environments -- it allows developers a streamlined -workflow when iterating on a single package without the overhead of the ``spack install`` command. -The additional features of the install command are unnecessary when tightly iterating between building - and testing a particular package. For example, the workflow modifying ``scr`` that we just went through - can be simplified to: +Diving into the build environment was introduced previously in the packaging tutorial +with the ``spack build-env -- bash`` command. This is a helpful feature because +it allows you to run commands interactively inside the build environment that Spack +sets up. In the packaging tutorial, this was combined with ``spack cd`` to demonstrate +a manual build outside of Spack's automated process. +This command is particularly useful in developer environments, offering a streamlined +workflow when iterating on a single package without the full overhead of the ``spack install`` +command each time. The dependency management and environment setup features of Spack +are still active, but you gain finer control over the build steps for the package +you are actively developing. For example, the workflow modifying ``scr`` that we +just went through can be simplified for rapid iteration: .. code-block:: console + # Enter the build environment for scr $ spack build-env scr -- bash - # Shell wrappers didn't propagate to the subshell - $ source $SPACK_ROOT/share/spack/setup-env.sh - # Let's look at navigation features - $ spack cd --help - $ spack cd -c scr - $ touch src/scr_copy.c + # If spack shell integration isn't active in the subshell, you might need to re-source it: + # $ source $SPACK_ROOT/share/spack/setup-env.sh + # (Alternatively, spack build-env tries to handle this for you) + + # Navigate to the source directory of scr + $ spack cd -s scr + # Make your code changes, e.g., edit src/scr_copy.c + # $ vim src/scr_copy.c (or your favorite editor) + # $ touch src/scr_copy.c # Example: simulate a change + + # Navigate to the build directory of scr $ spack cd -b scr - # Let's look at what's here + # Let's see what's here $ ls - # Build and run tests - $ make -j2 + # Build (e.g., assuming a Makefile-based system for scr) + $ make -j$(nproc) # Use available processors + # Run tests if available $ make test + # When done with this iteration, exit the subshell $ exit -Working with the build environment and along with spack navigation features -provides a nice way to iterate quickly and navigate through the hash heavy -spack directory structures. +Working directly within the build environment, along with Spack navigation features +(``spack cd``), provides a powerful way to iterate quickly and navigate through +the hash-heavy Spack directory structures without repeatedly running ``spack install``. Combinatorics ------------ -The final note we will look at in this tutorial will be the power of combinatoric -development builds. There are many instances where developers want to see how -a single set of changes affects multiple builds i.e. ``+cuda`` vs ``~cuda``, -``%gcc`` vs ``%clang``, ``build_type=Release`` vs ``build_type=Debug``, etc. +The final note we will look at in this tutorial will be the power of combinatorial +development builds. There are many instances where developers want to see how +a single set of local source code changes affects multiple build configurations +(e.g., ``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, ``build_type=Release`` vs +``build_type=Debug``). -Developers can achieve builds of both cases from a single ``spack install`` as -long as the develop spec is generic enough to cover the packages' spec variations +Developers can achieve builds of multiple variants from the same local source using +a single ``spack install`` command, as long as the ``spack develop`` spec +is generic enough (e.g., ``scr`` without specific variants) to apply to all desired +package spec variations in the environment. .. code-block:: console - # First we have to allow repeat specs in the environment - $ spack config add concretizer:unify:false - # Next we need to specify the specs we want ('==' propagates the variant to deps) - $ spack change macsio build_type==Release - $ spack add macsio+scr build_type==Debug - # Inspect the graph for multiple dev_path= + # First, we might need to allow multiple, non-unified versions of packages + # if we are testing different configurations of the same package. + # For this example, we assume 'macsio' will have two different specs. + # If scr itself had different variants we wanted to test from the same dev_path, + # we would add those variations of scr to the environment. + $ spack config add concretizer:unify:false # If testing different versions of 'scr' itself. + # For different 'macsio' using the *same* 'scr' dev version, + # this might not be strictly needed for 'scr' if 'scr' spec is unified. + + # Add two different configurations of macsio, both depending on our dev version of scr. + # Ensure 'scr' is added to the environment and marked with 'spack develop scr@version'. + $ spack add macsio build_type=Release ^scr@3.1.0 + $ spack add macsio build_type=Debug ^scr@3.1.0 + # Mark 'scr' for development (assuming it's already added with the correct version) + $ spack develop scr@3.1.0 + + # Concretize to see the plan. Both macsio specs should point to the same dev_path for scr. $ spack concretize -f - -While we won't build out this example it illustrates how the ``dev_path`` for -``build_type=Release`` and ``build_type=Debug`` points to the same source code. - -Now if we want to do most of our incremental builds using the ``Release`` build -and periodically check the results using the ``Debug`` build we can combine the -workflow from the previous example: dive into the ``Release`` versions build -environment using ``spack build-env scr build_type=Release -- bash`` and -navigate with ``spack cd -b scr build_type=Release``. Note that since there -are two ``scr`` specs in the environment we must distinguish which one we -want for these commands. When we are ready to check our changes for the debug -build we can exit out of the build environment subshell, -rerun ``spack install`` to rebuild everything, and then inspect the debug build -through our method of choice. + # $ spack spec -l macsio # to see details including dev_paths + +This setup illustrates how the ``dev_path`` for ``scr`` can be used by multiple +dependent specs (``macsio build_type=Release`` and ``macsio build_type=Debug``), +both pointing to the same local source code for ``scr``. When you run ``spack install``, +Spack will build both versions of ``macsio``, each linking against the ``scr`` built +from your local development path. + +Now, if we want to do most of our incremental builds targeting the ``Release`` +configuration of ``macsio`` (which uses our dev ``scr``), and periodically check +the results with the ``Debug`` configuration, we can combine workflows. +For focused iteration on `scr` that primarily affects the `Release` `macsio`: +Dive into the build environment of the `scr` that is a dependency of the `Release` `macsio`. +You might need to be specific if `scr` itself had variants: +``spack build-env scr -- bash`` (if `scr` spec is unambiguous or unified for dev) +or more specifically, find the hash for `scr` under the release `macsio` and use that. +Navigate with ``spack cd -b scr ...``. +When ready to check changes against the `Debug` build of `macsio`, exit any specific +build environment subshell, then run ``spack install``. This will rebuild your +local `scr` (if changed) and then rebuild both `macsio` configurations that depend on it. +You can then inspect or run the `Debug` version of `macsio`. diff --git a/tutorial_environments.rst b/tutorial_environments.rst index bbe91c5ea4..2b793a46a2 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -28,10 +28,10 @@ systems (e.g., `Python venv `_), but they are based around file formats (``spack.yaml`` and ``spack.lock``) that can be shared easily and re-used by others across systems. -Administering properly configured software involving lots of packages -and/or varying configuration requirements (e.g., different implementations -of ``mpi``) for multiple projects and efforts can be overwhelming. Spack -environments allow you to readily: +Administering software that involves many packages and/or varying +configuration requirements (e.g., different implementations of ``mpi``) +for multiple projects can be overwhelming. Spack environments allow you to +readily: * establish standard software requirements for your project(s); * set up run environments for users; @@ -43,7 +43,7 @@ environments allow you to readily: This tutorial introduces the basics of creating and using environments, then explains how to expand, configure, and build software in them. We will start with the command line interface, then cover editing key -environment file directly. We will describe the difference between +environment files directly. We will describe the difference between Spack-managed and independent environments, then finish with a section on reproducible builds. @@ -94,9 +94,9 @@ command: :language: console .. note:: - If you use the ``-p`` option for ``spack env activate``, Spack - will prepend the environment name to the prompt. This is a handy - way to be reminded if and which environment you are in. + If you use the ``-p`` or ``--prompt`` option for ``spack env activate``, + Spack will prepend the environment name to your shell prompt. This is a + handy way to be reminded if, and which, environment you are in. You can also use the shorter ``spacktivate`` alias for ``spack env activate``. @@ -153,9 +153,9 @@ Try the usual install command first: .. literalinclude:: outputs/environments/env-fail-install-1.out :language: console -Environments are special in that you must *add* specs to them before installing. -``spack add`` allows us to do queue up several specs to be installed together. -Let's try it: +Environments are special in that you must *add* specs to them before they +can be installed into the environment. ``spack add`` allows us to declare +these desired specs. Let's try it: .. literalinclude:: outputs/environments/env-add-1.out :language: console @@ -207,9 +207,10 @@ If you create environments incrementally, Spack ensures that already installed roots are not re-concretized. So, adding specs to an environment at a later point in time will not cause existing packages to rebuild. -Do note however that incrementally creating an environment can give you different -package versions from an environment created all at once. We will cover this after -we've discussed different concretization strategies. +Do note, however, that incrementally creating an environment can give you different +package versions from an environment created all at once if specs are concretized +at different times. We will cover this after we've discussed different +concretization strategies. Further, there are two other advantages of concretizing and installing an environment all at once: @@ -264,9 +265,10 @@ in your path: Uninstalling packages ^^^^^^^^^^^^^^^^^^^^^ -We can uninstall packages from an environment without affecting -other environments. This is possible since, while Spack shares -common installations, environments only link to those installations. +We can uninstall or remove packages from one environment without affecting +other environments or the centrally installed packages, as environments primarily +manage which packages are part of their configuration and view. +Spack shares common installations at the instance level. Let's demonstrate this feature by creating another environment. Suppose ``myproject`` requires ``trilinos`` but we have another @@ -290,17 +292,21 @@ contents of the environment: :language: console -We can see that ``trilinos`` won't be uninstalled because it is still referenced -in another environment managed by spack. If we want to remove it from the roots -list we need to use ``spack remove``: +We can see that ``trilinos`` won't be uninstalled from the Spack instance because +it is still referenced as a root in another environment (``myproject``). +If we want to remove it from the list of root specs in ``myproject2``, we need +to use ``spack remove`` (or its alias ``spack env remove``): .. literalinclude:: outputs/environments/env-remove-1.out :language: console -When the spec is first removed, we see that it is no longer a root but -is still present in the installed specs. Once we reconcretize, the -vestigial spec is removed. Now, it is no longer a root and will need -to be re-added before being installed as part of this environment. +When the spec is first removed, we see that it is no longer a root in the +current environment (``myproject2``) but is still listed under "installed specs" +if it was previously installed in this environment. After re-concretizing +(e.g. ``spack concretize -f`` or by running ``spack install`` which might trigger it), +Spack will determine if the package is still needed as a dependency or if it can be +removed from the environment's view (if no longer needed at all). +It is no longer a root and would need to be re-added to be explicitly part of this environment again. We know ``trilinos`` is still needed for the ``myproject`` environment, so let's switch back to confirm that it is still installed in that environment. @@ -400,7 +406,7 @@ to include the ``packages:mpi:require`` entry below: We introduce this here to show you how environment configuration can affect concretization. Configuration options are covered in much - more detail in the :ref:`configuration tutorial `. + more detail in the :ref:`Configuration Tutorial `. We've only scratched the surface here by requiring a specific @@ -491,22 +497,27 @@ code: .. code-block:: c #include + #include // For getpid() #include - #include + #include // Provided by zlib or zlib-ng int main(int argc, char **argv) { int rank; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); - printf("Hello world from rank %d\n", rank); + printf("Hello world from rank %d (PID: %d)\n", rank, getpid()); if (rank == 0) { - printf("zlib version: %s\n", ZLIB_VERSION); - printf("zlib-ng version: %s\n", ZLIBNG_VERSION); + // ZLIB_VERSION is standard for zlib compatible libraries + printf("zlib compatible version: %s\n", ZLIB_VERSION); + #ifdef ZLIBNG_VERSION // ZLIBNG_VERSION is specific to zlib-ng + printf("zlib-ng actual version: %s\n", ZLIBNG_VERSION); + #endif } MPI_Finalize(); + return 0; } This program includes headers from ``mpi`` and ``zlib``. @@ -519,10 +530,16 @@ Let's build and run our program: :language: console -Notice that we only needed to pass the include path to the -compiler. +Notice that while `mpicc` typically handles MPI-related paths, we would still +need to ensure the zlib headers and libraries are found by the compiler and linker. +Activating a Spack environment containing `zlib` (or `zlib-ng`) and `mpi` +often sets `CMAKE_PREFIX_PATH` or other variables that help build tools find these. +For direct `mpicc` calls, if `zlib` is not in a standard system location, you might need +flags like `-I -L -lz`. +However, the example output implies `mpicc` within the activated environment +successfully found everything. We also see that ``Hello world`` is output for each of the ranks -and the version of ``zlib`` used to build the program is printed. +and the version of the zlib-compatible library used to build the program is printed. We can confirm the version of ``zlib`` used to build the program is in our environment using ``spack find``: @@ -555,7 +572,7 @@ the environment configuration that we previously edited through ``spack config edit``. The ``spack.lock`` file is automatically generated during concretization. -The two files represent two fundamental concepts: +The two files represent two fundamental states or views of an environment: * ``spack.yaml``: *abstract* specs and configuration to install; and * ``spack.lock``: all fully *concrete* specs. @@ -782,11 +799,13 @@ Using ``spack.yaml`` """""""""""""""""""" An approximate build is created using the ``spack.yaml`` file. This -approach is relevant when we want to build the same specs on a new -platform, for example. It allows you to reproduce the environment -by preserving the abstract requirements in the file. However, the -software may actually build differently in part because the concretizer -may choose different dependencies. +This approach is relevant when we want to build the same set of desired software +on a new platform or at a later time, for example. It allows you to reproduce +the environment by preserving the abstract requirements from the ``spack.yaml`` +file. However, the software may build differently if the underlying Spack +configuration (compilers, external packages), package recipes, or default +versions in Spack have changed, as the concretizer might choose different +versions or dependencies to satisfy the abstract specs. Let's use ``spack env create`` to create an abstract environment from the file that we'll call ``abstract``: @@ -835,8 +854,10 @@ installed in the environment as we can see from calling .. note:: - Use of ``spack.lock`` to reproduce a build (currently) requires you - to be on the same type of machine. + Use of ``spack.lock`` to reproduce a build exactly (currently) requires + you to be on the same type of machine (OS, architecture) and often with a + compatible Spack instance (especially regarding compiler configurations and + paths to external dependencies if they are hardcoded or system-specific). ------------------- More information diff --git a/tutorial_modules.rst b/tutorial_modules.rst index a7420443ea..c858cd181a 100644 --- a/tutorial_modules.rst +++ b/tutorial_modules.rst @@ -12,11 +12,11 @@ Module Files Tutorial ===================== This tutorial illustrates how Spack can be used to generate module files -for the software that has been installed. Both hierarchical and non-hierarchical -deployments will be discussed in details and we will show how to customize +for installed software. Both hierarchical and non-hierarchical module +deployments will be discussed in detail, and we will show how to customize the content and naming of each module file. -At the end of the tutorial readers should have a clear understanding of: +At the end of this tutorial, readers will have a clear understanding of: * What module files are and how they are used on HPC clusters * How Spack generates module files for the software it installs @@ -42,11 +42,11 @@ earlier in the tutorial: $ spack uninstall -ay -and by enabling ``tcl`` module files, which are disabled by default since Spack v0.20: +and by enabling ``tcl`` module files (if they are not already enabled in your Spack version or site configuration), which were disabled by default in some Spack versions after v0.19: .. code-block:: console - $ spack config add "modules:default:enable:[tcl]" + $ spack config add "modules:default:enable:['tcl']" # Ensures tcl is in the list of enabled module systems ^^^^^^^^^^^^^^^^^^^ @@ -60,26 +60,32 @@ The first thing that we need is the module tool itself. In the tutorial we will $ spack install lmod -Once the module tool is installed we need to have it available in the -current shell. Installation directories in Spack's store are definitely not easy -to remember, but they can be retrieved with the ``spack location`` command: +Once the module tool is installed, we need to make its commands available in the +current shell. Spack installation directories can be complex, but they can be +retrieved easily with the ``spack location`` command. For Lmod, you typically +source an initialization script appropriate for your shell (e.g., `init/bash` +for bash, `init/csh` for csh, etc.): .. code-block:: console $ . $(spack location -i lmod)/lmod/lmod/init/bash -Now we can re-source the setup file and Spack modules will be put in -our module path. +Now, when Spack's environment setup script is sourced (or re-sourced), Spack's +generated module files will be discoverable by the module tool. If you are in +the root of your Spack installation, you can run: .. code-block:: console - $ . spack/share/spack/setup-env.sh + $ . share/spack/setup-env.sh -.. FIXME: this needs bootstrap support for ``lmod`` +Or, from any location, using the `$SPACK_ROOT` environment variable (if set): -.. FIXME: check the docs here, update them if necessary - If you need to install Lmod or Environment module you can refer - to the documentation `here `_. +.. code-block:: console + + $ . $SPACK_ROOT/share/spack/setup-env.sh + +Spack's main documentation provides further details if you need to install Lmod +or Environment Modules manually or encounter issues with their setup. ^^^^^^^^^^^^^^^^^^ @@ -113,7 +119,7 @@ To check which compilers are available you can use ``spack compiler list``: .. literalinclude:: outputs/modules/list-compiler.out :language: console -Finally, when you confirmed ``gcc@12.3.0`` is properly registered, clean the environment +Finally, once you have confirmed that ``gcc@12.3.0`` is properly registered, clean the environment with ``spack unload``: .. code-block:: console @@ -167,9 +173,9 @@ and to undo the modifications, you can use ``module unload``: Module Systems ^^^^^^^^^^^^^^ -There are two main module systems used in HPC, both installable by Spack. -In this tutorial we will be working with ``lmod`` and be showing examples -with both Tcl and Lua. +There are two main module systems used in HPC, both of which are installable by Spack. +In this tutorial, we will be working primarily with Lmod and showing examples +for both Tcl and Lua module files, as Lmod can handle both. """"""""""""""""""" @@ -192,8 +198,8 @@ we refer to its `documentation `_. """" Lmod """" -Lmod is a module system written in Lua, originally created at the -"Texas Advanced Computing Center" (TACC) by Robert McLay. You can get it with: +Lmod is a module system written in Lua, originally created at the +Texas Advanced Computing Center (TACC) by Robert McLay. You can get it with: .. code-block:: console @@ -297,13 +303,13 @@ the following content: tcl: all: filter: - exclude_env_vars: - - "CC" - - "CXX" - - "FC" - - "F77" + exclude_env_vars: # List of environment variables to not set in module files + - CC + - CXX + - FC + - F77 -This can be done either editing the configuration manually, or directly from the command line: +This can be done either by editing the configuration file manually or directly from the command line: .. code-block:: console @@ -360,10 +366,10 @@ directory. :language: console -if you look closely you'll see though that we went too far in +If you look closely, you'll see, though, that we went too far in excluding modules: the module for ``gcc@12.3.0`` disappeared as it was -bootstrapped with ``gcc@11``. To specify exceptions to the ``exclude`` -rules you can use ``include``: +bootstrapped with ``gcc@11`` (which we excluded). To specify exceptions to the ``exclude`` +rules, you can use ``include``: .. code-block:: yaml :emphasize-lines: 4,5 @@ -371,10 +377,10 @@ rules you can use ``include``: modules: default: tcl: - include: - - gcc + include: # Rules for specs to always include, takes precedence over exclude + - gcc # Matches any package named 'gcc' exclude: - - '%gcc@11' + - '%gcc@11' # Exclude anything built with gcc@11 all: filter: exclude_env_vars: @@ -393,9 +399,9 @@ you'll see that now the module for ``gcc@12.3.0`` has reappeared: .. literalinclude:: outputs/modules/module-avail-4.out :language: console -An additional feature that you can leverage to unclutter the environment -is to skip the generation of module files for implicitly installed -packages. In this case you only need to add the following line: +An additional feature that you can leverage to unclutter the module environment +is to skip the generation of module files for implicitly installed (i.e., dependency) +packages. In this case, you only need to add the following line: .. code-block:: yaml :emphasize-lines: 4 @@ -422,9 +428,9 @@ to ``modules.yaml`` and regenerate the module file tree as above. Change module file naming ^^^^^^^^^^^^^^^^^^^^^^^^^ -The next step in making module files more user-friendly is to -improve their naming scheme. -To reduce the length of the hash or remove it altogether you can +The next step in making module files more user-friendly is to +improve their naming scheme. Spack's default includes a long hash. +To reduce the length of the hash or remove it altogether, you can use the ``hash_length`` keyword in the configuration file: .. code-block:: yaml @@ -487,16 +493,20 @@ the names are formatted to differentiate them: netlib-scalapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}-{^mpi.name}' ^python^lapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}' -As you can see it is possible to specify rules that apply only to a -restricted set of packages using `anonymous specs -`_ -like ``^python^lapack``. Here we declare a conflict between any two modules -with the same name, so they cannot be loaded together. We also format the -names of modules according to compiler, compiler version, and MPI provider -name using the `spec format syntax -`_. -This allows us to match specs by their dependencies, and format them -based on their DAGs. +The ``projections`` section allows you to define different naming schemes for different sets of packages. +Spack will use the projection associated with the *most specific* anonymous spec that matches a given package. +For instance, a projection for ``netlib-scalapack`` is more specific than one for ``all``. +You can use `anonymous specs `_ +(specs without a root package name, like ``^python^lapack`` which matches anything depending on both Python and LAPACK) +to categorize packages. + +In this example, we also declare a Tcl-specific ``conflict`` between any two modules +with the same base name (e.g., two versions of `hdf5`), so they cannot be loaded together. +The module names themselves (which determine their paths in the module tree) are formatted +according to compiler, compiler version, and specific dependency names (like LAPACK provider and MPI provider) +using the `spec format syntax `_. +This allows us to match specs by their dependencies and format their module names based on their DAGs. +If no specific projection matches, a general projection (like for `all`) or a global `naming_scheme` (if defined, though `projections` are more flexible) would apply. .. literalinclude:: outputs/modules/tcl-refresh-5.out :language: console @@ -547,8 +557,8 @@ is installed. You can achieve this with Spack by adding an ^python^lapack: '{name}/{version}-{compiler.name}-{compiler.version}-{^lapack.name}' -Under the hood Spack uses the :meth:`~spack.spec.Spec.format` API to substitute -tokens in either environment variable names or values. There are two caveats though: +Under the hood, Spack uses the :meth:`~spack.spec.Spec.format` API to substitute +tokens in both environment variable names and their values. There are two caveats, however: - The set of allowed tokens in variable names is restricted to ``name``, ``version``, ``compiler``, ``compiler.name``, @@ -613,15 +623,14 @@ This time we will be more selective and regenerate only the ``openmpi`` module f :language: console -.. FIXME: remove this? ^^^^^^^^^^^^^^^^^^^^^ Autoload dependencies ^^^^^^^^^^^^^^^^^^^^^ -Spack can also generate module files that contain code to load the -dependencies automatically. You can, for instance generate python -modules that load their dependencies by adding the ``autoload`` +Spack can also generate module files that contain code to load their +dependencies automatically. You can, for instance, generate Python +modules that load their direct dependencies by adding the ``autoload`` directive and assigning it the value ``direct``: .. code-block:: yaml @@ -678,10 +687,11 @@ sufficient to omit the line setting ``verbose: true`` in the configuration file Hierarchical Module Files ------------------------- -So far we worked with non-hierarchical module files, i.e. with module files -that are all generated in the same root directory and don't attempt to -dynamically modify the ``MODULEPATH``. This results in a flat module structure where -all the software is visible at the same time: +So far, we have worked with non-hierarchical module files, i.e., module files +that are all generated into a single root directory (per module type like Tcl/Lmod) +and don't attempt to dynamically modify the ``MODULEPATH`` based on other loaded modules. +This results in a "flat" module structure where all generated software modules +are visible at the same time (after adding the root to `MODULEPATH`): .. literalinclude:: outputs/modules/lmod-intro-avail.out :language: console @@ -697,27 +707,31 @@ Even if ``conflicts`` directives are carefully placed in module files, they: - won't enforce a consistent environment, but will just report an error - need constant updates, for instance as soon as a new compiler or MPI library is installed -`Hierarchical module files `_ try to -overcome these shortcomings by showing at start-up only a restricted view of what is -available on the system: more specifically only the software that has been installed with -OS provided compilers. Among this software there will be other - usually more recent - compilers -that, once loaded, will prepend new directories to ``MODULEPATH`` unlocking all the software -that was compiled with them. This "unlocking" idea can then be extended arbitrarily to -virtual dependencies, as we'll see in the following section. +`Hierarchical module files `_ +(primarily an Lmod feature) try to overcome these shortcomings by initially showing +only a restricted view of what is available on the system. More specifically, +they often show only software that has been installed with OS-provided compilers +(or other "core" compilers). Among this software will be other, usually more recent, +compilers that, once loaded, will prepend new directories to ``MODULEPATH``, +"unlocking" all the software that was compiled with them. This "unlocking" idea +can then be extended arbitrarily to other dependencies like MPI libraries, as +we'll see in the following section. ^^^^^^^^^^^^^^^^^ Core/Compiler/MPI ^^^^^^^^^^^^^^^^^ -The most widely used hierarchy is the so called ``Core/Compiler/MPI`` where, on top -of the compilers, different MPI libraries also unlock software linked to them. -There are just a few steps needed to adapt the ``modules.yaml`` file we used previously: +The most widely used hierarchy is the so-called ``Core/Compiler/MPI`` structure, where, +in addition to compilers, different MPI libraries also unlock software linked to them. +There are just a few steps needed to adapt the ``modules.yaml`` file we used previously +for non-hierarchical Tcl modules to this Lmod hierarchical setup: #. enable the ``lmod`` file generator #. change the ``tcl`` tag to ``lmod`` #. remove the ``tcl`` specific ``conflict`` directive - #. declare which compilers are considered ``core_compilers`` - #. remove the ``mpi`` related suffixes in projections (as they will be substituted by hierarchies) + #. declare which compilers are considered ``core_compilers`` (these form the base of the hierarchy) + #. define the hierarchy components (e.g., `mpi`) using the `hierarchy` key + #. remove MPI-related suffixes from `projections`, as these distinctions will now be handled by the hierarchy structure itself After these modifications your configuration file should look like: @@ -764,10 +778,10 @@ After these modifications your configuration file should look like: that only ``lmod`` will be active (see `Overriding entire sections `_ for more details). -The directive ``core_compilers`` accepts a list of compilers. Everything built -using these compilers will create a module in the ``Core`` part of the hierarchy, -which is the entry point for hierarchical module files. It is -common practice to put the OS provided compilers in the list and only build common utilities +The ``core_compilers`` directive accepts a list of compiler specs. Everything built +using these compilers will have its module file placed in the ``Core`` part of the +hierarchy, which is the entry point for hierarchical module files. It is +common practice to list OS-provided compilers here and primarily build common utilities and other compilers with them. If we now regenerate the module files: @@ -779,23 +793,28 @@ and update ``MODULEPATH`` to point to the ``Core``: .. code-block:: console - $ module purge - $ module unuse $HOME/spack/share/spack/modules/linux-ubuntu18.04-x86_64 - $ module use $HOME/spack/share/spack/lmod/linux-ubuntu18.04-x86_64/Core + $ module purge # Clear existing modules + # The following paths depend on your Spack installation and system configuration. + # Adjust $HOME/spack to your $SPACK_ROOT if different. + # The 'linux-ubuntu...' part is OS-dependent. + $ module unuse $HOME/spack/share/spack/modules/linux-ubuntu22.04-x86_64 # Path from previous non-hierarchical Tcl setup + $ module use $HOME/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core asking for the available modules will return: .. literalinclude:: outputs/modules/module-avail-6.out :language: console -Unsurprisingly, the only visible module is ``gcc``. Loading that we'll unlock -the ``Compiler`` part of the hierarchy: +Unsurprisingly, after this setup, the modules initially available might be restricted (e.g., only core compilers like `gcc@11.x.y` if it's in `core_compilers`). +Loading a compiler module that is *not* a core compiler (e.g., `gcc@12.3.0` if it was built by a core compiler) will then "unlock" +the next level of the hierarchy, revealing software built with `gcc@12.3.0`. .. literalinclude:: outputs/modules/module-avail-7.out :language: console -The same holds true also for the ``MPI`` part, that you can enable by loading -either ``mpich`` or ``openmpi``. Let's start by loading ``mpich``: +The same "unlocking" principle applies to the ``MPI`` part of the hierarchy (as defined by the `hierarchy: [mpi]` setting). +You enable it by loading an MPI implementation (e.g., ``mpich`` or ``openmpi``) that was itself built +with the currently loaded compiler (e.g., `gcc@12.3.0`). Let's start by loading `mpich` (assuming it was built with `gcc@12.3.0`): .. literalinclude:: outputs/modules/module-avail-8.out :language: console @@ -809,11 +828,12 @@ over a non-hierarchical one: .. literalinclude:: outputs/modules/module-swap-mpi.out :language: console -``Lmod`` took care of swapping the MPI provider for us, and it also substituted the -``netlib-scalapack`` module to conform to the change in the MPI. -In this way we can't accidentally pull-in two different MPI providers at the -same time or load a module file for a package linked to ``openmpi`` when ``mpich`` is also loaded. -Consistency for compilers and MPI is ensured by the tool. +Lmod took care of swapping the MPI provider for us. When `openmpi` was loaded, +it automatically unloaded `mpich` and any modules that depended on `mpich` (like +the `netlib-scalapack` built with `mpich`). It then made available the `netlib-scalapack` +module built with `openmpi`. This ensures that you cannot easily load conflicting MPI +implementations or packages built with an incompatible MPI. Consistency for compilers +and MPI is largely ensured by Lmod's hierarchy mechanism. ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -821,12 +841,14 @@ Add LAPACK to the hierarchy ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The hierarchy just shown is already a great improvement over non-hierarchical layouts, -but it still has an asymmetry: ``LAPACK`` providers cover the same semantic role -as ``MPI`` providers, but yet they are not part of the hierarchy. +but it still has an asymmetry in our example: ``LAPACK`` providers (like `openblas` or `netlib-lapack`) +cover a similar semantic role to ``MPI`` providers (selecting an implementation for a virtual dependency), +but they are not yet part of the hierarchy in our configuration. -To be more practical, this means that although we have gained an improved consistency in -our environment when it comes to ``MPI``, we still have the same problems as we had before -for ``LAPACK`` implementations: +To be more practical, this means that although we have gained improved consistency in +our environment when it comes to ``MPI``, we might still have the same potential for +conflicts or inconsistencies with ``LAPACK`` implementations if multiple are available +and not managed by the hierarchy: .. literalinclude:: outputs/modules/lapack-conflict.out :language: console @@ -839,21 +861,23 @@ among multiple providers quickly becomes quite involved. For instance, having both ``MPI`` and ``LAPACK`` in the hierarchy means we must classify software into one of four categories: - #. Software that doesn't depend on ``MPI`` or ``LAPACK`` - #. Software that depends only on ``MPI`` - #. Software that depends only on ``LAPACK`` - #. Software that depends on both + 1. Software that doesn't depend on ``MPI`` or ``LAPACK``. + 2. Software that depends only on ``MPI``. + 3. Software that depends only on ``LAPACK``. + 4. Software that depends on both ``MPI`` and ``LAPACK``. -to decide when to show it to the user. The situation becomes more involved as the number of virtual -dependencies in the hierarchy increases. +This classification determines when a module should be shown to the user based on the +loaded compiler, MPI, and LAPACK. The situation becomes more involved as the number of +virtual dependencies in the hierarchy increases. We can take advantage of the DAG that Spack maintains for the installed software and solve this combinatorial problem in a clean and automated way. In some sense Spack's ability to manage this combinatorial complexity makes deeper hierarchies feasible. -Coming back to our example, let's add ``lapack`` to the hierarchy and -remove the remaining suffix projection for ``lapack``: +Coming back to our example, let's add ``lapack`` to the hierarchy definition in +our ``modules.yaml`` and remove the remaining suffix projection for ``lapack``, +as the hierarchy will now handle this distinction: .. code-block:: yaml :emphasize-lines: 10 @@ -980,13 +1004,19 @@ Next, we need to create our custom template extension in the folder listed above .. code-block:: jinja - {% extends "modules/modulefile.lua" %} + {% extends "modules/modulefile.lua" %} {# This should be the path to Spack's base Lmod template #} {% block footer %} - -- Access is granted only to specific groups + {{ super() }} {# This includes the original content of the footer block, if any #} + + -- Custom footer: Access is granted only to specific groups. + -- This example uses isDir to check if the user can access the installation prefix. + -- A more robust check might involve os.execute or other site-specific calls. if not isDir("{{ spec.prefix }}") then - LmodError ( - "You don't have the necessary rights to run \"{{ spec.name }}\".\n\n", - "\tPlease write an e-mail to 1234@foo.com if you need further information on how to get access to it.\n" + LmodError( + "Access to '{{ spec.name }}' (installed at: {{ spec.prefix }}) " .. + "appears to be restricted for your current user/group.\n\n" .. + "You may not have the necessary directory permissions to access this software.\n" .. + "Please write an e-mail to 1234@foo.com if you believe this is an error or to request access.\n" ) end {% endblock %} @@ -1078,10 +1108,11 @@ the right e-mail address where to ask for it! Restore settings for future sections ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -For future sections of the tutorial, we will not use the ``gcc@12.3.0`` -compiler. Since it is currently the default compiler (our current -default is the most recent version of gcc available), we will remove -it now. +For future sections of the tutorial, we will not assume the ``gcc@12.3.0`` +compiler is the primary or default compiler. To ensure consistency with other +tutorial sections that might rely on system defaults or other specific compilers, +we will remove it from Spack's known compilers now. This also means it won't +be considered for default compiler selections by Spack unless re-added. .. code-block:: console diff --git a/tutorial_packaging.rst b/tutorial_packaging.rst index 5e5aef51bf..ec9032d6b4 100644 --- a/tutorial_packaging.rst +++ b/tutorial_packaging.rst @@ -22,8 +22,8 @@ which is an MPI debugging tool. What is a Spack Package? ------------------------ -Spack packages are installation scripts, which are essentially -recipes for building (and testing) software. +Spack packages are Python classes that act as recipes for building +and installing software. They define properties and behavior of the build, such as: @@ -35,24 +35,25 @@ They define properties and behavior of the build, such as: They can also define checks of the installed software that can be performed after the installation. -Once we've specified a package's recipe, users can ask Spack to -build the software with different features on any of the supported -systems. +Once we've specified a package's recipe (the `package.py` file), users can +ask Spack to build the software with different versions, compilers, variants (features), +and dependencies on any Spack-supported system. --------------- Getting Started --------------- -In order to avoid modifying your Spack installation with the package we -are creating, add a **package repository** just for this tutorial by -entering the following command: +In order to avoid adding our new test package directly into Spack's built-in +package repository (which is good practice during development and for custom packages), +we first create a new, separate **package repository** just for this tutorial. +Enter the following command to create and register this new repository with Spack: .. literalinclude:: outputs/packaging/repo-add.out :language: console -Doing this ensures changes we make here do not adversely affect other -parts of the tutorial. You can find out more about repositories at -`Package Repositories `_. +Doing this ensures that the package we create is isolated and does not +interfere with your main Spack instance or other tutorial sections. You can +find out more about repositories at `Package Repositories `_. ------------------------- Creating the Package File @@ -64,12 +65,12 @@ Creating the Package File is set to the name or path of your preferred text editor. -Suppose you want to install software that depends on mpileaks but found -Spack did not already have a built-in package for it. This means you are -going to have to create one. +Suppose you want to install software that depends on `mpileaks`, but you found +that Spack does not already have a built-in package for it. This means you +need to create one. -Spack's *create* command builds a new package from a template by taking -the location of the package's source code and using it to: +Spack's `spack create` command helps you start a new package file. +It takes the URL of the package's source code (e.g., a tarball) and uses it to: * fetch the code; * create a package skeleton; and @@ -89,27 +90,27 @@ run ``spack create`` with the URL: .. literalinclude:: outputs/packaging/create.out :language: console -You should now be in your text editor of choice, with the ``package.py`` -file open for editing. +You should now be in your text editor of choice, with a new file named +``package.py`` open for editing. -Your ``package.py`` file should reside in the ``tutorial-mpileaks`` -subdirectory of your tutorial repository's ``packages`` directory, i.e., +Based on the output from `spack create`, this ``package.py`` file for +``tutorial-mpileaks`` will be located in your new repository, typically at a path like: ``$SPACK_ROOT/var/spack/repos/tutorial/packages/tutorial-mpileaks/package.py`` +(The exact path to the `tutorial` repository might vary if you changed the default location when running `spack repo add`). Take a moment to look over the file. As we can see from the skeleton contents, shown below, the Spack template: -* provides instructions for how to contribute your package to - the Spack repository; -* indicates that the software is built with Autotools; -* provides a docstring template; -* provides an example homepage URL; -* shows how to specify a list of package maintainers; -* specifies the version directive, with checksum, for the software; -* shows a dependency directive example; and -* provides a skeleton ``configure_args`` method. +* provides commented instructions on how to contribute your package to Spack's built-in repository (if desired). +* indicates the detected build system (e.g., Autotools, CMake, etc.) or provides a generic `Package` base class. +* includes a template for the package's docstring (its description). +* provides a placeholder for the software's homepage URL. +* shows how to specify a list of package maintainers (GitHub usernames). +* includes a `version` directive, often with a checksum, for the downloaded source code. +* may show an example `depends_on` directive for dependencies. +* provides a skeleton method relevant to the detected build system (e.g., `configure_args` for Autotools, `cmake_args` for CMake). .. literalinclude:: tutorial/examples/packaging/0.package.py :caption: tutorial-mpileaks/package.py (from tutorial/examples/packaging/0.package.py) @@ -124,9 +125,9 @@ template: maintain a Spack package for their own software and/or rely on software maintained by other people. -Since we are providing a ``url``, we can confirm the checksum, or ``sha256`` -calculation. Exit your editor to return to the command line and use the -``spack checksum`` command: +Since `spack create` usually fills in the `sha256` checksum based on the downloaded tarball, +it should be correct. However, if you needed to manually verify or add a checksum for a new version later, +you would download the tarball and then use the `spack checksum ` command: .. literalinclude:: outputs/packaging/checksum-mpileaks-1.out :language: console @@ -148,7 +149,7 @@ by trying to install the package using the ``spack install`` command: It clearly did not build. The error indicates ``configure`` is unable to find the installation location of a dependency. -So let's start to customize the package for our software. +So let's start customizing the generated `package.py` file for our software. ---------------------------- Adding Package Documentation @@ -156,7 +157,7 @@ Adding Package Documentation First, let's fill in the documentation. -Bring mpileaks' ``package.py`` file back into your ``$EDITOR`` with the +Bring the `tutorial-mpileaks` package file back into your ``$EDITOR`` with the ``spack edit`` command: .. code-block:: console @@ -169,8 +170,8 @@ Let's make the following changes: * replace the first ``FIXME`` comment with a description of ``mpileaks`` in the docstring; * replace the homepage property with the correct link; and -* uncomment the ``maintainers`` directive and add your GitHub user name. -* add the license of the project and your GitHub user name. +* uncomment the ``maintainers`` directive and add your GitHub username(s). +* add the `license` string for the project (e.g., `license("BSD-3-Clause")`). .. note:: @@ -214,18 +215,16 @@ command will become more informative. .. note:: - More information on using Autotools packages is provided in - `AutotoolsPackage - `_. + More information on using Autotools packages is provided in the documentation for + `AutotoolsPackage `_. The full list of build systems known to Spack can be found at - `Build Systems - `_. + `Build Systems `_. - More information on the build-time tests can be found at - ``_. + More information on build-time tests can be found in the Packaging Guide section on + `Build-time Tests `_. - Refer to the links at the end of this section for more information. + Refer to the links at the end of this tutorial for more information. Now we're ready to start filling in the build recipe. @@ -233,9 +232,9 @@ Now we're ready to start filling in the build recipe. Adding Dependencies ------------------- -First we'll add the dependencies determined by reviewing documentation -in the software's repository (https://github.com/LLNL/mpileaks). The -``mpileaks`` software relies on three third-party libraries: +First, we'll add the dependencies determined by reviewing the `mpileaks` +documentation (e.g., its README or INSTALL files, often found in the source repository: https://github.com/LLNL/mpileaks). +The ``mpileaks`` software relies on three main third-party libraries: * ``mpi``, * ``adept-utils``, and @@ -243,10 +242,10 @@ in the software's repository (https://github.com/LLNL/mpileaks). The .. note:: - Luckily, all of these dependencies are built-in packages in Spack; - otherwise, we would have to create packages for them as well. + Luckily, all of these dependencies are already packaged in Spack; + otherwise, we would have to create Spack packages for them first. -Bring mpileaks' ``package.py`` file back up in your ``$EDITOR`` with +Bring the `tutorial-mpileaks` package file back up in your ``$EDITOR`` with the ``spack edit`` command: .. code-block:: console @@ -268,12 +267,12 @@ installed *before* it can build our package. .. note:: The ``mpi`` dependency is different from the other two in that it is - a *virtual dependency*. That means Spack must satisfy the dependency - with a package that *provides* the ``mpi`` interface, such as ``openmpi`` - or ``mvapich2``. - - We call such packages **providers**. More information on virtual dependencies - can be found in the *Packaging Guide* linked at the bottom of this tutorial. + a *virtual dependency*. This means that ``mpi`` is an interface, not a + specific package. Spack must satisfy this dependency with an actual package + that *provides* the ``mpi`` interface, such as ``openmpi``, ``mpich``, + or ``mvapich2``. We call these concrete implementation packages **providers**. + More information on virtual dependencies can be found in the Packaging Guide + (see the link at the end of this tutorial). Let's check that dependencies are effectively built when we try to install ``tutorial-mpileaks``: @@ -298,10 +297,10 @@ It found that: Debugging Package Builds ------------------------ -Our ``tutorial-mpileaks`` package is still not building due to the -``adept-utils`` package's ``configure`` error. Experienced -Autotools developers will likely already see the problem and -its solution. +Our ``tutorial-mpileaks`` package is still not building. The error from the +previous `spack install` attempt indicated that the `configure` script for +`tutorial-mpileaks` failed, likely because it couldn't find `adept-utils`. +Experienced Autotools developers might guess the cause and solution. But let's take this opportunity to use Spack features to investigate the problem. Our options for proceeding are: @@ -313,9 +312,9 @@ the problem. Our options for proceeding are: Reviewing the Build Log ~~~~~~~~~~~~~~~~~~~~~~~ -The build log might yield some clues so let's look at the contents of -the ``spack-build-out.txt`` file at the path recommended above by our -failed installation: +The build log might yield some clues. You can view the log file directly (Spack +prints its location on error) or use the command `spack build-log tutorial-mpileaks`. +Let's assume its contents are similar to: .. literalinclude:: outputs/packaging/build-output.out :language: console @@ -329,10 +328,14 @@ Most importantly, the last line is very clear: the installation path of the .. note:: - Spack automatically adds standard include and library directories - to the compiler's search path *but* it is not uncommon for this - information to not get picked up. Some software, like ``mpileaks``, - requires the paths to be explicitly provided on the command line. + Spack automatically adds standard include and library directories of + dependencies to the compiler's search paths (e.g., via `CPATH`, `LIBRARY_PATH`, + and by passing flags to compiler wrappers). However, it's not uncommon for + Autotools `configure` scripts or other build systems to not automatically + pick up these paths for all dependencies, or they might require explicit + options like `--with-=`. Some software, like ``mpileaks`` + in this example, requires the paths to certain dependencies to be explicitly + provided on the `configure` command line. So let's investigate further from the staged build directory. @@ -340,20 +343,20 @@ So let's investigate further from the staged build directory. Building Manually ~~~~~~~~~~~~~~~~~ -First let's try to build the package manually to see if we can -figure out how to solve the problem. +First, let's try to build the package manually within the Spack build +environment to see if we can figure out how to solve the problem. -Let's move to the build directory using the ``spack cd`` command: +Let's navigate to the build directory using the ``spack cd --stage tutorial-mpileaks`` +command (or `spack cd -s tutorial-mpileaks` to go to the source directory first, then navigate to the build path): .. code-block:: console $ spack cd tutorial-mpileaks -You should now be in the appropriate stage directory since this -command moves us into the working directory of the last attempted -build. If not, you can ``cd`` into the directory above that contained -the ``spack-build-out.txt`` file then into it's ``spack-src`` -subdirectory. +You should now be in the appropriate stage (source) directory for `tutorial-mpileaks`. +Spack performs out-of-source builds for Autotools, so the actual build happens +in a separate directory, usually a sibling to `spack-src` within the stage area. +The `spack build-env` command below will drop you into the correct build directory. Now let's ensure the environment is properly set up using the ``spack build-env`` command: @@ -362,9 +365,10 @@ Now let's ensure the environment is properly set up using the $ spack build-env tutorial-mpileaks bash -This command spawned a new shell containing the same environment -that Spack used to build the ``tutorial-mpileaks`` package. (Feel -free to substitute your favorite shell for ``bash``.) +This command spawns a new shell (``bash`` in this case, but you can use others) +with the same environment variables (paths, compilers, etc.) that Spack +would use to build the ``tutorial-mpileaks`` package. It also changes the +current directory to the package's build directory. .. note:: @@ -381,9 +385,9 @@ And we get the same results as before. Unfortunately, the output does not provide any additional information that can help us with the build. -Given that this is a simple package built with ``configure`` and we know -that installation directories need to be specified, we can use its -help to see what command line options are available for the software. +Given that this is a simple package built with `configure` and we suspect +that installation directories for dependencies need to be specified, we can +use `configure --help` to see what command-line options are available. .. literalinclude:: outputs/packaging/configure-help.out :language: console @@ -422,7 +426,7 @@ So let's add the configuration arguments for specifying the paths to the two concrete dependencies in the ``configure_args`` method of our package. -Bring mpileaks' ``package.py`` file back up in your ``$EDITOR`` with +Bring the `tutorial-mpileaks` package file back up in your ``$EDITOR`` with the ``spack edit`` command: .. code-block:: console @@ -469,25 +473,26 @@ as opposed to simply allowing them to be enabled or disabled. :language: console :emphasize-lines: 18-23 -According to the software's documentation (https://github.com/LLNL/mpileaks), -the integer values for the ``--with-stack-start-*`` options represent the -numbers of calls to shave off of the top of the stack traces for each -language, effectively reducing the noise of internal mpileaks library function +According to the software's documentation (e.g., output from `./configure --help` +or its `README`), the integer values for the ``--with-stack-start-*`` options +represent the number of initial stack frames to exclude from traces for each +language. This can help reduce noise from internal mpileaks library function calls in generated traces. For simplicity, we'll use one variant to supply the value for both arguments. -Supporting this optional feature will require two changes to the package: +Supporting this optional feature in our Spack package will require two changes to the `package.py` file: * add a ``variant`` directive; and * change the configure options to use the value. -Let's add the variant to expect an ``int`` value with a default of -``0``. Defaulting to ``0`` effectively disables the option. Also change -``configure_args`` to retrieve the value and add the corresponding -configure arguments when a non-zero value is provided by the user. +Let's add the variant to expect an `int` value with a default of `0`. +Defaulting to `0` effectively means the feature is off or uses the software's +own default behavior if the configure flags are omitted. We will also modify +`configure_args` to retrieve the variant's value and add the corresponding +configure arguments only when a non-zero value is provided by the user. -Bring mpileaks' ``package.py`` file back up in your ``$EDITOR`` with +Bring the `tutorial-mpileaks` package file back up in your ``$EDITOR`` with the ``spack edit`` command: .. code-block:: console @@ -502,9 +507,9 @@ and add the ``variant`` directive and associated arguments as follows: :language: python :emphasize-lines: 16-21,45-52 -Notice that the ``variant`` directive is translated into a ``variants`` dictionary -in ``self.spec``. Also note that the value provided by the user is accessed -by the entry's ``value`` property. +Notice that the `variant` directive results in an entry in the `self.spec.variants` +dictionary-like object. The value specified by the user (or the default) is +accessed using `self.spec.variants['stackstart'].value`. Now run the installation again with the ``--verbose`` install option -- to get more output during the build -- and the new ``stackstart`` package option: @@ -559,9 +564,10 @@ Installing again we can see we've fixed the problem. :language: console This is just scratching the surface of testing an installation. We could -leverage the examples from this package to add post-install phase tests -and/or stand-lone tests. Refer to the links at the bottom for more -information on checking an installation. +leverage the examples from this package (if it has any) to add more comprehensive +post-install phase tests (e.g., by overriding `check()`) or build-time tests. +Refer to the links at the bottom of this tutorial for more information on +checking an installation. ------------------------ @@ -590,21 +596,21 @@ Querying Spec Versions You can customize the build based on the version of the package, compiler, and dependencies. Examples of each are: -* Am I building my package with version ``1.1`` or greater? +* Is the current package version `1.1` or greater? .. code-block:: python if self.spec.satisfies("@1.1:"): # Do things needed for version 1.1 or newer -* Am I building with a ``gcc`` version up to ``5.0``? +* Is the current compiler `gcc` with a version up to `5.0`? .. code-block:: python if self.spec.satisfies("%gcc@:5.0"): # Add arguments specific to gcc's up to 5.0 -* Is my ``dyninst`` dependency at least version ``8.0``? +* Is the `dyninst` dependency (if present) at least version `8.0`? .. code-block:: python @@ -618,7 +624,7 @@ Querying Spec Names If the build has to be customized to the concrete version of an abstract ``Spec`` you can use its ``name`` property. For example: -* Is ``openmpi`` the MPI I'm building with? +* Is `openmpi` the concrete provider for the virtual `mpi` dependency? .. code-block:: python @@ -632,7 +638,7 @@ Querying Variants Adjusting build options based on enabled variants can be done by querying the ``Spec`` itself, such as: -* Am I building with the ``debug`` variant? +* Is the `debug` variant enabled for the current package? .. code-block:: python @@ -649,20 +655,22 @@ of your package. You can find these packages in Multiple Build Systems ---------------------- -There are cases where software actively supports two build systems, or changes -build systems as it evolves, or needs different build systems on different platforms. -Spack allows you to write a single, neat recipe for these cases too. It will only -require a slight change in the recipe's structure compared to what we have seen -so far. +There are cases where software actively supports two build systems, changes +build systems as it evolves, or needs different build systems on different +platforms. Spack allows you to write a single, clean package recipe for these +scenarios. This typically requires inheriting from multiple build system base +classes and using conditional logic based on the spec. Let's take ``uncrustify``, a source code beautifier, as an example. This software used to build with Autotools until version 0.63, and then switched build systems to CMake at version 0.64. -Compared to previous recipes in this tutorial, in this case we need ``Uncrustify`` to -inherit from both ``CMakePackage`` and ``AutotoolsPackage``. We also need to explicitly -specify the ``build_system`` directive, and add conditional dependencies based on -build system: +Compared to previous recipes in this tutorial, for a package like `Uncrustify` +that supports multiple build systems, the class would inherit from all applicable +build system base classes (e.g., `CMakePackage`, `AutotoolsPackage`). +We also need to explicitly specify the allowed values for the built-in `build_system` +variant and usually provide a default. Conditional dependencies can then be added +based on the selected `build_system` variant: .. code-block:: python @@ -675,28 +683,39 @@ build system: version("0.64", commit="1d7d97") version("0.63", commit="44ce0f") - build_system( - conditional("cmake", when="@0.64:"), - conditional("autotools", when="@:0.63"), - default="cmake", - ) + # Declare supported build systems and when they apply + build_system("cmake", "autotools", default="cmake") + + # Conditional logic based on version for selecting the build system + # This logic usually goes into the package's __init__ or a helper method + # if more complex, or can be implicitly handled if versions only support one. + # For simplicity, Spack's 'build_system' variant itself can be conditional + # or one might use 'conflicts' for unsupported combinations. + # A more direct way for this version-based switch is often handled by + # having different methods or conditional logic within shared methods. + # The 'build_system' variant itself is how users can *choose*, if multiple are valid for a spec. + # Example of conditional dependency based on the chosen build system: with when("build_system=cmake"): depends_on("cmake@3.18:", type="build") - -We didn't mention it so far, but each spec has a ``build_system`` variant that specifies -the build system it uses. In most cases that variant has a single allowed value, inherited from the -corresponding base package - so, usually, you don't have to think about it. - -When your package supports more than one build system though, you have to explicitly declare which ones are -allowed and under which conditions. In the example above it's ``cmake`` for version 0.64 and higher and -``autotools`` for version 0.63 and lower. - -The ``build_system`` variant can also be used to declare other properties which are conditional on the build -system being selected. For instance, above we declare that when using ``cmake``, CMake 3.18+ is required. - -The other relevant difference, compared to the previous recipes we have seen so far, is that the code prescribing -the installation procedure will live into two separate classes: + # No specific build dependency for autotools shown here, but could be added. + +Spack has a built-in `build_system` variant (values like `autotools`, `cmake`, etc.). +If a package class inherits from only one build system base class (e.g., `AutotoolsPackage`), +this variant typically has only one allowed value. +When a package supports multiple build systems (by inheriting from multiple such base classes), +you must declare the allowed values for the `build_system` variant in your `package.py` +(e.g., `build_system("cmake", "autotools", default="cmake")`). +You can then use `when="version@X:"` or other context in `conflicts` directives or +conditional logic in methods to guide or restrict the choice of build system based on version +or other properties. For instance, `uncrustify` versions `@0.64:` might only support `cmake`, +while versions `@:0.63` only support `autotools`. + +The `build_system` variant choice also dictates which `Builder` internal class Spack uses. +The installation logic specific to each build system (like arguments to `cmake` or `configure`) +will live in methods within these corresponding `Builder` classes (e.g., `CMakeBuilder`, `AutotoolsBuilder`). +Spack automatically selects the correct `Builder` based on the resolved `build_system` variant. +You define these `Builder` classes as inner classes in your `package.py`: .. code-block:: python @@ -708,8 +727,8 @@ the installation procedure will live into two separate classes: def configure_args(self): pass -Depending on the ``spec``, and more specifically on the value of the ``build_system`` directive, a ``Builder`` -object will be instantiated from one of the two classes when an installation is requested from a user. +Depending on the resolved spec (specifically the value of its `build_system` variant), +an instance of the corresponding `Builder` inner class will be used to drive the build process. ----------- Cleaning Up @@ -731,51 +750,34 @@ recipes. The `Packaging Guide `_ more thoroughly covers packaging topics. -Additional information on key topics can be found at the links below. +Additional information on key topics can be found at the links below: -~~~~~~~~~~~~~~~~~~~~~~~ -Testing an installation -~~~~~~~~~~~~~~~~~~~~~~~ +**Testing an installation** + +* `Checking an installation `_: + For more information on adding tests that run at build-time and against an installation. + +**Customizing package-related environments** + +* `Retrieving Library Information `_: + For supporting unique configuration options needed to locate libraries. +* `Modifying a Package's Build Environment `_: + For customizing package and dependency build and run environments. + +**Using other build systems** + +* `Build Systems `_: + For the full list of built-in build systems. +* `Spack Package Build Systems tutorial `_: + For tutorials on common build systems. +* `Multiple Build Systems `_: + For a reference on writing packages with multiple build systems. +* `Package Class Architecture `_: + For more insight on the inner workings of ``Package`` and ``Builder`` classes. +* `The GDAL Package `_: + For an example of a complex package which extends Python and supports two build systems. + +**Making a package externally detectable** -* `Checking an installation - `_: - for more information on adding tests that run at build-time and against an installation - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Customizing package-related environments -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -* `Retrieving Library Information - `_: - for supporting unique configuration options needed to locate libraries -* `Modifying a Package's Build Environment - `_: - for customizing package and dependency build and run environments - -~~~~~~~~~~~~~~~~~~~~~~~~~ -Using other build systems -~~~~~~~~~~~~~~~~~~~~~~~~~ - -* `Build Systems - `_: - for the full list of built-in build systems -* `Spack Package Build Systems tutorial - `_: - for tutorials on common build systems -* `Multiple Build Systems - `_: - for a reference on writing packages with multiple build systems -* `Package Class Architecture - `_: - for more insight on the inner workings of ``Package`` and ``Builder`` classes. -* `The GDAL Package - `_: - for an example of a complex package which extends Python and supports two build systems. - -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Making a package externally detectable -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -* `Making a package externally discoverable - `_: - for making a package discoverable using the ``spack external find`` command +* `Making a package externally discoverable `_: + For making a package discoverable using the ``spack external find`` command. From 98bb7f12594d6c75f6048f78c554f3c22b8db485 Mon Sep 17 00:00:00 2001 From: Harmen Stoppels Date: Tue, 27 May 2025 23:41:23 +0200 Subject: [PATCH 2/2] scripting --- tutorial_scripting.rst | 196 +++++++++++++++++++++++------------------ 1 file changed, 109 insertions(+), 87 deletions(-) diff --git a/tutorial_scripting.rst b/tutorial_scripting.rst index a3edec8ca4..52d8d0f972 100644 --- a/tutorial_scripting.rst +++ b/tutorial_scripting.rst @@ -12,13 +12,13 @@ Scripting with Spack ==================== This tutorial introduces advanced Spack features related to scripting. -Specifically, we will show you how to write scripts using ``spack find`` -and ``spack python``. +Specifically, we will show you how to write scripts using features of +``spack find`` for machine-readable output and how to leverage +``spack python`` for more complex scripting tasks. Earlier sections of the tutorial demonstrated using ``spack find`` to -list and search installed packages. -The ``spack python`` command gives you access to all of Spack's `internal -APIs `_, allowing -you to write more complex queries, for example. +list and search installed packages for human users. +The ``spack python`` command gives you access to all of Spack's internal +Python APIs, allowing you to write powerful custom queries and automation scripts. Since Spack has an extensive API, we'll only scratch the surface here. We'll give you enough information to start writing your own scripts and @@ -28,27 +28,26 @@ to find what you need, with a little digging. Scripting with ``spack find`` ----------------------------- -So far, the output we've seen from ``spack find`` has been for human -consumption. But you can take advantage of some advanced options of -the command to generate machine-readable output suitable for piping -to a script. +So far, the output we've seen from ``spack find`` has been primarily for human +consumption. However, ``spack find`` also provides options to generate +machine-readable output suitable for piping to other command-line tools or +for use in scripts. ^^^^^^^^^^^^^^^^^^^^^^^ ``spack find --format`` ^^^^^^^^^^^^^^^^^^^^^^^ -The main job of ``spack find`` is to show the user a bunch of concrete -specs that correspond to installed packages. By default, we display them -with some default attributes, like the ``@version`` suffix you're used to -seeing in the output. +The main job of ``spack find`` is to display information about concrete +specs that correspond to installed packages. By default, it displays them +with common attributes like the version suffix (e.g., ``@1.2.3``) and compiler. -The ``--format`` argument allows you to display the specs however you -choose, using custom format strings. Format strings let you specify the -names of particular *parts* of the specs you want displayed. Let's see -the first option in action. +The ``--format`` argument allows you to customize the output string for each +found package. Format strings use curly braces ``{}`` to denote placeholders +that Spack will fill with attributes of the ``Spec`` object. Let's see this +in action. -Suppose you only want to display the *name*, *version*, and first ten (10) -characters of the *hash* for every package installed in your Spack +Suppose you only want to display the *name*, *version*, and the first ten (10) +characters of the package *hash* for every package installed in your Spack instance. You can generate that output with the following command: .. literalinclude:: outputs/scripting/find-format.out @@ -56,8 +55,12 @@ instance. You can generate that output with the following command: :emphasize-lines: 1 Note that ``name``, ``version``, and ``hash`` are attributes of Spack's -internal ``Spec`` object and enclosing them in braces ensures they are -output according to your format string. +internal ``Spec`` object. Enclosing them in braces (e.g., ``{name}``) in the +format string tells Spack to substitute their values. You can also use format +specifiers, like ``{hash:10s}`` to get the first 10 characters of the hash +string, or ``{/hash}`` to get the full hash prefixed by a slash. +Many attributes of a ``Spec`` can be used here (see `spack help --spec-format` +for a full list). Using ``spack find --format`` allows you to retrieve just the information you need to do things like pipe the output to typical UNIX command-line @@ -77,7 +80,8 @@ get attributes for all installations of ``zlib-ng`` by entering: The ``spack find --json`` command gives you everything we know about the specs in a structured format. You can pipe its output to -JSON filtering tools like ``jq`` to extract just the parts you want. +JSON filtering tools like `jq `_ +to extract just the parts you want. Check out the `basic usage docs `_ @@ -90,9 +94,11 @@ Introducing the ``spack python`` command What if we need to perform more advanced queries? -Spack provides the ``spack python`` command to launch a python interpreter -with Spack's python modules available to import. It uses the underlying -python for the rest of its commands. So you can write scripts to: +Spack provides the ``spack python`` command to launch a Python interpreter +session with Spack's Python modules automatically added to the `PYTHONPATH`. +This means you can directly import and use Spack's internal APIs. +It uses the same Python interpreter that Spack itself uses. +You can write scripts to: - run Spack commands; - explore abstract and concretized specs; and @@ -104,37 +110,42 @@ Let's launch a Spack-aware python interpreter by entering: :language: console :emphasize-lines: 1,5 -Since we are in a python interpreter, use ``exit()`` to end +Since we are in a Python interpreter, use ``exit()`` or ``quit()`` to end the session and return to the terminal. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Accessing the ``Spec`` object ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Now let's take a look at the internal representation of the Spack ``Spec``. -As you already know, specs can be either *abstract* or *concrete*. The -specs you've seen in ``package.py`` files (e.g., in the ``install()`` -method) have been *concrete*, or fully specified. The specs you've typed -on the command line have been *abstract*. Understanding the differences -between the two types is key to using Spack's internal API. +Now let's take a look at the internal representation of a Spack ``Spec`` object. +As you already know, specs can be either *abstract* (not fully defined, e.g., ``zlib``) +or *concrete* (all choices like version, compiler, variants, etc., are determined). +The specs you typically work with inside a `package.py` file's methods (like +``self.spec`` in the ``install()`` method) are *concrete*. The specs you often +type on the command line (e.g., ``spack install zlib%gcc``) are initially *abstract* +and then Spack *concretizes* them. Understanding this distinction is key to +using Spack's internal API effectively, as some attributes are only available +on concrete specs. -Let's open another python interpreter with ``spack python``, instantiate -the ``zlib`` spec, and check a few properties of an abstract spec: +Let's open another Python interpreter with ``spack python``, import the ``Spec`` +class, instantiate an abstract spec for ``zlib``, and check a few of its properties: .. literalinclude:: outputs/scripting/spack-python-abstract.out :language: console :emphasize-lines: 1-3,5,11,13 -Notice that there are ``Spec`` properties and methods that are not -accessible to abstract specs; specifically: +Notice that for an abstract spec: -- an exception -- ``SpecError`` -- is raised if we try to access its - ``version``; -- there are no associated ``versions``; and -- the spec's operating system is ``None``. +- an attempt to access its ``version`` (which is a single, concrete version) + would typically raise a ``SpecError`` or return an undefined-like state because + an abstract spec doesn't have *a* version yet, it has a `VersionList` of possibilities. +- the ``versions`` property (a ``VersionList``) might be empty (any version allowed) or + contain version ranges, but not a single concrete version. +- other properties like ``architecture`` (which includes the OS) might be ``None`` or + represent a non-specific default. -Now, without exiting the interpreter, let's concretize the spec and try -again: +Now, without exiting the interpreter, let's concretize this spec using the +``concretize()`` method and observe the changes: .. literalinclude:: outputs/scripting/spack-python-concrete.out :language: console @@ -142,12 +153,13 @@ again: Notice that the concretized spec now: -- has a ``version``; -- has a single entry in its ``versions`` list; and -- the operating system is now ``ubuntu22.04``. +- now has a specific ``version`` (e.g., ``Version('1.2.13')``). +- its ``versions`` list now represents a single, concrete version. +- attributes like ``architecture`` (and thus ``os``, ``platform``, ``target``) are now specific (e.g., the OS might be `ubuntu22.04`). -It is not necessary to store the intermediate abstract spec -- you can -use the ``.concretized()`` method as shorthand: +It is not necessary to store the intermediate abstract spec if you only need +the concrete one. You can use the ``.concretized()`` method as a shorthand to +get a concrete spec directly (this creates a new concrete spec from the abstract one): .. literalinclude:: outputs/scripting/spack-python-sans-intermediate.out :language: console @@ -157,33 +169,35 @@ use the ``.concretized()`` method as shorthand: Querying the Spack database ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Even more powerful queries are available when we look at the information -stored in the Spack database. The ``Database`` object in Spack is in the -``spack.store.STORE.db`` variable. We'll interact with it mainly through the -``query()`` method. Let's see the documentation available for ``query()`` -using python's built-in ``help()`` function: +Even more powerful queries are available when we interact with the information +stored in Spack's installation database. You can access Spack's database object +via `spack.store.db` (after `import spack.store`). We'll interact with it +mainly through its ``query()`` method. Let's see the documentation available +for ``query()`` using Python's built-in ``help()`` function: .. literalinclude:: outputs/scripting/spack-python-db-query-help.out :language: console :emphasize-lines: 1-2,9-13 -We will primarily make use of the ``query_spec`` argument. +We will primarily make use of the first argument (which can be a spec string +or `Spec` object to match against) and the `installed=True` keyword argument +to query only installed packages. -Recall that queries using the ``spack find`` command are limited to -queries of attributes with matching values, not values they do *not* -have. In other words, we cannot use the ``spack find`` command for -all packages that *do not* satisfy a certain criterion. +Recall that queries using the ``spack find`` command are primarily for finding +packages that *match* certain criteria. It's harder to use `spack find` +for queries that involve *excluding* packages based on complex criteria +(e.g., "does *not* depend on X" while depending on Y). -We *can* use the python interface to write these types of queries. -For example, let's find all packages that were compiled with ``gcc`` but -do not depend on ``mpich``. We can do this by using custom python code -and Spack database queries. We will use the ``spack.cmd.display_specs`` -for output to achieve the same printing functionality as the ``spack -find`` command: +We *can* use the Python interface to write these more complex types of queries. +For example, let's find all installed packages that were compiled with any version +of ``gcc`` but do *not* depend on ``mpich``. We can do this by iterating +through specs compiled with `gcc` and then checking their dependencies. +For displaying the results, instead of relying on internal display functions, +we can simply print the spec strings. .. literalinclude:: outputs/scripting/spack-python-db-query-exclude.out :language: console - :emphasize-lines: 1-5 + :emphasize-lines: 1-9 Now we have a powerful query not available through ``spack find``. @@ -200,17 +214,19 @@ Using scripts ^^^^^^^^^^^^^ Now let's parameterize our script to accept arguments on the command -line. With a few generalizations to use the include and exclude specs +line. With a few generalizations to use the include and exclude spec strings as arguments, we can create a powerful, general-purpose query script. -Open a file called ``find_exclude.py`` in your preferred editor -and add the following code: +Create a file named ``find_exclude.py`` in your current directory with your +preferred text editor and add the following Python code: .. literalinclude:: outputs/scripting/0.find_exclude.py.example :language: python -Notice we added importing and using the system package (``sys``) -to access the first and second command line arguments. +Notice we added importing and using Python's built-in ``sys`` module +to access command-line arguments via ``sys.argv``. +The script expects two arguments: an "include" spec string and an "exclude" +spec string. Now we can run our new script by entering the following: @@ -218,8 +234,8 @@ Now we can run our new script by entering the following: :language: console :emphasize-lines: 1 -This is *great* for us, as long as we remember to use Spack's -``python`` command to run it. +This works well when invoked with `spack python find_exclude.py ...`, +as `spack python` ensures that Spack's libraries are in the `PYTHONPATH`. ------------------------------------- Using the ``spack-python`` executable @@ -228,13 +244,14 @@ Using the ``spack-python`` executable What if we want to make our script available for others to use without the hassle of having to remember to use ``spack python``? -We can take advantage of the shebang line typically added as the -first line of python executable files. But there is a catch, as -we will soon see. +We can make the script directly executable by adding a "shebang" line +(e.g., `#!/usr/bin/env spack python`) at the very beginning of the script +and making the file executable (e.g., `chmod +x find_exclude.py`). +This tells the system to use `spack python` to interpret the script. +However, there's a common catch with shebang lines on some systems. -Open the ``find_exclude.py`` script we created above in your preferred -editor and add the shebang line with ``spack python`` as the arguments -to ``env``: +Let's try it. Open the ``find_exclude.py`` script again and add the +following shebang line at the top: .. literalinclude:: outputs/scripting/1.find_exclude.py.example :language: python @@ -247,14 +264,17 @@ running it as follows: :language: console :emphasize-lines: 1-2 -If you are lucky, it worked on your system, but there is no guarantee. -Some systems only support a single argument on the shebang line (see -`here `_). -``spack-python``, which is a wrapper script for ``spack python``, solves -this issue. +If you are lucky, this might work on your system. However, there's no guarantee, +as some operating systems only support a single argument on the shebang line after +the interpreter (see `here `_ for details). +In `#!/usr/bin/env spack python`, `spack` is the command and `python` is its argument. -Bring up the file in your editor again and change the ``env`` argument -to ``spack-python`` as follows: +To make the script more portable across systems, Spack provides a helper executable +called `spack-python`. This wrapper script is designed to be used in shebang lines +and correctly invokes `spack python`. + +Bring up the file in your editor again and change the shebang line to use +``spack-python``: .. literalinclude:: outputs/scripting/2.find_exclude.py.example :language: python @@ -266,10 +286,12 @@ Exit your editor and let's run the script again: :language: console :emphasize-lines: 1 -Congratulations! It will now work on any system with Spack installed. +Congratulations! The script should now be directly executable (e.g., `./find_exclude.py ...`) +on any system where Spack's `bin` directory (which includes `spack-python`) is in the `PATH`. You now have the basic tools to create your own custom Spack queries and -prototype ideas. We hope one day you'll contribute them back to Spack. +prototype ideas using Spack's Python API. We hope you'll find this useful +and perhaps even contribute your scripts or ideas back to the Spack community. .. LocalWords: LLC Spack's APIs hdf zlib literalinclude json uniq jq .. LocalWords: docs concretized REPL API SpecError spec's py ubuntu