Skip to content

Commit

Permalink
explain incrementally creating an environment (#235)
Browse files Browse the repository at this point in the history
  • Loading branch information
@haampie
haampie committed May 20, 2023
1 parent 4d70d6e commit 67e76ae
Show file tree
Hide file tree
Showing 4 changed files with 175 additions and 28 deletions.
8 changes: 8 additions & 0 deletions outputs/environments.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,14 @@ spack config add packages:all:providers:mpi:[mpich]
example environments/concretize-f-1 "spack concretize --force --fresh"
# spack install

example --tee environments/incremental-1 "spack env activate --temp"
spack env activate --temp
example environments/incremental-1 "spack install --add python"
example environments/incremental-1 "spack install --add py-numpy@1.20 2>&1 | tail -n1"
example environments/incremental-2 "spack add py-numpy@1.20"
example environments/incremental-2 "spack concretize -f"
spack env deactivate

example --tee environments/show-mpicc-1 "spack env activate myproject2"
spack env activate myproject2
example environments/show-mpicc-1 "spack env status"
Expand Down
34 changes: 34 additions & 0 deletions outputs/environments/incremental-1.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
$ spack env activate --temp
$ spack install --add python
==> Concretized python
[+] wd4batz python@3.10.10%gcc@11.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,f2fd060 arch=linux-ubuntu22.04-x86_64_v3
[+] dca2qyg ^bzip2@1.0.8%gcc@11.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] zdl3dic ^diffutils@3.9%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ctxzisf ^expat@2.5.0%gcc@11.3.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] rr7ondv ^libbsd@0.11.7%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ksnahtg ^libmd@1.0.4%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ba4juc3 ^gdbm@1.23%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] p4obvgx ^gettext@0.21.1%gcc@11.3.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 7wr75ce ^libiconv@1.17%gcc@11.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] pcbenki ^libxml2@2.10.3%gcc@11.3.0~python build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] e77cf6a ^tar@1.34%gcc@11.3.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
[+] aln73eo ^pigz@2.7%gcc@11.3.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+] qoo4rlo ^zstd@1.5.5%gcc@11.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] t7yjn4d ^libffi@3.4.4%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 6oeb4y7 ^libxcrypt@4.4.33%gcc@11.3.0~obsolete_api build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] j5tavds ^perl@5.36.0%gcc@11.3.0+cpanm+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] 7sohpaz ^berkeley-db@18.1.40%gcc@11.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
[+] 4dokmxj ^ncurses@6.4%gcc@11.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] w2by2b2 ^openssl@1.1.1t%gcc@11.3.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
[+] 5pxkrf4 ^ca-certificates-mozilla@2023-01-10%gcc@11.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] mh73nkp ^pkgconf@1.9.5%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] tddm2ff ^readline@8.2%gcc@11.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
[+] ismbqj3 ^sqlite@3.40.1%gcc@11.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] lyay7bs ^util-linux-uuid@2.38.1%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 4kpkw5a ^xz@5.4.1%gcc@11.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] mntflxr ^zlib@1.2.13%gcc@11.3.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.04-x86_64_v3

==> All of the packages are already installed
==> Updating view at /tmp/spack-b8vmqegs/.spack-env/view
$ spack install --add py-numpy@1.20 2>&1 | tail -n1
. Couldn't concretize without changing the existing environment. If you are ok with changing it, try `spack concretize --force`. You could consider setting `concretizer:unify` to `when_possible` or `false` to allow multiple versions of some packages.
66 changes: 66 additions & 0 deletions outputs/environments/incremental-2.out
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
$ spack add py-numpy@1.20
==> Adding py-numpy@1.20 to environment /tmp/spack-b8vmqegs
$ spack concretize -f
==> Concretized py-numpy@1.20
- ofv5kha py-numpy@1.20.3%gcc@11.3.0+blas+lapack build_system=python_pip patches=802970a,873745d arch=linux-ubuntu22.04-x86_64_v3
[+] eaq3ss4 ^openblas@0.3.23%gcc@11.3.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-ubuntu22.04-x86_64_v3
[+] j5tavds ^perl@5.36.0%gcc@11.3.0+cpanm+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] 7sohpaz ^berkeley-db@18.1.40%gcc@11.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
- edgyet5 ^py-cython@0.29.33%gcc@11.3.0 build_system=python_pip patches=71de066 arch=linux-ubuntu22.04-x86_64_v3
- c4ua5tu ^py-pip@23.0%gcc@11.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
- woysmgw ^py-setuptools@59.4.0%gcc@11.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
- dwc7cdk ^py-wheel@0.37.1%gcc@11.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
- veukjrw ^python@3.9.8%gcc@11.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu22.04-x86_64_v3
[+] dca2qyg ^bzip2@1.0.8%gcc@11.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] zdl3dic ^diffutils@3.9%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ctxzisf ^expat@2.5.0%gcc@11.3.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] rr7ondv ^libbsd@0.11.7%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ksnahtg ^libmd@1.0.4%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ba4juc3 ^gdbm@1.23%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] p4obvgx ^gettext@0.21.1%gcc@11.3.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 7wr75ce ^libiconv@1.17%gcc@11.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] pcbenki ^libxml2@2.10.3%gcc@11.3.0~python build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] e77cf6a ^tar@1.34%gcc@11.3.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
[+] aln73eo ^pigz@2.7%gcc@11.3.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+] qoo4rlo ^zstd@1.5.5%gcc@11.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] t7yjn4d ^libffi@3.4.4%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 6oeb4y7 ^libxcrypt@4.4.33%gcc@11.3.0~obsolete_api build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 4dokmxj ^ncurses@6.4%gcc@11.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] w2by2b2 ^openssl@1.1.1t%gcc@11.3.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
[+] 5pxkrf4 ^ca-certificates-mozilla@2023-01-10%gcc@11.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] mh73nkp ^pkgconf@1.9.5%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] tddm2ff ^readline@8.2%gcc@11.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
[+] ismbqj3 ^sqlite@3.40.1%gcc@11.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] lyay7bs ^util-linux-uuid@2.38.1%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 4kpkw5a ^xz@5.4.1%gcc@11.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] mntflxr ^zlib@1.2.13%gcc@11.3.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.04-x86_64_v3

==> Concretized python
- veukjrw python@3.9.8%gcc@11.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu22.04-x86_64_v3
[+] dca2qyg ^bzip2@1.0.8%gcc@11.3.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] zdl3dic ^diffutils@3.9%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ctxzisf ^expat@2.5.0%gcc@11.3.0+libbsd build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] rr7ondv ^libbsd@0.11.7%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ksnahtg ^libmd@1.0.4%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] ba4juc3 ^gdbm@1.23%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] p4obvgx ^gettext@0.21.1%gcc@11.3.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 7wr75ce ^libiconv@1.17%gcc@11.3.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] pcbenki ^libxml2@2.10.3%gcc@11.3.0~python build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] e77cf6a ^tar@1.34%gcc@11.3.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-x86_64_v3
[+] aln73eo ^pigz@2.7%gcc@11.3.0 build_system=makefile arch=linux-ubuntu22.04-x86_64_v3
[+] qoo4rlo ^zstd@1.5.5%gcc@11.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] t7yjn4d ^libffi@3.4.4%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 6oeb4y7 ^libxcrypt@4.4.33%gcc@11.3.0~obsolete_api build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] j5tavds ^perl@5.36.0%gcc@11.3.0+cpanm+open+shared+threads build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] 7sohpaz ^berkeley-db@18.1.40%gcc@11.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-x86_64_v3
[+] 4dokmxj ^ncurses@6.4%gcc@11.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] w2by2b2 ^openssl@1.1.1t%gcc@11.3.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-x86_64_v3
[+] 5pxkrf4 ^ca-certificates-mozilla@2023-01-10%gcc@11.3.0 build_system=generic arch=linux-ubuntu22.04-x86_64_v3
[+] mh73nkp ^pkgconf@1.9.5%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] tddm2ff ^readline@8.2%gcc@11.3.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-x86_64_v3
[+] ismbqj3 ^sqlite@3.40.1%gcc@11.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] lyay7bs ^util-linux-uuid@2.38.1%gcc@11.3.0 build_system=autotools arch=linux-ubuntu22.04-x86_64_v3
[+] 4kpkw5a ^xz@5.4.1%gcc@11.3.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-x86_64_v3
[+] mntflxr ^zlib@1.2.13%gcc@11.3.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.04-x86_64_v3

==> Updating view at /tmp/spack-b8vmqegs/.spack-env/view
95 changes: 67 additions & 28 deletions tutorial_environments.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,42 @@ Now confirm the contents of the environment using ``spack find``:

We can see that the roots and all their dependencies have been installed.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Creating an environment incrementally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

As a short-hand, you can use the ``install --add`` flag to accomplish
the same thing in one step:

.. code-block:: console
$ spack install --add tcl trilinos
This both adds the specs to the environment and installs them.

You can also add and install specs to an environment incrementally. For example:

.. code-block:: console
$ spack install --add tcl
$ spack install --add trilinos
If you create environments incrementally, Spack ensures that already installed
roots are not re-concretized. So, adding specs to an environment at a later point
in time will not cause existing packages to rebuild.

Do note however that incrementally creating an environment can give you different
package versions from an environment created all at once. We will cover this after
we've discussed different concretization strategies.

Further, there are two other advantages of concretizing and installing an environment
all at once:

* If you have a number of specs that can be installed together,
adding them first and installing them together enables them to
share dependencies and reduces total installation time.

* You can launch all builds in parallel by taking advantage of Spack's `install-level build parallelism <https://spack.readthedocs.io/en/latest/packaging_guide.html#install-level-build-parallelism>`_.

^^^^^^^^^^^^^^
Using packages
Expand Down Expand Up @@ -274,31 +310,8 @@ spec. Spack uses reference counting to ensure that we don't remove
Trilinos would only have been uninstalled by Spack if it were
no longer needed by any environments or their package dependencies.


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Shorthand for ``add`` and ``remove``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

We've seen how to use ``add`` and ``remove`` to add and remove roots from
an environment, and we've seen how to ``install`` all of the roots
at once, and how to ``uninstall`` particular packages.

It may seem tedious to have to ``add`` then ``install`` (or
``uninstall`` then ``remove``) every spec in an environment. You can
combine these steps into single commands if you want:

* ``spack install --add``: add and install a package
* ``spack uninstall --remove`` uninstall and remove a package

However, there are are advantages to processing all the specs
of an environment at once:

* If you have a number of specs that can be installed together,
adding them first and installing them together enables them to
share dependencies and reduces total installation time.

* You can launch all builds in parallel by taking advantage of Spack's `install-level build parallelism <https://spack.readthedocs.io/en/latest/packaging_guide.html#install-level-build-parallelism>`_.

You can also uninstall a package and remove it from the environment
in one go with ``spack uninstall --remove trilinos``.

-----------------------
The ``spack.yaml`` file
Expand Down Expand Up @@ -336,9 +349,14 @@ The ``specs`` list should look familiar; these are the specs we've been
modifying with ``spack add``.

``concretizer:unify:true``, the default, means that they are concretized
*together*, so that there is only one version of any package in the
environment. We'll cover ``unify:false`` and ``unify:when_possible`` later,
in the Stacks tutorial.
*together*, so that there is only one version of each package in the
environment. Other options for ``unify`` are ``false`` and ``when_possible``.
``false`` means that the specs are concretized *independently*, so that
there may be multiple versions of the same package in the environment.
``when_possible`` lies between those options. In this case, Spack will unify
as many packages in the environment, but will not fail if it cannot unify
all of them.


^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Editing environment configuration
Expand Down Expand Up @@ -412,6 +430,27 @@ all the environment's specs:
All the specs are now concrete **and** ready to be installed with
``mpich`` as the MPI implementation.

Re-concretization is sometimes also necessary when creating an
environment *incrementally* with unification enabled. Spack makes
sure that already concretized specs in the environment are not
modified when adding something new.

Adding and installing specs one by one leads to greedy
concretization. When you first install ``python`` in an environment,
Spack will pick a recent version for it. If you then add ``py-numpy``,
it may be in conflict with the ``python`` version already installed,
and fail to concretize:

.. literalinclude:: outputs/environments/incremental-1.out
:language: console

The solution is to re-concretize the environment as a whole,
which causes ``python`` to downgrade to a version compatible
with ``py-numpy``:

.. literalinclude:: outputs/environments/incremental-2.out
:language: console

------------------------
Building in environments
------------------------
Expand Down

0 comments on commit 67e76ae

Please sign in to comment.