Release v0.10.2

.github/workflows/ci.yml

@@ -131,6 +131,10 @@ jobs:
             conda env update --file install/gen_deps_environment.yml
 
             pip install ax-platform==0.2.8
+
+        - name: Install surmise
+          if: matrix.os != 'macos-latest' && steps.cache.outputs.cache-hit != 'true'
+          run: |
             pip install --upgrade git+https://github.com/surmising/surmise.git@develop
 
         - name: Build ytopt and dependencies

.wci.yml

@@ -17,8 +17,8 @@ description: |
 language: Python
 
 release:
-  version: 0.10.1
-  date: 2023-07-10
+  version: 0.10.2
+  date: 2023-07-24
 
 documentation:
   general: https://libensemble.readthedocs.io

CHANGELOG.rst

@@ -8,6 +8,26 @@ GitHub issues are referenced, and can be viewed with hyperlinks on the `github r
 
 .. _`github releases page`: https://github.com/Libensemble/libensemble/releases
 
+Release 0.10.2
+--------------
+
+:Date: July 24, 2023
+
+* Fixes issues with workflow directories:
+  * Ensure relative paths are interpreted from where libEnsemble is run. #1020
+  * Create intermediate directories for workflow paths. #1017
+
+* Fixes issue where libEnsemble pre-initialized a shared multiprocssing queue. #1026
+
+:Note:
+
+* Tested platforms include Linux, MacOS, Windows and major systems including Frontier (OLCF), Polaris (ALCF), Perlmutter (NERSC), Theta (ALCF) and Bebop. The major system tests ran heterogeneous workflows.
+
+:Known issues:
+
+* On systems using SLURM 23.02, some issues have been experienced when using ``mpi4py`` comms.
+* See the known issues section in the documentation for more information (https://libensemble.readthedocs.io/en/main/known_issues.html).
+
 Release 0.10.1
 --------------
 
@@ -27,7 +47,6 @@ Hotfix for breaking changes in Pydantic.
 
 * See known issues section in the documentation.
 
-
 Release 0.10.0
 --------------
 
@@ -36,16 +55,16 @@ Release 0.10.0
 New capabilities:
 
 * Enhance portability and simplify the assignment of procs/GPUs to worker resources #928 / #983
- * Auto-detect GPUs across systems (inc. Nvidia, AMD, and Intel GPUs).
- * Auto-determination of GPU assignment method by MPI runner or provided platform.
- * Portable `auto_assign_gpus` / `match_procs_to_gpus` and `num_gpus` arguments added to the MPI executor submit.
- * Add `set_to_gpus` function (similar to `set_to_slots`).
- * Allow users to specify known systems via option or environment variable.
- * Allow users to specify their own system configurations.
- * These changes remove a number of tweaks that were needed for particular platforms.
+  * Auto-detect GPUs across systems (inc. Nvidia, AMD, and Intel GPUs).
+  * Auto-determination of GPU assignment method by MPI runner or provided platform.
+  * Portable `auto_assign_gpus` / `match_procs_to_gpus` and `num_gpus` arguments added to the MPI executor submit.
+  * Add `set_to_gpus` function (similar to `set_to_slots`).
+  * Allow users to specify known systems via option or environment variable.
+  * Allow users to specify their own system configurations.
+  * These changes remove a number of tweaks that were needed for particular platforms.
 
 *  Resource management supports GPU and non-GPU simulations in the same ensemble. #993
- * User's can specify `num_procs` and `num_gpus` in the generator for each evaluation.
+  * User's can specify `num_procs` and `num_gpus` in the generator for each evaluation.
 
 * Pydantic models are used for validating major libE input (input can be provided as classes or dictionaries). #878
 * Added option to store output and ensemble directories in a workflow directory. #982
@@ -71,10 +90,10 @@ Documentation:
 Tests and Examples:
 
 * Updated forces_gpu tutorial example. #956
- * Source code edit is not required for the GPU version.
- * Reports whether running on device or host.
- * Increases problem size.
- * Added versions with persistent generator and multi-task (GPU v non-GPU).
+  * Source code edit is not required for the GPU version.
+  * Reports whether running on device or host.
+  * Increases problem size.
+  * Added versions with persistent generator and multi-task (GPU v non-GPU).
 * Moved multiple tests, generators, and simulators to the community repo.
 * Added ytopt example. And updated heFFTe example. #943
 * Support Python 3.11 #922

README.rst

@@ -1,4 +1,4 @@
-.. image:: docs/images/libEnsemble_Logo.svg
+.. image:: https://raw.githubusercontent.com/Libensemble/libensemble/main/docs/images/libE_logo.png
    :align: center
    :alt: libEnsemble
 
@@ -7,6 +7,14 @@
 .. image:: https://img.shields.io/pypi/v/libensemble.svg?color=blue
    :target: https://pypi.org/project/libensemble
 
+.. image:: https://img.shields.io/conda/v/conda-forge/libensemble?color=blue
+   :target: https://anaconda.org/conda-forge/libensemble
+
+.. image:: https://img.shields.io/spack/v/py-libensemble?color=blue
+   :target: https://spack.readthedocs.io/en/latest/package_list.html#py-libensemble
+
+|
+
 .. image:: https://github.com/Libensemble/libensemble/workflows/libEnsemble-CI/badge.svg?branch=main
    :target: https://github.com/Libensemble/libensemble/actions
 
@@ -73,7 +81,7 @@ Resources
   @article{Hudson2022,
     title   = {{libEnsemble}: A Library to Coordinate the Concurrent
                Evaluation of Dynamic Ensembles of Calculations},
-    author  = {Stephen Hudson and Jeffrey Larson and John-Luke Navarro and Stefan Wild},
+    author  = {Stephen Hudson and Jeffrey Larson and John-Luke Navarro and Stefan M. Wild},
     journal = {{IEEE} Transactions on Parallel and Distributed Systems},
     volume  = {33},
     number  = {4},
@@ -82,6 +90,7 @@ Resources
     doi     = {10.1109/tpds.2021.3082815}
   }
 
+.. _Community Examples repository: https://github.com/Libensemble/libe-community-examples
 .. _conda-forge: https://conda-forge.org/
 .. _Contributions: https://github.com/Libensemble/libensemble/blob/main/CONTRIBUTING.rst
 .. _docs: https://libensemble.readthedocs.io/en/main/advanced_installation.html
@@ -90,6 +99,6 @@ Resources
 .. _libEnsemble Slack page: https://libensemble.slack.com
 .. _MPICH: http://www.mpich.org/
 .. _mpmath: http://mpmath.org/
-.. _Quickstart: https://libensemble.readthedocs.io/en/main/introduction.html
 .. _PyPI: https://pypi.org
+.. _Quickstart: https://libensemble.readthedocs.io/en/main/introduction.html
 .. _ReadtheDocs: http://libensemble.readthedocs.org/

docs/advanced_installation.rst

@@ -9,8 +9,9 @@ automatically installed alongside libEnsemble:
 * Python_ 3.8 or above
 * NumPy_
 * psutil_
-* setuptools_
 * pydantic_
+* pyyaml_
+* tomli_
 
 In view of libEnsemble's compiled dependencies, the following installation
 methods each offer a trade-off between convenience and the ability
@@ -28,13 +29,6 @@ To install the latest PyPI release::
 
     pip install libensemble
 
-The above comes with required dependencies only. To install with some
-common user function dependencies (as used in the examples and tests)::
-
-    pip install libensemble[extras]
-
-Note that since ``PETSc`` will build from source, this may take a while.
-
 To pip install libEnsemble from the latest develop branch::
 
     python -m pip install --upgrade git+https://github.com/Libensemble/libensemble.git@develop
@@ -177,15 +171,16 @@ the given system (rather than building from scratch). This may include
 ``Python`` and the packages distributed with it (e.g., ``numpy``), and will
 often include the system MPI library.
 
-.. _GitHub: https://github.com/Libensemble/libensemble
 .. _Conda: https://docs.conda.io/en/latest/
 .. _conda-forge: https://conda-forge.org/
+.. _GitHub: https://github.com/Libensemble/libensemble
 .. _MPICH: https://www.mpich.org/
 .. _NumPy: http://www.numpy.org
 .. _`Open MPI`: https://www.open-mpi.org/
 .. _psutil: https://pypi.org/project/psutil/
 .. _pydantic: https://pydantic-docs.helpmanual.io/
+.. _pyyaml: https://github.com/yaml/pyyaml
 .. _Python: http://www.python.org
-.. _setuptools: https://setuptools.pypa.io/en/latest/
 .. _Spack: https://spack.readthedocs.io/en/latest
 .. _spack_libe: https://github.com/Libensemble/spack_libe
+.. _tomli: https://github.com/hukkin/tomli

docs/function_guides/sim_gen_alloc_api.rst

@@ -11,7 +11,7 @@ libEnsemble package.
 
 :doc:`See here for more in-depth guides to writing user functions<function_guide_index>`
 
-As of v0.9.3+dev, valid simulator and generator functions
+As of v0.10.0, valid simulator and generator functions
 can *accept and return a smaller subset of the listed parameters and return values*. For instance,
 a ``def my_simulation(one_Input) -> one_Output`` function is now accepted,
 as is ``def my_generator(Input, persis_info) -> Output, persis_info``.

docs/history_output_logging.rst

@@ -12,7 +12,7 @@ by libEnsemble.  If libEnsemble aborts on an exception, these structures are
 dumped automatically to these files:
 
 * ``libE_history_at_abort_<sim_count>.npy``
-* ``libE_history_at_abort_<sim_count>.pickle``
+* ``libE_persis_info_at_abort_<sim_count>.pickle``
 
 To suppress libEnsemble from producing these two files, set ``libE_specs["save_H_and_persis_on_abort"] = False``.
 

docs/introduction_latex.rst

@@ -29,7 +29,6 @@
 .. _NLopt documentation: https://nlopt.readthedocs.io/en/latest/NLopt_Installation/
 .. _nlopt: https://nlopt.readthedocs.io/en/latest/
 .. _NumPy: http://www.numpy.org
-.. _Quickstart: https://libensemble.readthedocs.io/en/main/introduction.html
 .. _OPAL: http://amas.web.psi.ch/docs/opal/opal_user_guide-1.6.0.pdf
 .. _petsc4py: https://bitbucket.org/petsc/petsc4py
 .. _PETSc/TAO: http://www.mcs.anl.gov/petsc
@@ -44,6 +43,7 @@
 .. _pytest: https://pypi.org/project/pytest/
 .. _Python: http://www.python.org
 .. _pyyaml: https://pyyaml.org/
+.. _Quickstart: https://libensemble.readthedocs.io/en/main/introduction.html
 .. _ReadtheDocs: http://libensemble.readthedocs.org/
 .. _SciPy: http://www.scipy.org
 .. _scipy.optimize: https://docs.scipy.org/doc/scipy/reference/optimize.html

docs/known_issues.rst

@@ -4,9 +4,15 @@ Known Issues
 The following selection describes known bugs, errors, or other difficulties that
 may occur when using libEnsemble.
 
-* As of 10/13/2022, on Perlmutter there was an issue running concurrent applications
-  on a node, following a recent system update. This also affects previous versions
-  of libEnsemble, and is being investigated.
+* Platforms using SLURM version 23.02 experience a `pickle error`_ when using
+  ``mpi4py`` comms. Disabling matching probes via the environment variable
+  ``export MPI4PY_RC_RECV_MPROBE=0`` or adding ``mpi4py.rc.recv_mprobe = False``
+  at the top of the calling script should resolve this error. If using the MPI
+  executor and multiple workers per node, some users may experience failed
+  applications with the message
+  ``srun: error: CPU binding outside of job step allocation, allocated`` in
+  the application's standard error. This is being investigated. If this happens
+  we recommend using ``local`` comms in place of ``mpi4py``.
 * When using the Executor: OpenMPI does not work with direct MPI task
   submissions in mpi4py comms mode, since OpenMPI does not support nested MPI
   executions. Use either ``local`` mode or the Balsam Executor instead.
@@ -23,3 +29,5 @@ may occur when using libEnsemble.
   :doc:`FAQ<FAQ>` for more information.
 * We currently recommended running in Central mode on Bridges as distributed
   runs are experiencing hangs.
+
+.. _pickle error: https://docs.nersc.gov/development/languages/python/using-python-perlmutter/#missing-support-for-matched-proberecv

docs/platforms/polaris.rst

@@ -42,12 +42,15 @@ for installing libEnsemble, including using Spack.
 Ensuring use of mpiexec
 -----------------------
 
-If using the :doc:`MPIExecutor<../executor/mpi_executor>` it is recommended to
-ensure you are using ``mpiexec`` instead of ``aprun``. When setting up the executor use::
+Prior to libE v 0.10.0, when using the :doc:`MPIExecutor<../executor/mpi_executor>` it
+is necessary to manually tell libEnsemble to use``mpiexec`` instead of ``aprun``.
+When setting up the executor use::
 
     from libensemble.executors.mpi_executor import MPIExecutor
     exctr = MPIExecutor(custom_info={'mpi_runner':'mpich', 'runner_name':'mpiexec'})
 
+From version 0.10.0, this is not necessary.
+
 Job Submission
 --------------
 

docs/platforms/srun.rst

@@ -65,7 +65,7 @@ Note on Resource Binding
 ------------------------
 
 .. note::
-    Update: From version version 0.10.0, it is recommended that GPUs are assigned
+    Update: From version 0.10.0, it is recommended that GPUs are assigned
     automatically by libEnsemble. See the :doc:`forces_gpu<../tutorials/forces_gpu_tutorial>`
     tutorial as an example.
 

docs/tutorials/executor_forces_tutorial.rst

@@ -2,31 +2,23 @@
 Executor with Electrostatic Forces
 ==================================
 
-This tutorial highlights libEnsemble's capability to execute
+This tutorial highlights libEnsemble's capability to portably execute
 and monitor external scripts or user applications within simulation or generator
-functions using the :doc:`executor<../executor/overview>`. In this tutorial,
-our calling script registers a compiled executable that simulates
+functions using the :doc:`executor<../executor/overview>`.
+
+This tutorial's calling script registers a compiled executable that simulates
 electrostatic forces between a collection of particles. The simulator function
 launches instances of this executable and reads output files to determine
 if the run was successful.
 
-It is possible to use ``subprocess`` calls from Python to issue
-commands such as ``jsrun`` or ``aprun`` to run applications. Unfortunately,
-hard-coding such commands within user scripts isn't portable.
-Furthermore, many systems like Argonne's :doc:`Theta<../platforms/theta>` do not
-allow libEnsemble to submit additional tasks from the compute nodes. On these
-systems, a proxy launch mechanism (such as Balsam) is required.
-libEnsemble's Executors were developed to directly address such issues.
-
-In particular, we'll be experimenting with
-libEnsemble's :doc:`MPI Executor<../executor/mpi_executor>`, since it can automatically
-detect available MPI runners and resources, and by default divide them equally among workers.
+This tutorial uses libEnsemble's :doc:`MPI Executor<../executor/mpi_executor>`,
+which automatically detects available MPI runners and resources.
 
 Getting Started
 ---------------
 
 The simulation source code ``forces.c`` can be obtained directly from the
-libEnsemble repository here_.
+libEnsemble repository in the forces_app_ directory.
 
 Assuming MPI and its C compiler ``mpicc`` are available, compile
 ``forces.c`` into an executable (``forces.x``) with:
@@ -35,9 +27,14 @@ Assuming MPI and its C compiler ``mpicc`` are available, compile
 
     $ mpicc -O3 -o forces.x forces.c -lm
 
+Alternative build lines for different platforms can be found in the ``build_forces.sh``
+file in the same directory.
+
 Calling Script
 --------------
 
+Complete scripts for this example can be found in the forces_simple_ directory.
+
 Let's begin by writing our calling script to parameterize our simulation and
 generation functions and call libEnsemble. Create a Python file called `run_libe_forces.py`
 containing:
@@ -66,13 +63,11 @@ containing:
     sim_app = os.path.join(os.getcwd(), "../forces_app/forces.x")
     exctr.register_app(full_path=sim_app, app_name="forces")
 
-On line 15, we instantiate our :doc:`MPI Executor<../executor/mpi_executor>` class instance,
-which can optionally be customized by specifying alternative MPI runners. The
-auto-detected default should be sufficient.
+On line 15, we instantiate our :doc:`MPI Executor<../executor/mpi_executor>`.
 
 Registering an application is as easy as providing the full file-path and giving
-it a memorable name. This Executor instance will later be retrieved within our
-simulation function to launch the registered app.
+it a memorable name. This Executor will later be used within our simulation
+function to launch the registered app.
 
 Next define the :ref:`sim_specs<datastruct-sim-specs>` and
 :ref:`gen_specs<datastruct-gen-specs>` data structures. Recall that these
@@ -225,12 +220,10 @@ for starters:
 
 We retrieve the generated number of particles from ``H`` and construct
 an argument string for our launched application. The particle count doubles up
-as a random number seed here. Note a fourth argument can be added to forces
-that gives forces a chance of a "bad run" (a float between 0 and 1), but
-for now that will default to zero.
+as a random number seed here.
 
-We then retrieve our previously instantiated Executor instance from the
-class definition, where it was automatically stored as an attribute.
+We then retrieve our previously instantiated Executor from the class definition,
+where it was automatically stored as an attribute.
 
 After submitting the "forces" app for execution,
 a :ref:`Task<task_tag>` object is returned that correlates with the launched app.
@@ -283,10 +276,10 @@ This completes our calling script and simulation function. Run libEnsemble with:
 
     $ python run_libe_forces.py --comms local --nworkers [nworkers]
 
-This may take up to a minute to complete. Output files---including ``forces.stat``
-and files containing ``stdout`` and ``stderr`` content for each task---should
-appear in the current working directory. Overall workflow information
-should appear in ``libE_stats.txt`` and ``ensemble.log`` as usual.
+Output files---including ``forces.stat`` and files containing ``stdout`` and
+``stderr`` content for each task---should appear in the current working
+directory. Overall workflow information should appear in ``libE_stats.txt``
+and ``ensemble.log`` as usual.
 
 For example, my ``libE_stats.txt`` resembled::
 
@@ -338,6 +331,8 @@ Each of these example files can be found in the repository in `examples/tutorial
 For further experimentation, we recommend trying out this libEnsemble tutorial
 workflow on a cluster or multi-node system, since libEnsemble can also manage
 those resources and is developed to coordinate computations at huge scales.
+See ref:`HPC platform guides<platform-index>` for more information.
+
 Please feel free to contact us or open an issue on GitHub_ if this tutorial
 workflow doesn't work properly on your cluster or other compute resource.
 
@@ -386,6 +381,7 @@ These may require additional browsing of the documentation to complete.
 
         ...
 
-.. _here: https://raw.githubusercontent.com/Libensemble/libensemble/main/libensemble/tests/scaling_tests/forces/forces.c
+.. _forces_app: https://github.com/Libensemble/libensemble/tree/main/libensemble/tests/scaling_tests/forces/forces_app
+.. _forces_simple: https://github.com/Libensemble/libensemble/tree/main/libensemble/tests/scaling_tests/forces/forces_simple
 .. _examples/tutorials/forces_with_executor: https://github.com/Libensemble/libensemble/tree/develop/examples/tutorials/forces_with_executor
 .. _GitHub: https://github.com/Libensemble/libensemble/issues