Libensemble · jlnav · Apr 22, 2022 · Dec 3, 2021 · Dec 7, 2021 · Dec 14, 2021
diff --git a/README.rst b/README.rst
@@ -82,10 +82,9 @@ Optional dependencies:
 
 * Balsam_
 
-If running on the the compute nodes of three-tier systems
-like OLCF's Summit_ or ALCF's Theta_, libEnsemble's workers may use the Balsam service
-to schedule and launch MPI applications. Otherwise, libEnsemble can be run with
-multiprocessing on the intermediate launch nodes.
+As of v0.8.0+dev, libEnsemble features an updated `Balsam Executor`_
+for workers to schedule and launch applications to *anywhere* with a running
+Balsam site, including to remote machines.
 
 * pyyaml_
 
@@ -296,7 +295,8 @@ See a complete list of `example user scripts`_.
 .. _across: https://libensemble.readthedocs.io/en/develop/platforms/platforms_index.html#funcx-remote-user-functions
 .. _APOSMM: https://link.springer.com/article/10.1007/s12532-017-0131-4
 .. _AWA: https://link.springer.com/article/10.1007/s12532-017-0131-4
-.. _Balsam: https://www.alcf.anl.gov/support-center/theta/balsam
+.. _Balsam: https://balsam.readthedocs.io/en/latest/
+.. _Balsam Executor: https://libensemble.readthedocs.io/en/develop/executor/balsam_2_executor.html
 .. _Community Examples repository: https://github.com/Libensemble/libe-community-examples
 .. _Conda: https://docs.conda.io/en/latest/
 .. _conda-forge: https://conda-forge.org/

diff --git a/docs/executor/balsam_2_executor.rst b/docs/executor/balsam_2_executor.rst
@@ -0,0 +1,14 @@
+Balsam Executor - Remote apps
+=============================
+
+.. automodule:: balsam_executor
+  :no-undoc-members:
+
+.. autoclass:: BalsamExecutor
+  :show-inheritance:
+  :members: __init__, register_app, submit_allocation, revoke_allocation, submit
+
+.. autoclass:: BalsamTask
+  :show-inheritance:
+  :member-order: bysource
+  :members: poll, wait, kill
diff --git a/docs/executor/balsam_executor.rst b/docs/executor/balsam_executor.rst
diff --git a/docs/executor/ex_index.rst b/docs/executor/ex_index.rst
@@ -1,16 +1,18 @@
 .. _executor_index:
 
-Executor
-========
+Executors
+=========
 
-libEnsemble's Executor can be used within the simulator (and, potentially, the generator)
-functions to provide a simple, portable interface for running and managing user
-applications.
+libEnsemble's Executors can be used within user functions to provide a simple,
+portable interface for running and managing user applications.
 
 .. toctree::
    :maxdepth: 2
    :titlesonly:
-   :caption: libEnsemble Executor:
+   :caption: libEnsemble Executors:
 
    overview
    executor
+   mpi_executor
+   legacy_balsam_executor
+   balsam_2_executor
diff --git a/docs/executor/executor.rst b/docs/executor/executor.rst
@@ -1,5 +1,5 @@
-Executor Modules
-================
+Base Executor - Local apps
+==========================
 
 .. automodule:: executor
   :no-undoc-members:
@@ -13,7 +13,8 @@ See the Executor APIs for optional arguments.
    :caption: Alternative Executors:
 
    mpi_executor
-   balsam_executor
+   legacy_balsam_executor
+   balsam_2_executor
 
 Executor Class
 ---------------

diff --git a/docs/executor/legacy_balsam_executor.rst b/docs/executor/legacy_balsam_executor.rst
@@ -0,0 +1,17 @@
+Legacy Balsam MPI Executor
+==========================
+
+.. automodule:: legacy_balsam_executor
+  :no-undoc-members:
+
+.. autoclass:: LegacyBalsamMPIExecutor
+  :show-inheritance:
+  :inherited-members:
+  :member-order: bysource
+  :members: __init__, submit, poll, manager_poll, kill, set_kill_mode
+
+.. autoclass:: LegacyBalsamTask
+  :show-inheritance:
+  :member-order: bysource
+  :members: workdir_exists, file_exists_in_workdir, read_file_in_workdir, stdout_exists, read_stdout
+  :inherited-members:
diff --git a/docs/executor/mpi_executor.rst b/docs/executor/mpi_executor.rst
@@ -1,5 +1,5 @@
-MPI Executor
-============
+MPI Executor - MPI apps
+=======================
 
 .. automodule:: mpi_executor
   :no-undoc-members:

diff --git a/docs/executor/overview.rst b/docs/executor/overview.rst
@@ -23,9 +23,9 @@ to an application instance instead of a callable. They feature the ``cancel()``,
 from the standard.
 
 The main ``Executor`` class is an abstract class, inherited by the ``MPIExecutor``
-for direct running of MPI applications, and the ``BalsamMPIExecutor``
-for submitting MPI run requests from a worker running on a compute node to a
-Balsam service running on a launch node. This second approach is suitable for
+for direct running of MPI applications, and the ``BalsamExecutor``
+for submitting MPI run requests from a worker running on a compute node to the
+Balsam service. This second approach is suitable for
 systems that don't allow submitting MPI applications from compute nodes.
 
 Typically, users choose and parameterize their ``Executor`` objects in their
@@ -46,8 +46,8 @@ In calling script::
     USE_BALSAM = False
 
     if USE_BALSAM:
-        from libensemble.executors.balsam_executor import BalsamMPIExecutor
-        exctr = BalsamMPIExecutor()
+        from libensemble.executors.balsam_executor import LegacyBalsamMPIExecutor
+        exctr = LegacyBalsamMPIExecutor()
     else:
         from libensemble.executors.mpi_executor import MPIExecutor
         exctr = MPIExecutor()

diff --git a/docs/introduction_latex.rst b/docs/introduction_latex.rst
@@ -25,7 +25,8 @@ We now present further information on running and testing libEnsemble.
 .. _across: https://libensemble.readthedocs.io/en/develop/platforms/platforms_index.html#funcx-remote-user-functions
 .. _APOSMM: https://link.springer.com/article/10.1007/s12532-017-0131-4
 .. _AWA: https://link.springer.com/article/10.1007/s12532-017-0131-4
-.. _Balsam: https://www.alcf.anl.gov/support-center/theta/balsam
+.. _Balsam: https://balsam.readthedocs.io/en/latest/
+.. _Balsam Executor: https://libensemble.readthedocs.io/en/develop/executor/balsam_2_executor.html
 .. _Community Examples repository: https://github.com/Libensemble/libe-community-examples
 .. _Conda: https://docs.conda.io/en/latest/
 .. _conda-forge: https://conda-forge.org/

diff --git a/docs/overview_usecases.rst b/docs/overview_usecases.rst
@@ -111,7 +111,7 @@ its capabilities.
 
 * **Executor**: The executor can be used within user functions to provide a
   simple, portable interface for running and managing user tasks (applications).
-  There are multiple executors including the ``MPIExecutor`` and ``BalsamMPIExecutor``.
+  There are multiple executors including the ``MPIExecutor`` and ``LegacyBalsamMPIExecutor``.
   The base ``Executor`` class allows local sub-processing of serial tasks.
 
 * **Submit**: Enqueue or indicate that one or more jobs or tasks needs to be

diff --git a/docs/platforms/platforms_index.rst b/docs/platforms/platforms_index.rst
@@ -84,26 +84,46 @@ Systems with Launch/MOM nodes
 Some large systems have a 3-tier node setup. That is, they have a separate set of launch nodes
 (known as MOM nodes on Cray Systems). User batch jobs or interactive sessions run on a launch node.
 Most such systems supply a special MPI runner which has some application-level scheduling
-capability (eg. aprun, jsrun). MPI applications can only be submitted from these nodes. Examples
+capability (eg. ``aprun``, ``jsrun``). MPI applications can only be submitted from these nodes. Examples
 of these systems include: Summit, Sierra and Theta.
 
 There are two ways of running libEnsemble on these kind of systems. The first, and simplest,
-is to run libEnsemble on the launch nodes. This is often sufficient if the worker's sim or
-gen scripts are not doing too much work (other than launching applications). This approach
+is to run libEnsemble on the launch nodes. This is often sufficient if the worker's simulation
+or generation functions are not doing much work (other than launching applications). This approach
 is inherently centralized. The entire node allocation is available for the worker-launched
 tasks.
 
-To run libEnsemble on the compute nodes of these systems requires an alternative Executor,
-such as :doc:`Balsam<../executor/balsam_executor>`, which runs on the
-launch nodes and launches tasks submitted by workers. Running libEnsemble on the compute
-nodes is potentially more scalable and will better manage ``sim_f`` and ``gen_f`` functions
-that contain considerable computational work or I/O.
+However, running libEnsemble on the compute nodes is potentially more scalable and
+will better manage simulation and generation functions that contain considerable
+computational work or I/O. Therefore the second option is to use proxy task-execution
+services like Balsam_.
 
-    .. image:: ../images/centralized_new_detailed_balsam.png
+Balsam - Externally managed applications
+----------------------------------------
+
+Running libEnsemble on the compute nodes while still submitting additional applications
+requires alternative Executors that connect to external services like Balsam_. Balsam
+can take tasks submitted by workers and execute them on the remaining compute nodes,
+or if using Balsam 2, *to entirely different systems*.
+
+    .. figure:: ../images/centralized_new_detailed_balsam.png
         :alt: central_balsam
         :scale: 30
         :align: center
 
+        Single-System: libEnsemble + LegacyBalsamMPIExecutor
+
+    .. figure:: ../images/balsam2.png
+        :alt: balsam2
+        :scale: 40
+        :align: center
+
+        (New) Multi-System: libEnsemble + BalsamExecutor
+
+As of v0.8.0+dev, libEnsemble supports both "legacy" Balsam via the
+:doc:`LegacyBalsamMPIExecutor<../executor/legacy_balsam_executor>`
+and Balsam 2 via the :doc:`BalsamExecutor<../executor/balsam_2_executor>`.
+
 Submission scripts for running on launch/MOM nodes and for using Balsam, can be be found in
 the :doc:`examples<example_scripts>`.
 
@@ -178,7 +198,7 @@ key. For example::
         'sim_f': sim_f,
         'in': ['x'],
         'out': [('f', float)],
-        'funcx_endpoint': 3af6dc24-3f27-4c49-8d11-e301ade15353,
+        'funcx_endpoint': '3af6dc24-3f27-4c49-8d11-e301ade15353',
     }
 
 See the ``libensemble/tests/scaling_tests/funcx_forces`` directory for a complete

diff --git a/docs/running_libE.rst b/docs/running_libE.rst
@@ -41,7 +41,7 @@ Limitations of MPI mode
 
 If you are launching MPI applications from workers, then MPI is being nested. This is not
 supported with Open MPI. This can be overcome by using a proxy launcher
-(see :doc:`Balsam<executor/balsam_executor>`). This nesting does work, however,
+(see :doc:`Balsam<executor/balsam_2_executor>`). This nesting does work, however,
 with MPICH and its derivative MPI implementations.
 
 It is also unsuitable to use this mode when running on the **launch** nodes of three-tier

diff --git a/docs/tutorials/executor_forces_tutorial.rst b/docs/tutorials/executor_forces_tutorial.rst
@@ -241,7 +241,7 @@ and evaluated in a variety of helpful ways. For now, we're satisfied with waitin
 for the task to complete via ``task.wait()``.
 
 We can assume that afterward, any results are now available to parse. Our application
-produces a ``forces[particles].stat`` file that contains either energy
+produces a ``forces.stat`` file that contains either energy
 computations for every time-step or a "kill" message if particles were lost, which
 indicates a failed simulation.
 
@@ -254,7 +254,7 @@ to ``WORKER_DONE``. Otherwise, send back ``NAN`` and a ``TASK_FAILED`` status:
     :linenos:
 
         # Stat file to check for bad runs
-        statfile = "forces{}.stat".format(particles)
+        statfile = "forces.stat"
 
         # Try loading final energy reading, set the sim's status
         try:

diff --git a/examples/calling_scripts/run_libe_forces.py b/examples/calling_scripts/run_libe_forces.py
@@ -1 +1 @@
-../../libensemble/tests/scaling_tests/forces/run_libe_forces.py
+../../libensemble/tests/scaling_tests/forces/forces_adv/run_libe_forces.py
diff --git a/examples/calling_scripts/run_libe_forces_from_yaml.py b/examples/calling_scripts/run_libe_forces_from_yaml.py
@@ -1 +1 @@
-../../libensemble/tests/scaling_tests/forces/run_libe_forces_from_yaml.py
+../../libensemble/tests/scaling_tests/forces/forces_adv/run_libe_forces_from_yaml.py
diff --git a/examples/tutorials/forces_with_executor/build_forces.sh b/examples/tutorials/forces_with_executor/build_forces.sh
@@ -1 +1 @@
-../../../libensemble/tests/scaling_tests/forces/build_forces.sh
+../../../libensemble/tests/scaling_tests/forces/forces_app/build_forces.sh
diff --git a/examples/tutorials/forces_with_executor/cleanup.sh b/examples/tutorials/forces_with_executor/cleanup.sh
diff --git a/examples/tutorials/forces_with_executor/forces.c b/examples/tutorials/forces_with_executor/forces.c
@@ -1 +1 @@
-../../../libensemble/tests/scaling_tests/forces/forces.c
+../../../libensemble/tests/scaling_tests/forces/forces_app/forces.c
diff --git a/examples/tutorials/forces_with_executor/forces_simf_simple.py b/examples/tutorials/forces_with_executor/forces_simf_simple.py
@@ -0,0 +1 @@
+../../../libensemble/tests/scaling_tests/forces/forces_simple/forces_simf_simple.py
diff --git a/examples/tutorials/forces_with_executor/run_libe_forces_simple.py b/examples/tutorials/forces_with_executor/run_libe_forces_simple.py
@@ -0,0 +1 @@
+../../../libensemble/tests/scaling_tests/forces/forces_simple/run_libe_forces_simple.py
diff --git a/libensemble/executors/__init__.py b/libensemble/executors/__init__.py
@@ -1,9 +1,18 @@
 from libensemble.executors.executor import Executor
 from libensemble.executors.mpi_executor import MPIExecutor
 
-import os
-import sys
-if 'BALSAM_DB_PATH' in os.environ and int(sys.version[2]) >= 6:
-    from libensemble.executors.balsam_executor import BalsamMPIExecutor
+import pkg_resources
 
-__all__ = ['BalsamMPIExecutor', 'Executor', 'MPIExecutor']
+try:
+    if pkg_resources.get_distribution("balsam"):  # Balsam 0.7.0 onward (Balsam 2)
+        from libensemble.executors.balsam_executor import BalsamExecutor
+
+except (ModuleNotFoundError, ImportError, pkg_resources.DistributionNotFound):
+    try:
+        if pkg_resources.get_distribution("balsam-flow"):  # Balsam up through 0.5.0
+            from libensemble.executors.legacy_balsam_executor import LegacyBalsamMPIExecutor
+    except (ModuleNotFoundError, ImportError, pkg_resources.DistributionNotFound):
+        pass
+
+
+__all__ = ["LegacyBalsamMPIExecutor", "Executor", "MPIExecutor", "BalsamExecutor"]
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		../../libensemble/tests/scaling_tests/forces/run_libe_forces.py
		../../libensemble/tests/scaling_tests/forces/forces_adv/run_libe_forces.py
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		../../../libensemble/tests/scaling_tests/forces/build_forces.sh
		../../../libensemble/tests/scaling_tests/forces/forces_app/build_forces.sh
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		../../../libensemble/tests/scaling_tests/forces/forces_simple/forces_simf_simple.py