Documentation: Explain Interface Class Hierarchy

README.md

@@ -156,6 +156,7 @@ as hierarchical job scheduler within the allocations.
   * [Pull Requests](https://executorlib.readthedocs.io/en/latest/4-developer.html#pull-requests)
   * [License](https://executorlib.readthedocs.io/en/latest/4-developer.html#license)
   * [Modules](https://executorlib.readthedocs.io/en/latest/4-developer.html#modules)
+  * [Interface Class Hierarchy](https://executorlib.readthedocs.io/en/latest/4-developer.html#interface-class-hierarchy)
   * [Communication](https://executorlib.readthedocs.io/en/latest/4-developer.html#communication)
   * [External Executables](https://executorlib.readthedocs.io/en/latest/4-developer.html#external-executables)
 * [Interface](https://executorlib.readthedocs.io/en/latest/api.html)

notebooks/4-developer.ipynb

@@ -7,7 +7,7 @@
   },
   "language_info": {
    "name": "python",
-   "version": "3.12.9",
+   "version": "3.12.10",
    "mimetype": "text/x-python",
    "codemirror_mode": {
     "name": "ipython",
@@ -77,7 +77,26 @@
     "* `interfaces` - the different `Executor` classes are defined here, namely `SingleNodeExecutor`, `SlurmClusterExecutor`, `SlurmJobExecutor`, `FluxClusterExecutor` and `FluxJobExecutor`.\n",
     "* `standalone` - the standalone module contains a number of utility functions which only depend on external libraries and do not have any internal dependency to other parts of `executorlib`. This includes the functionality to generate executable commands, the [h5py](https://www.h5py.org) based interface for caching, a number of input checks, routines to plot the dependencies of a number of future objects, functionality to interact with the [queues defined in the Python standard library](https://docs.python.org/3/library/queue.html), the interface for serialization based on [cloudpickle](https://github.com/cloudpipe/cloudpickle) and finally an extension to the [threading](https://docs.python.org/3/library/threading.html) of the Python standard library.\n",
     "\n",
-    "Given the level of separation the integration of submodules from the standalone module in external software packages should be the easiest way to benefit from the developments in executorlib beyond just using the `Executor` class. "
+    "Given the level of separation the integration of submodules from the standalone module in external software packages should be the easiest way to benefit from the developments in executorlib beyond just using the `Executor` class. \n",
+    "\n",
+    "## Interface Class Hierarchy\n",
+    "executorlib provides five different interfaces, namely `SingleNodeExecutor`, `SlurmClusterExecutor`, `SlurmJobExecutor`, `FluxClusterExecutor` and `FluxJobExecutor`, internally these are mapped to four types of `Executor` classes, namely `BlockAllocationExecutor`, `DependencyExecutor`, `FileExecutor` and `OneTaskPerProcessExecutor` depending on which options are selected. The dependence is illustrated in the following table:\n",
+    "\n",
+    "|                                                                         | `BlockAllocationExecutor` | `DependencyExecutor`     | `FileExecutor` | `OneTaskPerProcessExecutor` |\n",
+    "|-------------------------------------------------------------------------|---------------------------|--------------------------|----------------|-----------------------------|\n",
+    "| `SingleNodeExecutor(disable_dependencies=False)`                        |                           | with `MpiExecSpawner`    |                |                             |\n",
+    "| `SingleNodeExecutor(disable_dependencies=True, block_allocation=False)` |                           |                          |                | with `MpiExecSpawner`       |\n",
+    "| `SingleNodeExecutor(disable_dependencies=True, block_allocation=True)`  | with `MpiExecSpawner`     |                          |                |                             |\n",
+    "| `SlurmClusterExecutor(plot_dependency_graph=False)`                     |                           |                          | with `pysqa`   |                             |\n",
+    "| `SlurmClusterExecutor(plot_dependency_graph=True)`                      |                           | with `SrunSpawner`       |                |                             |\n",
+    "| `SlurmJobExecutor(disable_dependencies=False)`                          |                           | with `SrunSpawner`       |                |                             |\n",
+    "| `SlurmJobExecutor(disable_dependencies=True, block_allocation=False)`   |                           |                          |                | with `SrunSpawner`          |\n",
+    "| `SlurmJobExecutor(disable_dependencies=True, block_allocation=True)`    | with `SrunSpawner`        |                          |                |                             |\n",
+    "| `FluxClusterExecutor(plot_dependency_graph=False)`                      |                           |                          | with `pysqa`   |                             |\n",
+    "| `FluxClusterExecutor(plot_dependency_graph=True)`                       |                           | with `FluxPythonSpawner` |                |                             |\n",
+    "| `FluxJobExecutor(disable_dependencies=False)`                           |                           | with `FluxPythonSpawner` |                |                             |\n",
+    "| `FluxJobExecutor(disable_dependencies=True, block_allocation=False)`    |                           |                          |                | with `FluxPythonSpawner`    |\n",
+    "| `FluxJobExecutor(disable_dependencies=True, block_allocation=True)`     | with `FluxPythonSpawner`  |                          |                |                             |"
    ],
    "metadata": {}
   },
@@ -110,7 +129,7 @@
    "cell_type": "code",
    "source": "from executorlib import SingleNodeExecutor",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [],
    "execution_count": 1
@@ -120,7 +139,7 @@
    "cell_type": "code",
    "source": "def execute_shell_command(\n    command: list, universal_newlines: bool = True, shell: bool = False\n):\n    import subprocess\n\n    return subprocess.check_output(\n        command, universal_newlines=universal_newlines, shell=shell\n    )",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [],
    "execution_count": 2
@@ -130,7 +149,7 @@
    "cell_type": "code",
    "source": "with SingleNodeExecutor() as exe:\n    future = exe.submit(\n        execute_shell_command,\n        [\"echo\", \"test\"],\n        universal_newlines=True,\n        shell=False,\n    )\n    print(future.result())",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [
     {
@@ -152,7 +171,7 @@
    "cell_type": "code",
    "source": "count_script = \"\"\"\\\ndef count(iterations):\n    for i in range(int(iterations)):\n        print(i)\n    print(\"done\")\n\n\nif __name__ == \"__main__\":\n    while True:\n        user_input = input()\n        if \"shutdown\" in user_input:\n            break\n        else:\n            count(iterations=int(user_input))\n\"\"\"\n\nwith open(\"count.py\", \"w\") as f:\n    f.writelines(count_script)",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [],
    "execution_count": 4
@@ -168,7 +187,7 @@
    "cell_type": "code",
    "source": "def init_process():\n    import subprocess\n\n    return {\n        \"process\": subprocess.Popen(\n            [\"python\", \"count.py\"],\n            stdin=subprocess.PIPE,\n            stdout=subprocess.PIPE,\n            universal_newlines=True,\n            shell=False,\n        )\n    }",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [],
    "execution_count": 5
@@ -184,7 +203,7 @@
    "cell_type": "code",
    "source": "def interact(shell_input, process, lines_to_read=None, stop_read_pattern=None):\n    process.stdin.write(shell_input)\n    process.stdin.flush()\n    lines_count = 0\n    output = \"\"\n    while True:\n        output_current = process.stdout.readline()\n        output += output_current\n        lines_count += 1\n        if stop_read_pattern is not None and stop_read_pattern in output_current:\n            break\n        elif lines_to_read is not None and lines_to_read == lines_count:\n            break\n    return output",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [],
    "execution_count": 6
@@ -200,7 +219,7 @@
    "cell_type": "code",
    "source": "def shutdown(process):\n    process.stdin.write(\"shutdown\\n\")\n    process.stdin.flush()",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [],
    "execution_count": 7
@@ -216,7 +235,7 @@
    "cell_type": "code",
    "source": "with SingleNodeExecutor(\n    max_workers=1,\n    init_function=init_process,\n    block_allocation=True,\n) as exe:\n    future = exe.submit(\n        interact, shell_input=\"4\\n\", lines_to_read=5, stop_read_pattern=None\n    )\n    print(future.result())\n    future_shutdown = exe.submit(shutdown)\n    print(future_shutdown.result())",
    "metadata": {
-    "trusted": true
+    "trusted": false
    },
    "outputs": [
     {