From acb6b4fb967b7b9a193cc721ba7fb8279ad32b61 Mon Sep 17 00:00:00 2001 From: Jan Janssen Date: Tue, 25 Mar 2025 09:15:07 +0100 Subject: [PATCH] Add files via upload --- notebooks/4-developer.ipynb | 350 +----------------------------------- 1 file changed, 1 insertion(+), 349 deletions(-) diff --git a/notebooks/4-developer.ipynb b/notebooks/4-developer.ipynb index faa64224..f2a38d0f 100644 --- a/notebooks/4-developer.ipynb +++ b/notebooks/4-developer.ipynb @@ -1,349 +1 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "511b34e0-12af-4437-8915-79f033fe7cda", - "metadata": {}, - "source": [ - "# Developer\n", - "executorlib is designed to work out of the box for up-scaling Python functions and distribute them on a high performance computing (HPC) cluster. Most users should only import the `Executor` class from executorlib and should not need to use any of the internal functionality covered in this section. Still for more advanced applications beyond the submission of Python functions executorlib provides additional functionality. The functionality in this section is not officially supported and might change in future versions without further notice. " - ] - }, - { - "cell_type": "markdown", - "id": "7bc073aa-6036-48e7-9696-37af050d438a", - "metadata": {}, - "source": [ - "## Communication\n", - "The key functionality of the `executorlib` package is the up-scaling of python functions with thread based parallelism, \n", - "MPI based parallelism or by assigning GPUs to individual python functions. In the background this is realized using a \n", - "combination of the [zero message queue](https://zeromq.org) and [cloudpickle](https://github.com/cloudpipe/cloudpickle) \n", - "to communicate binary python objects. The `executorlib.standalone.interactive.communication.SocketInterface` is an abstraction of this \n", - "interface, which is used in the other classes inside `executorlib` and might also be helpful for other projects. It \n", - "comes with a series of utility functions:\n", - "\n", - "* `executorlib.standalone.interactive.communication.interface_bootup()`: To initialize the interface\n", - "* `executorlib.standalone.interactive.communication.interface_connect()`: To connect the interface to another instance\n", - "* `executorlib.standalone.interactive.communication.interface_send()`: To send messages via this interface \n", - "* `executorlib.standalone.interactive.communication.interface_receive()`: To receive messages via this interface \n", - "* `executorlib.standalone.interactive.communication.interface_shutdown()`: To shutdown the interface\n", - "\n", - "While `executorlib` was initially designed for up-scaling python functions for HPC, the same functionality can be \n", - "leveraged to up-scale any executable independent of the programming language it is developed in." - ] - }, - { - "cell_type": "markdown", - "id": "8754df33-fa95-4ca6-ae02-6669967cf4e7", - "metadata": {}, - "source": [ - "## External Executables\n", - "On extension beyond the submission of Python functions is the communication with an external executable. This could be any kind of program written in any programming language which does not provide Python bindings so it cannot be represented in Python functions. " - ] - }, - { - "cell_type": "markdown", - "id": "75af1f8a-7ad7-441f-80a2-5c337484097f", - "metadata": {}, - "source": [ - "### Subprocess\n", - "If the external executable is called only once, then the call to the external executable can be represented in a Python function with the [subprocess](https://docs.python.org/3/library/subprocess.html) module of the Python standard library. In the example below the shell command `echo test` is submitted to the `execute_shell_command()` function, which itself is submitted to the `Executor` class." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "83515b16-c4d5-4b02-acd7-9e1eb57fd335", - "metadata": {}, - "outputs": [], - "source": "from executorlib import SingleNodeExecutor" - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "f1ecee94-24a6-4bf9-8a3d-d50eba994367", - "metadata": {}, - "outputs": [], - "source": [ - "def execute_shell_command(\n", - " command: list, universal_newlines: bool = True, shell: bool = False\n", - "):\n", - " import subprocess\n", - "\n", - " return subprocess.check_output(\n", - " command, universal_newlines=universal_newlines, shell=shell\n", - " )" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "32ef5b63-3245-4336-ac0e-b4a6673ee362", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "test\n", - "\n" - ] - } - ], - "source": [ - "with SingleNodeExecutor() as exe:\n", - " future = exe.submit(\n", - " execute_shell_command,\n", - " [\"echo\", \"test\"],\n", - " universal_newlines=True,\n", - " shell=False,\n", - " )\n", - " print(future.result())" - ] - }, - { - "cell_type": "markdown", - "id": "54837938-01e0-4dd3-b989-1133d3318929", - "metadata": {}, - "source": [ - "### Interactive\n", - "The more complex case is the interaction with an external executable during the run time of the executable. This can be implemented with executorlib using the block allocation `block_allocation=True` feature. The external executable is started as part of the initialization function `init_function` and then the indivdual functions submitted to the `Executor` class interact with the process which is connected to the external executable. \n", - "\n", - "Starting with the definition of the executable, in this example it is a simple script which just increases a counter. The script is written in the file `count.py` so it behaves like an external executable, which could also use any other progamming language. " - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "dedf138f-3003-4a91-9f92-03983ac7de08", - "metadata": {}, - "outputs": [], - "source": [ - "count_script = \"\"\"\\\n", - "def count(iterations):\n", - " for i in range(int(iterations)):\n", - " print(i)\n", - " print(\"done\")\n", - "\n", - "\n", - "if __name__ == \"__main__\":\n", - " while True:\n", - " user_input = input()\n", - " if \"shutdown\" in user_input:\n", - " break\n", - " else:\n", - " count(iterations=int(user_input))\n", - "\"\"\"\n", - "\n", - "with open(\"count.py\", \"w\") as f:\n", - " f.writelines(count_script)" - ] - }, - { - "cell_type": "markdown", - "id": "771b5b84-48f0-4989-a2c8-c8dcb4462781", - "metadata": {}, - "source": [ - "The connection to the external executable is established in the initialization function `init_function` of the `Executor` class. By using the [subprocess](https://docs.python.org/3/library/subprocess.html) module from the standard library two process pipes are created to communicate with the external executable. One process pipe is connected to the standard input `stdin` and the other is connected to the standard output `stdout`. " - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "8fe76668-0f18-40b7-9719-de47dacb0911", - "metadata": {}, - "outputs": [], - "source": [ - "def init_process():\n", - " import subprocess\n", - "\n", - " return {\n", - " \"process\": subprocess.Popen(\n", - " [\"python\", \"count.py\"],\n", - " stdin=subprocess.PIPE,\n", - " stdout=subprocess.PIPE,\n", - " universal_newlines=True,\n", - " shell=False,\n", - " )\n", - " }" - ] - }, - { - "cell_type": "markdown", - "id": "09dde7a1-2b43-4be7-ba36-38200b9fddf0", - "metadata": {}, - "source": [ - "The interaction function handles the data conversion from the Python datatypes to the strings which can be communicated to the external executable. It is important to always add a new line `\\n` to each command send via the standard input `stdin` to the external executable and afterwards flush the pipe by calling `flush()` on the standard input pipe `stdin`. " - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "7556f2bd-176f-4275-a87d-b5c940267888", - "metadata": {}, - "outputs": [], - "source": [ - "def interact(shell_input, process, lines_to_read=None, stop_read_pattern=None):\n", - " process.stdin.write(shell_input)\n", - " process.stdin.flush()\n", - " lines_count = 0\n", - " output = \"\"\n", - " while True:\n", - " output_current = process.stdout.readline()\n", - " output += output_current\n", - " lines_count += 1\n", - " if stop_read_pattern is not None and stop_read_pattern in output_current:\n", - " break\n", - " elif lines_to_read is not None and lines_to_read == lines_count:\n", - " break\n", - " return output" - ] - }, - { - "cell_type": "markdown", - "id": "5484b98b-546f-4f2c-8db1-919ce215e228", - "metadata": {}, - "source": [ - "Finally, to close the process after the external executable is no longer required it is recommended to define a shutdown function, which communicates to the external executable that it should shutdown. In the case of the `count.py` script defined above this is achieved by sending the keyword `shutdown`. " - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "d5344d2b-cb53-4d38-8cae-621e3b98bb56", - "metadata": {}, - "outputs": [], - "source": [ - "def shutdown(process):\n", - " process.stdin.write(\"shutdown\\n\")\n", - " process.stdin.flush()" - ] - }, - { - "cell_type": "markdown", - "id": "3899467c-dc54-41cb-b05e-b60f5cf97e46", - "metadata": {}, - "source": [ - "With these utility functions is to possible to communicate with any kind of external executable. Still for the specific implementation of the external executable it might be necessary to adjust the corresponding Python functions. Therefore this functionality is currently limited to developers and not considered a general feature of executorlib. " - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "747c1b78-4804-467b-9ac8-8144d8031da3", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0\n", - "1\n", - "2\n", - "3\n", - "done\n", - "\n", - "None\n" - ] - } - ], - "source": [ - "with SingleNodeExecutor(\n", - " max_workers=1,\n", - " init_function=init_process,\n", - " block_allocation=True,\n", - ") as exe:\n", - " future = exe.submit(\n", - " interact, shell_input=\"4\\n\", lines_to_read=5, stop_read_pattern=None\n", - " )\n", - " print(future.result())\n", - " future_shutdown = exe.submit(shutdown)\n", - " print(future_shutdown.result())" - ] - }, - { - "cell_type": "markdown", - "id": "96e56af9-3031-4d7b-9111-d2d031a0a6e4", - "metadata": {}, - "source": [ - "## License\n", - "```\n", - "BSD 3-Clause License\n", - "\n", - "Copyright (c) 2022, Jan Janssen\n", - "All rights reserved.\n", - "\n", - "Redistribution and use in source and binary forms, with or without\n", - "modification, are permitted provided that the following conditions are met:\n", - "\n", - "* Redistributions of source code must retain the above copyright notice, this\n", - " list of conditions and the following disclaimer.\n", - "\n", - "* Redistributions in binary form must reproduce the above copyright notice,\n", - " this list of conditions and the following disclaimer in the documentation\n", - " and/or other materials provided with the distribution.\n", - "\n", - "* Neither the name of the copyright holder nor the names of its\n", - " contributors may be used to endorse or promote products derived from\n", - " this software without specific prior written permission.\n", - "\n", - "THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n", - "AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n", - "IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\n", - "DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\n", - "FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n", - "DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n", - "SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\n", - "CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\n", - "OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n", - "OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "c2522d54-a00b-49ae-81e4-69e8fa05c9c3", - "metadata": {}, - "source": [ - "## Modules\n", - "While it is not recommended to link to specific internal components of executorlib in external Python packages but rather only the `Executor` class should be used as central interface to executorlib, the internal architecture is briefly outlined below. \n", - "* `backend` - the backend module contains the functionality for the Python processes created by executorlib to execute the submitted Python functions.\n", - "* `base` - the base module contains the definition of the executorlib `ExecutorBase` class which is internally used to create the different interfaces. To compare if an given `Executor` class is based on executorlib compare with the `ExecutorBase` class which can be imported as `from executorlib.base.executor import ExecutorBase`.\n", - "* `cache` - the cache module defines the file based communication for the [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html).\n", - "* `interactive` - the interactive modules defines the [zero message queue](https://zeromq.org) based communication for the [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html) and the [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html).\n", - "* `standalone` - the standalone module contains a number of utility functions which only depend on external libraries and do not have any internal dependency to other parts of `executorlib`. This includes the functionality to generate executable commands, the [h5py](https://www.h5py.org) based interface for caching, a number of input checks, routines to plot the dependencies of a number of future objects, functionality to interact with the [queues defined in the Python standard library](https://docs.python.org/3/library/queue.html), the interface for serialization based on [cloudpickle](https://github.com/cloudpipe/cloudpickle) and finally an extension to the [threading](https://docs.python.org/3/library/threading.html) of the Python standard library.\n", - "\n", - "Given the level of separation the integration of submodules from the standalone module in external software packages should be the easiest way to benefit from the developments in executorlib beyond just using the `Executor` class. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "39096340-f169-4438-b9c6-90c48ea37e4d", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.12.5" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} +{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.9","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":5,"nbformat":4,"cells":[{"id":"511b34e0-12af-4437-8915-79f033fe7cda","cell_type":"markdown","source":"# Developer\nexecutorlib is designed to work out of the box for up-scaling Python functions and distribute them on a high performance computing (HPC) cluster. Most users should only import the `Executor` class from executorlib and should not need to use any of the internal functionality covered in this section. Still for more advanced applications beyond the submission of Python functions executorlib provides additional functionality. The functionality in this section is not officially supported and might change in future versions without further notice. ","metadata":{}},{"id":"7bc073aa-6036-48e7-9696-37af050d438a","cell_type":"markdown","source":"## Communication\nThe key functionality of the `executorlib` package is the up-scaling of python functions with thread based parallelism, \nMPI based parallelism or by assigning GPUs to individual python functions. In the background this is realized using a \ncombination of the [zero message queue](https://zeromq.org) and [cloudpickle](https://github.com/cloudpipe/cloudpickle) \nto communicate binary python objects. The `executorlib.standalone.interactive.communication.SocketInterface` is an abstraction of this \ninterface, which is used in the other classes inside `executorlib` and might also be helpful for other projects. It \ncomes with a series of utility functions:\n\n* `executorlib.standalone.interactive.communication.interface_bootup()`: To initialize the interface\n* `executorlib.standalone.interactive.communication.interface_connect()`: To connect the interface to another instance\n* `executorlib.standalone.interactive.communication.interface_send()`: To send messages via this interface \n* `executorlib.standalone.interactive.communication.interface_receive()`: To receive messages via this interface \n* `executorlib.standalone.interactive.communication.interface_shutdown()`: To shutdown the interface\n\nWhile `executorlib` was initially designed for up-scaling python functions for HPC, the same functionality can be \nleveraged to up-scale any executable independent of the programming language it is developed in.","metadata":{}},{"id":"8754df33-fa95-4ca6-ae02-6669967cf4e7","cell_type":"markdown","source":"## External Executables\nOn extension beyond the submission of Python functions is the communication with an external executable. This could be any kind of program written in any programming language which does not provide Python bindings so it cannot be represented in Python functions. ","metadata":{}},{"id":"75af1f8a-7ad7-441f-80a2-5c337484097f","cell_type":"markdown","source":"### Subprocess\nIf the external executable is called only once, then the call to the external executable can be represented in a Python function with the [subprocess](https://docs.python.org/3/library/subprocess.html) module of the Python standard library. In the example below the shell command `echo test` is submitted to the `execute_shell_command()` function, which itself is submitted to the `Executor` class.","metadata":{}},{"id":"83515b16-c4d5-4b02-acd7-9e1eb57fd335","cell_type":"code","source":"from executorlib import SingleNodeExecutor","metadata":{"trusted":false},"outputs":[],"execution_count":1},{"id":"f1ecee94-24a6-4bf9-8a3d-d50eba994367","cell_type":"code","source":"def execute_shell_command(\n command: list, universal_newlines: bool = True, shell: bool = False\n):\n import subprocess\n\n return subprocess.check_output(\n command, universal_newlines=universal_newlines, shell=shell\n )","metadata":{"trusted":false},"outputs":[],"execution_count":2},{"id":"32ef5b63-3245-4336-ac0e-b4a6673ee362","cell_type":"code","source":"with SingleNodeExecutor() as exe:\n future = exe.submit(\n execute_shell_command,\n [\"echo\", \"test\"],\n universal_newlines=True,\n shell=False,\n )\n print(future.result())","metadata":{"trusted":false},"outputs":[{"name":"stdout","output_type":"stream","text":"test\n\n"}],"execution_count":3},{"id":"54837938-01e0-4dd3-b989-1133d3318929","cell_type":"markdown","source":"### Interactive\nThe more complex case is the interaction with an external executable during the run time of the executable. This can be implemented with executorlib using the block allocation `block_allocation=True` feature. The external executable is started as part of the initialization function `init_function` and then the indivdual functions submitted to the `Executor` class interact with the process which is connected to the external executable. \n\nStarting with the definition of the executable, in this example it is a simple script which just increases a counter. The script is written in the file `count.py` so it behaves like an external executable, which could also use any other progamming language. ","metadata":{}},{"id":"dedf138f-3003-4a91-9f92-03983ac7de08","cell_type":"code","source":"count_script = \"\"\"\\\ndef count(iterations):\n for i in range(int(iterations)):\n print(i)\n print(\"done\")\n\n\nif __name__ == \"__main__\":\n while True:\n user_input = input()\n if \"shutdown\" in user_input:\n break\n else:\n count(iterations=int(user_input))\n\"\"\"\n\nwith open(\"count.py\", \"w\") as f:\n f.writelines(count_script)","metadata":{"trusted":false},"outputs":[],"execution_count":4},{"id":"771b5b84-48f0-4989-a2c8-c8dcb4462781","cell_type":"markdown","source":"The connection to the external executable is established in the initialization function `init_function` of the `Executor` class. By using the [subprocess](https://docs.python.org/3/library/subprocess.html) module from the standard library two process pipes are created to communicate with the external executable. One process pipe is connected to the standard input `stdin` and the other is connected to the standard output `stdout`. ","metadata":{}},{"id":"8fe76668-0f18-40b7-9719-de47dacb0911","cell_type":"code","source":"def init_process():\n import subprocess\n\n return {\n \"process\": subprocess.Popen(\n [\"python\", \"count.py\"],\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE,\n universal_newlines=True,\n shell=False,\n )\n }","metadata":{"trusted":false},"outputs":[],"execution_count":5},{"id":"09dde7a1-2b43-4be7-ba36-38200b9fddf0","cell_type":"markdown","source":"The interaction function handles the data conversion from the Python datatypes to the strings which can be communicated to the external executable. It is important to always add a new line `\\n` to each command send via the standard input `stdin` to the external executable and afterwards flush the pipe by calling `flush()` on the standard input pipe `stdin`. ","metadata":{}},{"id":"7556f2bd-176f-4275-a87d-b5c940267888","cell_type":"code","source":"def interact(shell_input, process, lines_to_read=None, stop_read_pattern=None):\n process.stdin.write(shell_input)\n process.stdin.flush()\n lines_count = 0\n output = \"\"\n while True:\n output_current = process.stdout.readline()\n output += output_current\n lines_count += 1\n if stop_read_pattern is not None and stop_read_pattern in output_current:\n break\n elif lines_to_read is not None and lines_to_read == lines_count:\n break\n return output","metadata":{"trusted":false},"outputs":[],"execution_count":6},{"id":"5484b98b-546f-4f2c-8db1-919ce215e228","cell_type":"markdown","source":"Finally, to close the process after the external executable is no longer required it is recommended to define a shutdown function, which communicates to the external executable that it should shutdown. In the case of the `count.py` script defined above this is achieved by sending the keyword `shutdown`. ","metadata":{}},{"id":"d5344d2b-cb53-4d38-8cae-621e3b98bb56","cell_type":"code","source":"def shutdown(process):\n process.stdin.write(\"shutdown\\n\")\n process.stdin.flush()","metadata":{"trusted":false},"outputs":[],"execution_count":7},{"id":"3899467c-dc54-41cb-b05e-b60f5cf97e46","cell_type":"markdown","source":"With these utility functions is to possible to communicate with any kind of external executable. Still for the specific implementation of the external executable it might be necessary to adjust the corresponding Python functions. Therefore this functionality is currently limited to developers and not considered a general feature of executorlib. ","metadata":{}},{"id":"747c1b78-4804-467b-9ac8-8144d8031da3","cell_type":"code","source":"with SingleNodeExecutor(\n max_workers=1,\n init_function=init_process,\n block_allocation=True,\n) as exe:\n future = exe.submit(\n interact, shell_input=\"4\\n\", lines_to_read=5, stop_read_pattern=None\n )\n print(future.result())\n future_shutdown = exe.submit(shutdown)\n print(future_shutdown.result())","metadata":{"trusted":false},"outputs":[{"name":"stdout","output_type":"stream","text":"0\n1\n2\n3\ndone\n\nNone\n"}],"execution_count":8},{"id":"96e56af9-3031-4d7b-9111-d2d031a0a6e4","cell_type":"markdown","source":"## License\n```\nBSD 3-Clause License\n\nCopyright (c) 2022, Jan Janssen\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n* Neither the name of the copyright holder nor the names of its\n contributors may be used to endorse or promote products derived from\n this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n```","metadata":{}},{"id":"c2522d54-a00b-49ae-81e4-69e8fa05c9c3","cell_type":"markdown","source":"## Modules\nWhile it is not recommended to link to specific internal components of executorlib in external Python packages but rather only the `Executor` classes should be used as central interfaces to executorlib, the internal architecture is briefly outlined below. \n* `backend` - the backend module contains the functionality for the Python processes created by executorlib to execute the submitted Python functions.\n* `base` - the base module contains the definition of the executorlib `ExecutorBase` class which is internally used to create the different interfaces. To compare if an given `Executor` class is based on executorlib compare with the `ExecutorBase` class which can be imported as `from executorlib.base.executor import ExecutorBase`.\n* `cache` - the cache module defines the file based communication for the [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html).\n* `interactive` - the interactive modules defines the [zero message queue](https://zeromq.org) based communication for the [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html) and the [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html).\n* `interfaces` - the different `Executor` classes are defined here, namely `SingleNodeExecutor`, `SlurmClusterExecutor`, `SlurmJobExecutor`, `FluxClusterExecutor` and `FluxJobExecutor`.\n* `standalone` - the standalone module contains a number of utility functions which only depend on external libraries and do not have any internal dependency to other parts of `executorlib`. This includes the functionality to generate executable commands, the [h5py](https://www.h5py.org) based interface for caching, a number of input checks, routines to plot the dependencies of a number of future objects, functionality to interact with the [queues defined in the Python standard library](https://docs.python.org/3/library/queue.html), the interface for serialization based on [cloudpickle](https://github.com/cloudpipe/cloudpickle) and finally an extension to the [threading](https://docs.python.org/3/library/threading.html) of the Python standard library.\n\nGiven the level of separation the integration of submodules from the standalone module in external software packages should be the easiest way to benefit from the developments in executorlib beyond just using the `Executor` class. ","metadata":{}},{"id":"89d974b1-1f00-472f-bae0-34d029989477","cell_type":"markdown","source":"## Test Environment\nThe test environment of the executorlib library consists of three components - they are all available in the executorlib [Github repository](https://github.com/pyiron/executorlib):\n* The [Jupyter Notebooks](https://github.com/pyiron/executorlib/tree/main/notebooks) in the executorlib Github repository demonstrate the usage of executorlib. These notebooks are used as examples for new users, as documentation available on [readthedocs.org](https://executorlib.readthedocs.io) and as integration tests.\n* The [likelihood benchmark](https://github.com/pyiron/executorlib/blob/main/tests/benchmark/llh.py) to compare the performance on a single compute node to the built-in interfaces in the standard library. The benchmark can be run with the following parameters `python llh.py static`. Here `static` refers to single process execution, `process` refers to the `ProcessPoolExecutor` from the standard library, `thread` refers to the `ThreadPoolExecutor` from the standard library, `executorlib` refers to the `SingleNodeExecutor` in executorlib and `block_allocation` to the `SingleNodeExecutor` in `executorlib` with block allocation enabled. Finally, for comparison to `mpi4py` the test can be executed with `mpiexec -n 4 python -m mpi4py.futures llh.py mpi4py`.\n* The [unit tests](https://github.com/pyiron/executorlib/tree/main/tests) these can be executed with `python -m unittest discover .` in the `tests` directory. The tests are structured based on the internal structure of executorlib. Tests for the `SingleNodeExecutor` are named `test_singlenodeexecutor_*.py` and correspondingly for the other modules. ","metadata":{}},{"id":"99825d59-d7d5-4ba6-b576-f00e3dfbc79a","cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]} \ No newline at end of file