Efficient Python for High-Performance Parallel Computing
This tutorial is targeted at the intermediate-to-advanced Python user who wants to extend Python into High-Performance Computing. The tutorial will provide hands-on examples and essential performance tips every developer should know for writing effective parallel Python. The result will be a clear sense of possibilities and best practices using Python in HPC environments.
Many of the examples you often find on parallel Python focus on the mechanics of getting the parallel infrastructure working with your code, and not on actually building good portable parallel Python. This tutorial is intended to be a broad introduction to writing high-performance parallel Python that is well suited to both the beginner and the veteran developer. Parallel efficiency starts with the speed of the target code itself, so we will start with how to evolve code from for-loops to Python looping constructs and vector programming. We will also discuss tools and techniques to optimize your code for speed and memory performance.
The tutorial will overview working with the common parallel communication technologies (threading, multiprocessing, MPI) and introduce the use of parallel programming models such as blocking and non-blocking pipes, asynchronous and iterative conditional maps, and map-reduce. We will discuss strategies for extending parallel workflow to utilize hierarchical and heterogeneous computing, distributed parallel computing, and job schedulers. We then return our focus to the speeding up our target code by leveraging parallelism within compiled code using Cython.
At the end of the tutorial, participants should be able to write simple parallel Python scripts, make use of effective parallel programming techniques, and have a framework in place to leverage the power of Python in High-Performance Computing.
All tutorial content can be obtained from this repository either with `git, or by downloading the repository content as a zip file. If you use git, you can clone this repostory with::
$ git clone https://github.com/mmckerns/tuthpc.git
or, download and unzip the zipfile.
As the day of the tutorial get nearer, it is highly recommended to update
this repository. When tutorial content is added or modified, it is
recommended to update your copy of the tutorial. Tutorial content may be
updated up to the day of the tutorial, during the tutorial, and beyond.
To update your copy of the tutorial content with git, change to the tutorial
tuthpc), then pull an update with::
$ git pull
or, download and unzip a new copy of the zipfile.
To be able to run the examples, demos, and exercises in this tutorial, the following packages must be installed::
numpy >= 1.0, flake8 >= 2.0, line_profiler >= 1.0, nose >= 1.3.1, cython >= 0.22, mpi4py >= 2.0.0, pox >= 0.2.2, dill >= 0.2.5, multiprocess >= 0.70.4, ppft >= 22.214.171.124, pathos >= 0.2.0, llvmlite >= 0.2.2, numba >= 0.17.0,
IPython >= 3.0.0, paramiko >= 1.15.2, pyina >= 0.2.0.dev0
All packages can be installed with
>$ pip install setuptools >$ pip install numpy >$ pip install flake8 >$ pip install line_profiler >$ pip install nose >$ pip install cython >$ pip install pathos >$ pip install mpi4py >$ pip install llvmlite >$ pip install numba
>$ pip install "ipython[parallel]" >$ pip install paramiko >$ pip install git+https://github.com/uqfoundation/pyina.git@master
The install of
llvmlite can fail. A more stable choice for
installing these two packages is to use a scientific python distribution
Note that on windows,
mpi4py is known to be a difficult install, and
commonly fails with
conda, etc). You may be able to get a
compatible wheel at either of these two locations::
The optional install for
ipython-parallel also can be difficult, and
commonly fails with
conda, etc). Compatible wheels may be
available at this location::
The optional install for
pyina is not supported on Windows.
The following steps were used by the tutorial author to test on Windows:
# installed Visual Studio Community 2015 RC # installed Python Tools 2.2 RC for Visual Studio 2015 # installed Microsoft Visual C++ Compiler for Python 2.7 # installed Miniconda 3.10.1 (64-bit) for Python 2.7 # installed Git for Windows 1.9.5-preview20150319 (including cmd tools) # installed Microsoft MPI v6 >$ conda install pip >$ conda install setuptools >$ conda install flake8 >$ conda install line_profiler >$ conda install nose >$ conda install numpy >$ conda install cython # get https://github.com/uqfoundation/pox/blob/master/tools/pythonstartup # save as ~\.python (also add as PYTHONSTARTUP to environment variables) # associate .py file with conda's python.exe # fix bug where conda doesn't respect all `sys.argv` # regedit HKEY_CLASSES_ROOT\Applications\python27.exe\shell\open\command # regedit HKEY_CLASSES_ROOT\py_auto_file\shell\open\command >$ pip install pathos >$ pip install https://ci.appveyor.com/api/buildjobs/38i20k6b1r4xn65q/artifacts/dist/mpi4py-2.0.0a0-cp27-none-win_amd64.whl >$ conda install numba
To test your installation, change to the tutorial directory, and run::
>$ python check_env.py OK. All required items installed.
If you choose not install all optional dependencies, you will see a warning::
>$ python check_env.py pyina:: No module named pyina OK. All required intems installed.
Feel free to ignore warnings for optional dependencies.