Skip to content
No description, website, or topics provided.
Jupyter Notebook Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
0_dask_tutorial
1_dask_array
2_digits_of_pi
dask_slurm use newer image May 14, 2019
.gitignore
GIL_counter.py
README.md
intro.ipynb
notebook_singularity.slrm
numba_groupby_pixels.ipynb

README.md

Webinar recording

https://www.sdsc.edu/Events/training/webinars/distributed_parallel_computing_with_python_2019/recording/

Python for HPC

This webinar provides an introduction to distributed computing with Python, we will show how to modify a standard Python script to use multiple CPU cores using the concurrent.futures module from the Python standard library and then the dask package. Then we will leverage dask workers running on other machines to distribute a data processing task and monitor its execution through the live dask dashboard. You will understand the difference between threads and processes, how the Global Interpreter Lock works and principles of distributed computing. All material will be available as Jupyter Notebooks.

  • notebook_singularity.slrm: SLURM script to launch a Jupyter Notebook job through Singularity
  • digits_of_pi/: example of parallel programming estimating the digits of pi, using concurrent.futures and dask
  • dask scripts: launch_workers.sh and launch_scheduler.sh and dask_workers.slrm launch all the components of dask distributed
  • Slides on Google Docs
You can’t perform that action at this time.