Citi/parfun

Lightweight parallelisation library for Python.

Parfun is a lightweight library providing helpers to make it easy to write and run a Python function in parallel and distributed systems.

The main feature of the library is its @parfun decorator that transparently executes standard Python functions following the map-reduce pattern:

from parfun import parfun
from parfun.combine.collection import list_concat
from parfun.partition.api import per_argument
from parfun.partition.collection import list_by_chunk

@parfun(
    split=per_argument(
        values=list_by_chunk
    ),
    combine_with=list_concat,
)
def list_pow(values: List[float], factor: float) -> List[float]:
    return [v**factor for v in values]

Features

• Provides significant speedups to existing Python functions
• Does not require any deep knowledge of parallel or distributed computing systems
• Automatically estimates the optimal sub-task splitting (the partition size)
• Automatically handles data transmission, caching and synchronization.
• Supports various distributed computing backends, including Python's multiprocessing, Scaler or Dask.

Benchmarks

Parfun efficiently parallelizes short-duration functions.

When running a short 0.28-second ML function on an AMD Epyc 7313 16-Cores Processor, Parfun provides an impressive 7.4x speedup. Source code for this experiment here.

Quick Start

The official documentation is availaible at citi.github.io/parfun/.

Alternatively, you can build the HTML documentation from the source code:

cd docs
pip install -r requirements.txt
make html

The documentation's main page can then ben found at docs/build/html/index.html.

Take a look at our documentation's quickstart tutorial to get more examples and a deeper overview of the library.

Contributing

Your contributions are at the core of making this a true open source project. Any contributions you make are greatly appreciated.

We welcome you to:

• Fix typos or touch up documentation
• Share your opinions on existing issues
• Help expand and improve our library by opening a new issue

Please review our community contribution guidelines and functional contribution guidelines to get started 👍.

Code of Conduct

We are committed to making open source an enjoyable and respectful experience for our community. See CODE_OF_CONDUCT for more information.

License

This project is distributed under the Apache-2.0 License. See LICENSE for more information.

Contact

If you have a query or require support with this project, raise an issue. Otherwise, reach out to opensource@citi.com.