$ pip install pandarallel [--user]
- Linux or macOS (Windows is not supported at the moment)
- Parallelization has a cost (instanciating new processes, sending data via shared memory, etc ...), so parallelization is efficiant only if the amount of calculation to parallelize is high enough. For very little amount of data, using parallezation not always worth it.
- Functions applied should NOT be lambda functions.
from pandarallel import pandarallel from math import sin pandarallel.initialize() # FORBIDDEN df.parallel_apply(lambda x: sin(x**2), axis=1) # ALLOWED def func(x): return sin(x**2) df.parallel_apply(func, axis=1)
An example of each API is available here.
For some examples, here is the comparative benchmark with and without using Pandaral·lel.
Computer used for this benchmark:
- OS: Linux Ubuntu 16.04
- Hardware: Intel Core i7 @ 3.40 GHz - 4 cores
For those given examples, parallel operations run approximatively 4x faster than the standard operations (except for
series.map which runs only 3.2x faster).
First, you have to import
from pandarallel import pandarallel
Then, you have to initialize it.
This method takes 3 optional parameters:
shm_size_mb: The size of the Pandarallel shared memory in MB. If the default one is too small, it is possible to set a larger one. By default, it is set to 2 GB. (int)
nb_workers: The number of workers. By default, it is set to the number of cores your operating system sees. (int)
progress_bar: Put it to
Trueto display a progress bar.
WARNING: Progress bar is an experimental feature. This can lead to a
considerable performance loss.
Not available for
df a pandas DataFrame,
series a pandas Series,
func a function to
args2 some arguments &
col_name a column name:
|Without parallelisation||With parallelisation|
You will find a complete example here for each line of this table.
I have 8 CPUs but
parallel_apply speeds up computation only about x4. Why ?
Actually Pandarallel can only speed up computation until about the number of cores your computer has. The majority of recent CPUs (like Intel core-i7) uses hyperthreading. For example, a 4 cores hyperthreaded CPU will show 8 CPUs to the Operating System, but will really have only 4 physical computation units.
On Ubuntu, you can get the number of cores with
$ grep -m 1 'cpu cores' /proc/cpuinfo.
When I run
from pandarallel import pandarallel, I get
ModuleNotFoundError: No module named 'pyarrow._plasma. Why?
Are you using Windows? Pandarallel relies on the Pyarrow Plasma shared memory to work. Currently, Pyarrow Plasma works only on Linux & macOS (Windows in not supported). So sorry, but for now Pandarallel is supported only on Linux & macOS ...