polars hangup using multi-process

[yanxiu0614]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

When I use polars and the process pool to improve efficiency, I find that the whole program will be stuck, and the expected work is not obtained. After debugging, I found that the following code will cause blocking. This is a recurring code that probably doesn't make any sense other than reproducing. Also, if I remove read_csv from the read function or use pandas read_csv instead, everything works as expected.

Reproducible example

import polars as pl
import numpy as np
from concurrent.futures import ProcessPoolExecutor


def read():
    df = pl.DataFrame({"value": np.random.standard_normal(700)}).write_csv("test.csv")
    _df = pl.read_csv("test.csv")
    return _df


def minmax(series):
    minval = series.min()
    return (series[-1] - minval) / (series.max() - minval)


def rolling(_df):
    print("enter")
    _df.with_columns(_df["value"].rolling_apply(minmax, 50).alias("score"))
    print("exit")


if __name__ == '__main__':
    df = pl.DataFrame({"value": np.random.standard_normal(700)})
    pool = ProcessPoolExecutor(1)
    for i in read():
        pool.submit(rolling, df)
    pool.shutdown()

Expected behavior

print enter exit

Installed versions

OS:Linux x64(debian bullseye or ubuntu jammy ) Python: 3.9.2 or 3.10.6 polars: polars-0.16.12

[cmdlineluser]

https://pola-rs.github.io/polars-book/user-guide/howcani/multiprocessing.html#combining-polars-with-pythons-multiprocessing

For the locking issue - you'd need to use spawn or forkserver.

e.g.

Pool(..., mp_context=multiprocessing.get_context("spawn"))

[yanxiu0614]

I confirm that I need to go through all the docs again, thanks for your help.

[tocab]

The page linked by @cmdlineluser does not exist anymore. Is there a new resource for using multiprocessing with polars?

[alicanb]

https://pola-rs.github.io/polars-book/user-guide/misc/multiprocessing/