PyO3+Polars Expr::AnonymousFunction: Execution (python) deadlocks when function points to python code

[AnatolyBuga]

Polars version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Issue description

Updated MRE

I'd like users to define functions in python to be converted into Expr(Anonymous functions) for later use.

However, as soon as I introduce a call to Python::with_gil from inside an AnonymousFunction - the interprete just hangs

Any thoughts would be much appreciated.

Reproducible example

use polars::lazy::dsl::{map_binary, GetOutput};
use polars::prelude::{DataFrame};
use polars::prelude::*;
use pyo3::{pyclass, pymodule, pymethods, Python, types::{PyModule, PyType}, PyResult, PyObject};
use pyo3::prelude::*;

#[pyclass]
pub struct DataSet {
    data: DataFrame
}
#[pymethods]
impl DataSet {
    #[classmethod]
    fn new(_: &PyType) -> Self {

        let data = df!["a" => [1, 2, 3],
         "b" => [4, 5, 6], 
         "c" => ["a", "d", "b"]].unwrap();
        Self{data}
    }

    fn execute(&self, e: &ExprWrapper)  {
        let res = self.data.clone().lazy().with_column(e.e.clone())
            .collect()
            .unwrap();
        dbg!(res);
    }
}

#[pyclass]
pub struct ExprWrapper {
    pub e: Expr
}

#[pyfunction]
fn ultimeasure(pyfnc: PyObject) -> ExprWrapper{
    let functor = move |_: Series, _: Series| { 
        let _ = pyfnc.clone();

        // Here I would convert Series to Python Side Series, call lambda, then result back to Rust Series
        Python::with_gil(move |_| {});
        let srs = Series::new("a", [1,2]);
        Ok(Some(srs))
        };

    let e = map_binary(col("a"), col("b"), functor, GetOutput::from_type(DataType::Float64));
    ExprWrapper{e}
}
#[pymodule]
fn pyo3issue(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_class::<DataSet>()?;
    m.add_wrapped(wrap_pyfunction!(ultimeasure)).unwrap();
    Ok(())
}

!!! Python side, the code I am executing:

import pyo3issue
import polars as pl

def calculator(a: pl.Series, b: pl.Series):
            print("here")
            return a**b

c = pyo3issue.DataSet.new()
m = pyo3issue.ultimeasure(lambda a,b : a**b)
print(c.execute(m))

Expected behavior

I don't understand why this doesn't work. It is very similar to map_mul of py-polars. With only exception that map_mul is a #[pyfunction].

Installed versions

Finally, Cargo.toml:

[lib]
name = "pyo3issue"
# "cdylib" is necessary to produce a shared library for Python to import from.
crate-type = ["cdylib"]

[dependencies]
polars = { version = "0.28.0", features = [
    "lazy"
] }
pyo3-polars = "0.2.0"
polars-arrow = "0.28.0"

[dependencies.pyo3]
version = "0.18.0"
# "abi3-py37" tells pyo3 (and maturin) to build using the stable ABI with minimum Python version 3.7
features = ["abi3-py37"]

[AnatolyBuga]

may be I would be able to achieve converting python functions into UDFs with #6325

[AnatolyBuga]

What was missing ispy.allow_threads() around .collect()(because this is where functor is actually called. For more details see this thread ).