TypeError: can't pickle _thread.lock objects when launching task from task

[olivier-lacroix]

Hi,

I am using distributed to read a dataframe on a LocalCluster, which works well, and am then trying to client.map a function taking said dataframe as an argument, following the 'launching tasks from tasks' guidelines.

I get a cloudpickle error on the function I try to map pointing to the c.compute line below when data is a dask dataframe. If I persist the dataframe prior to mapping, this seems to work though

def function(key, data=None):
    c = get_client()
    df = c.compute(data.loc[key])
    secede()
    df = c.gather(df)
    rejoin()

Is this a limitation of dask, a bug or just something I am missing?

Thanks

Olivier

[mrocklin]

Hi @olivier-lacroix , can I ask you to provide a small reproducible example that generates the error you're getting?

Also, can you say more about why you're trying to map a function on a dask dataframe? This is somewhat uncommon.

[mrocklin]

Closing this due to lack of response.

[olivier-lacroix]

Sorry Matt for the lack of response.

I was not able to come up with a simple replicable case and ended up moving to a groupy.apply approach in which the problem did not manifest itself.

Cheers,

Olivier