TypeError: cannot pickle 'sqlite3.Connection' object in pyCharm

[wiessall]

General

• Operating System: Ubuntu 22.04
• Python version: 3.9.7
• Pandas version: 1.4.2
• Pandarallel version: 1.6.3

Acknowledgement

• My issue is NOT present when using pandas without alone (without pandarallel)
• If I am on Windows, I read the Troubleshooting page
  before writing a new bug report

Bug description

I observe this when running parallel_apply in pyCharm. #76 sounds similar to me, but none of the tricks suggested there work for me. At this point I am also not sure if it's more of an issue with pyCharm.

Observed behavior

Traceback (most recent call last):
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-39-d230d86ff5ef>", line 1, in <module>
    iris.groupby("species").parallel_apply(lambda x: np.mean(x))
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/pandarallel/core.py", line 265, in closure
    dilled_user_defined_function = dill.dumps(user_defined_function)
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 364, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 336, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 487, in dump
    self.save(obj)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 692, in save_reduce
    save(args)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 886, in save_tuple
    save(element)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 717, in save_reduce
    save(state)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "~/anaconda3/envs/tor/lib/python3.9/site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 997, in _batch_setitems
    save(v)
  File "~/anaconda3/envs/tor/lib/python3.9/pickle.py", line 578, in save
    rv = reduce(self.proto)
TypeError: cannot pickle 'sqlite3.Connection' object

Minimal but working code sample to ease bug fix for pandarallel team

import pandas as pd
import numpy as np
from pandarallel import pandarallel
pandarallel.initialize(use_memory_fs=False)
iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')
iris.groupby("species").parallel_apply(lambda x: np.mean(x))

[jmrichardson]

I am having same issue

[nalepae]

Pandaral·lel is looking for a maintainer!
If you are interested, please open an GitHub issue.

[shermansiu]

The minimal code sample also produces an error for me outside of PyCharm.

Python: 3.10.13
Pandarallel: 1.6.5
Pandas: 2.2.0