Please document one user error pattern

[marxin]

It took me more than hour to realize what's wrong:

#!/usr/bin/env python3

from pebble import ProcessPool

class Tester:
    def __init__(self):
        self.futures = []

    def run_one(self, v):
        time.sleep(1)

    def run(self):
        while True:
            print('new ProcessPool')
            with ProcessPool(max_workers=16) as pool:
                f = pool.schedule(self.run_one, [1])
                self.futures.append(f)
                f.result()

Tester().run()

$ ./sample.py
new ProcessPool
Traceback (most recent call last):
  File "/tmp/pe.py", line 20, in <module>
    Tester().run()
  File "/tmp/pe.py", line 18, in run
    f.result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/lib64/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/site-packages/pebble/pool/process.py", line 204, in schedule
    self.worker_manager.dispatch(task)
  File "/usr/lib/python3.8/site-packages/pebble/pool/process.py", line 335, in dispatch
    raise error
  File "/usr/lib/python3.8/site-packages/pebble/pool/process.py", line 333, in dispatch
    self.pool_channel.send(WorkerTask(task.id, task.payload))
  File "/usr/lib/python3.8/site-packages/pebble/pool/channel.py", line 66, in send
    return self.writer.send(obj)
  File "/usr/lib64/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib64/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle '_thread.RLock' object

The problem is that pickle is called for my Tester which contains a reference to pebble.ProcessFuture.

[noxdafox]

As the documentation states, Pebble is affected by the same limitations of the Python multiprocessing module.

What you describe, is a very common issue with most of the Python pool implementations.

Some examples:
https://stackoverflow.com/questions/8804830/python-multiprocessing-picklingerror-cant-pickle-type-function
https://stackoverflow.com/questions/1816958/cant-pickle-type-instancemethod-when-using-multiprocessing-pool-map?noredirect=1&lq=1
https://stackoverflow.com/questions/25156768/cant-pickle-type-instancemethod-using-pythons-multiprocessing-pool-apply-a?noredirect=1&lq=1

[marxin]

What a noobish mistake I made, thank you for the links!