[Python] Dataset PartitioningFactory cannot be serialized in Python

[rjzamora]

Describe the bug, including details regarding any error messages, version, and platform.

I would like to be able to serialize a dictionary of pyarrow.dataset.dataset key-word arguments (for parallel processing in Dask). However, it is not possible to do this when one of those arguments contains a Partitioning/PartitioningFactory object, because those objects cannot be serialized in Python.

Reproducer:

In [1]: import pyarrow.dataset as ds
   ...: import pickle
   ...: 
   ...: partitioning = ds.partitioning(flavor="hive")
   ...: pickle.dumps(partitioning)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[1], line 5
      2 import pickle
      4 partitioning = ds.partitioning(flavor="hive")
----> 5 pickle.dumps(partitioning)

File stringsource:2, in pyarrow._dataset.PartitioningFactory.__reduce_cython__()

TypeError: self.factory,self.wrapped cannot be converted to a Python object for pickling

Component(s)

Python

[westonpace]

All the partitioning objects we have today can boil down to a schema (which can be saved as an empty parquet/arrow file) and a string denoting the type (e.g. "dictionary" or "hive" or "filename"). I'm not sure if this is helpful or not since I suspect the goal is automatic serialization.

[rjzamora]

Thanks @westonpace ! The current workaround in Dask is indeed to allow the user to specify a dictionary like {"flavor": "hive", "schema": ...}. This works fine, but the Dask UX would certainly be cleaner if the user could pass in something like an initialized HivePartitioning object.

[rjzamora]

Thanks @jorisvandenbossche !