[Python] Expose additional Cython wrap/unwrap helpers

[mikelui]

Describe the enhancement requested

This is a feature request to expose wrap/unwrap utilities for RecordBatchReader and RecordBatchWriter between C++/Python.

It's useful in custom utilities to expose core functionality in C++ (e.g. extending RecordBatchReader for my bespoke data store) to user-facing utilities in Python to consume them. And vice-versa.

Arrow exposes the ability to communicate types between C++/Python via a small wrap/unwrap utility (referenced below). I propose we extend RecordBatchReader and RecordBatchWriter for these as well.

arrow/python/pyarrow/public-api.pxi

Lines 409 to 426 in c9bfca5

	cdef api bint pyarrow_is_batch(object batch):
	return isinstance(batch, RecordBatch)
	cdef api shared_ptr[CRecordBatch] pyarrow_unwrap_batch(object batch):
	cdef RecordBatch bat
	if pyarrow_is_batch(batch):
	bat = <RecordBatch>(batch)
	return bat.sp_batch
	return shared_ptr[CRecordBatch]()
	cdef api object pyarrow_wrap_batch(
	const shared_ptr[CRecordBatch]& cbatch):
	cdef RecordBatch batch = RecordBatch.__new__(RecordBatch)
	batch.init(cbatch)
	return batch

arrow/python/pyarrow/src/arrow/python/pyarrow.h

Lines 48 to 56 in c9bfca5

	#define DECLARE_WRAP_FUNCTIONS(FUNC_SUFFIX, TYPE_NAME) \
	ARROW_PYTHON_EXPORT bool is_##FUNC_SUFFIX(PyObject*); \
	ARROW_PYTHON_EXPORT Result<std::shared_ptr<TYPE_NAME>> unwrap_##FUNC_SUFFIX( \
	PyObject*); \
	ARROW_PYTHON_EXPORT PyObject* wrap_##FUNC_SUFFIX(const std::shared_ptr<TYPE_NAME>&);
	DECLARE_WRAP_FUNCTIONS(buffer, Buffer)
	DECLARE_WRAP_FUNCTIONS(data_type, DataType)

arrow/python/pyarrow/src/arrow/python/pyarrow.cc

Lines 52 to 67 in c9bfca5

	#define DEFINE_WRAP_FUNCTIONS(FUNC_SUFFIX, TYPE_NAME) \
	bool is_##FUNC_SUFFIX(PyObject* obj) { return ::pyarrow_is_##FUNC_SUFFIX(obj) != 0; } \
	\
	PyObject* wrap_##FUNC_SUFFIX(const std::shared_ptr<TYPE_NAME>& src) { \
	return ::pyarrow_wrap_##FUNC_SUFFIX(src); \
	} \
	Result<std::shared_ptr<TYPE_NAME>> unwrap_##FUNC_SUFFIX(PyObject* obj) { \
	auto out = ::pyarrow_unwrap_##FUNC_SUFFIX(obj); \
	if (out) { \
	return std::move(out); \
	} else { \
	return UnwrapError(obj, #TYPE_NAME); \
	} \
	}
	DEFINE_WRAP_FUNCTIONS(buffer, Buffer)

Component(s)

C++, Python

[mikelui]

One discussion point is RecordBatchWriter. It's implemented as:

arrow/python/pyarrow/ipc.pxi

Lines 465 to 470 in c9bfca5

	cdef class _CRecordBatchWriter(_Weakrefable):
	"""The base RecordBatchWriter wrapper.
	Provides common implementations of convenience methods. Should not
	be instantiated directly by user code.
	"""

The internals (a single shared_ptr<RecordBatchWriter>) are exactly the same as RecordBatchReader, except it's name does not follow convention, and would not play nicely with existing macros. One simple path forward is to make a new Cython RecordBatchWriter(_CRecordBatchWriter) and then make other types inherit from that:

arrow/python/pyarrow/ipc.pxi

Lines 551 to 554 in c9bfca5

	cdef class _RecordBatchStreamWriter(_CRecordBatchWriter):
	cdef:
	CIpcWriteOptions options
	bint closed

---

Secondarily, it's defined in the ::arrow::ipc namespace, while RecordBatchReader is defined in the ::arrow namespace.

That seems like a larger change. I'll submit a PR just for RecordBatchReader and will wait for feedback on changes to RecordBatchWriter.

[thisisnic]

This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.