Unify Table representations

[timsaucer]

Which issue does this PR close?

Closes #1239
Closes #1245

Rationale for this change

This is built on top of #1243

With this change we have a single class and single entry point for turning any table-like object into a Table class in python. On the rust side, we use this one struct for doing things like registering with the schema provider or session context.

What changes are included in this PR?

Pushes the Python object type evaluation into the rust side with a constructor for the PyTable struct that handles:

• Existing PyTable objects
• pyarrow datasets
• DataFrames (as views)
• External table providers via PyCapsule

This removes the cognitive load from the end users from having to understand the differences between any of these. If they have something that is table-like they can simply call Table(my_obj) and get something that can be used to register in the schema provider or directly with the session context.

Are there any user-facing changes?

We have deprecated one method, register_table_provider.

[timsaucer]

@kosiew Would you mind taking a look at this? I started from your PR and made a few adjustments to try to make it as ergonomic as possible for the end users. My goal is that a consumer of this project does not need to know much about the internals of the table types as long as they have something that should be table-like.

With this change it pushes most of the type coercion into a single place in the rust side. I think it also makes it nice to not have to have multiple entry points such as from_dataset and from_capsule. Any python object that can be one of our table types is checked in the rust code.

What do you think?

[kosiew]

@timsaucer ,

Centralizing the coercion logic in PyTable definitely improves ergonomics; callers can now hand almost anything “table-like” to Rust and let it figure things out, which is great.
The remaining blocker is that the unified wrapper no longer exposes __datafusion_table_provider__, so anything that depends on that FFI capsule now fails. PyTableFunction.__call__ still converts Python-returned objects by invoking table_provider_from_pycapsule, which requires __datafusion_table_provider__ to be present; without it, Python table UDTFs raise PyNotImplementedError, and external integrations that unwrap tables via the capsule break as well.

Restoring the capsule hook (even if it just forwards to the inner provider) keeps those advanced scenarios working without forcing ordinary users to think about internals.

[timsaucer]

The remaining blocker is that the unified wrapper no longer exposes __datafusion_table_provider__, so anything that depends on that FFI capsule now fails. PyTableFunction.__call__ still converts Python-returned objects by invoking table_provider_from_pycapsule, which requires __datafusion_table_provider__ to be present; without it, Python table UDTFs raise PyNotImplementedError, and external integrations that unwrap tables via the capsule break as well.

Can you expand on this? I don't understand the use case. I did run the integration tests in our repo without issue. I removed some of the tests you had added that were looking for __datafusion_table_provider__ because I couldn't understand their utility.

[kosiew]

@timsaucer,

here's an example that shows the NotImplementedError

examples/table_capsule_failure.py

"""Demonstrate how missing __datafusion_table_provider__ breaks table UDTFs.

Run with ``python examples/table_capsule_failure.py``.

This example mirrors how advanced integrations unwrap ``Table`` instances via the
``__datafusion_table_provider__`` PyCapsule. The refactor that removed this method
means user-defined table functions returning ``Table`` now raise ``NotImplementedError``
and the script prints the resulting error message instead of crashing.
"""

from __future__ import annotations

from datafusion import SessionContext, Table, udtf


def main() -> None:
    """Register a Python table UDTF that returns a ``Table`` and trigger it."""

    ctx = SessionContext()
    failing_table = Table(ctx.sql("SELECT 1 AS value"))

    @udtf("capsule_dependent")
    def capsule_dependent_udtf() -> Table:
        """Return a ``Table`` so DataFusion unwraps it via the FFI capsule."""

        # Prior to the refactor the wrapper exposed ``__datafusion_table_provider__``
        # so this conversion succeeded. Without it the runtime raises a
        # ``NotImplementedError`` complaining about the missing attribute.
        return failing_table

    ctx.register_udtf(capsule_dependent_udtf)

    # Executing the UDTF now fails because ``Table`` no longer exposes the
    # ``__datafusion_table_provider__`` helper that PyTableFunction expects.
    try:
        ctx.sql("SELECT * FROM capsule_dependent()").collect()
        print("capsule_dependent() works")
    except NotImplementedError as err:
        # Document the regression by surfacing the missing capsule attribute
        # instead of crashing with a panic inside the execution engine.
        print(
            "capsule_dependent() failed due to missing __datafusion_table_provider__: "
            f"{err}"
        )


if __name__ == "__main__":
    main()

On my computer, I got this traceback:

  File "/Users/kosiew/GitHub/datafusion-python/examples/table_capsule_failure.py", line 48, in <module>
    main()
    ~~~~^^
  File "/Users/kosiew/GitHub/datafusion-python/examples/table_capsule_failure.py", line 36, in main
    ctx.sql("SELECT * FROM capsule_dependent()").collect()
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kosiew/GitHub/datafusion-python/python/datafusion/context.py", line 611, in sql
    return DataFrame(self.ctx.sql(query))
                     ~~~~~~~~~~~~^^^^^^^
Exception: DataFusion error: Execution("PyErr { type: <class 'NotImplementedError'>, value: NotImplementedError('__datafusion_table_provider__ does not exist on Table Provider object.'), traceback: None }")

[timsaucer]

I was able to reproduce the problem, but I think we can lean on the PyTable to handle these cases as well. I have pushed a unit test that fails without my addition and passes with my addition in the last commit. Do you still have concerns?

[kosiew]

If I have a python object that implements TableProvider via PyCapsule, I should be able to pass this object directly to SessionContext.read_table

Source: #1245

I tested with SessionContext.read_table and it chokes on raw PyCapsule objects.

from __future__ import annotations

import ctypes

from datafusion import SessionContext, Table


# Keep the backing memory alive for the lifetime of the module so the capsule
# always wraps a valid (non-null) pointer. The capsule content is irrelevant for
# this regression example—we only need a non-null address.
_DUMMY_CAPSULE_BYTES = ctypes.create_string_buffer(b"x")


def make_table_provider_capsule() -> object:
    """Create a dummy PyCapsule with the expected table provider name."""

    pycapsule_new = ctypes.pythonapi.PyCapsule_New
    pycapsule_new.restype = ctypes.py_object
    pycapsule_new.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p]
    dummy_ptr = ctypes.cast(_DUMMY_CAPSULE_BYTES, ctypes.c_void_p)
    return pycapsule_new(dummy_ptr, b"datafusion_table_provider", None)


def main() -> None:
    """Attempt to use the capsule the same way existing callers do."""

    ctx = SessionContext()
    try:
        capsule = make_table_provider_capsule()
    except Exception as err:
        print("Creating the PyCapsule failed:", err)
        return


    ctx.read_table(capsule)
    

if __name__ == "__main__":
    main()

raises this Traceback:

  File "/Users/kosiew/GitHub/datafusion-python/examples/raw_capsule_registration_failure.py", line 49, in <module>
    main()
    ~~~~^^
  File "/Users/kosiew/GitHub/datafusion-python/examples/raw_capsule_registration_failure.py", line 45, in main
    ctx.read_table(capsule)
    ~~~~~~~~~~~~~~^^^^^^^^^
  File "/Users/kosiew/GitHub/datafusion-python/python/datafusion/context.py", line 1184, in read_table
    return DataFrame(self.ctx.read_table(table))
                     ~~~~~~~~~~~~~~~~~~~^^^^^^^
ValueError: dataset argument must be a pyarrow.dataset.Dataset object

[timsaucer]

I don't think that example demonstrates how users are expected to provide PyCapsule based providers. I changed it slightly below to fit TableProviderExportable. With this change it does segfault, but I expect that because it is not a valid object.

from __future__ import annotations

import ctypes

from datafusion import SessionContext, Table


# Keep the backing memory alive for the lifetime of the module so the capsule
# always wraps a valid (non-null) pointer. The capsule content is irrelevant for
# this regression example—we only need a non-null address.
_DUMMY_CAPSULE_BYTES = ctypes.create_string_buffer(b"x")

class CapsuleContainer:
    def __init__(self):
        self.__datafusion_table_provider__ = make_table_provider_capsule

def make_table_provider_capsule() -> object:
    """Create a dummy PyCapsule with the expected table provider name."""

    pycapsule_new = ctypes.pythonapi.PyCapsule_New
    pycapsule_new.restype = ctypes.py_object
    pycapsule_new.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p]
    dummy_ptr = ctypes.cast(_DUMMY_CAPSULE_BYTES, ctypes.c_void_p)
    return pycapsule_new(dummy_ptr, b"datafusion_table_provider", None)


def main() -> None:
    """Attempt to use the capsule the same way existing callers do."""

    ctx = SessionContext()
    try:
        capsule = CapsuleContainer()
    except Exception as err:
        print("Creating the PyCapsule failed:", err)
        return


    ctx.read_table(capsule)
    

if __name__ == "__main__":
    main()

[kosiew]

@timsaucer

I changed it slightly below to fit TableProviderExportable. With this change it does segfault, but I expect that because it is not a valid object.

We still don't have an example/test to show that SessionContext.read_table can take a PyCapsule object (#1245).

Can you resolve the conflicts and also leave #1245 open to move the PR forward

[timsaucer]

@kosiew Here is the unit test for read_table with a PyCapsule based table provider.

[timsaucer]

I removed the statement this would close #1245 per your request, but I do think it has the appropriate unit test.

[kosiew]

I removed the statement this would close #1245 per your request, but I do think it has the appropriate unit test.

You're right.
This does close #1245 too.

[timsaucer]

Thank you @kosiew for the collaboration on this! I think the final result is a really nice step forward!