Enable Dataframe to be converted into views which can be used in register_table #1016

kosiew · 2025-02-06T10:15:01Z

Which issue does this PR close?

Closes #1004.

Rationale for this change

Currently, Datafusion supports views via ViewTable, allowing logical plans to be registered as tables. However, this feature is not exposed in the Python bindings. This PR addresses that gap by enabling Dataframes to be converted to view, enabling users to create views programmatically without relying on raw SQL strings.

What changes are included in this PR?

Adds DataFrame.into_view which can then be used in
register_table.

Are there any user-facing changes?

Yes, Python users will now be able to convert dataframes into view, then register_table with the view.

…return PyTable

…view

…comments and assertions

…ions for DataFrame results

kosiew · 2025-02-07T01:33:28Z

python/tests/test_view.py

+    view = df_filtered.into_view()
+
+    assert view.kind == "view"
+
+    ctx.register_table("view1", view)


This is modelled after how into_view is used in datafusion:

async fn with_column_renamed_ambiguous() -> Result<()> { let df = test_table().await?.select_columns(&["c1", "c2", "c3"])?; let ctx = SessionContext::new(); let table = df.into_view(); ctx.register_table("t1", table.clone())?; ctx.register_table("t2", table)?;

…able

timsaucer

I think this is a valuable addition. Thank you!

I do have a few concerns below, and I'm happy to help us come to a common solution.

timsaucer · 2025-02-15T19:28:50Z

src/dataframe.rs

+#[pyclass(name = "TableProvider", module = "datafusion")]
+pub struct PyTableProvider {
+    provider: Arc<dyn TableProvider>,
+}
+
+impl PyTableProvider {
+    pub fn new(provider: Arc<dyn TableProvider>) -> Self {
+        Self { provider }
+    }
+
+    pub fn as_table(&self) -> PyTable {
+        let table_provider: Arc<dyn TableProvider> = self.provider.clone();
+        PyTable::new(table_provider)
+    }
+}
+


In general I think this is a good idea, but I'm worried about causing confusion with a table provider created from a view and a table provider that is passed from an external source using pycapsule. I can imagine a user would think that a table provider object from one place can be used with another. That is, if I create a table provider with into_view I should be able to register it with the session context. Now, I don't think that operation is strictly necssary but I do expect it would cause some confusion.

What I think we want to do is to have a single common PyTableProvider that can be created either via a pycapsule or into_view.

a single common PyTableProvider that can be created either via a pycapsule or into_view.

Do you mean a constructor that takes a pycapsule argument, then extract provider to use in
PyTableProvider::new(provider)?

Can I check how I can obtain the provider from pub struct PyCapsule(PyAny)?

@timsaucer

Any chance you can give me some code points or reference PRs that would help with implementation?

What if we just skip the whole creating a view as a table provider and instead go straight to registering a view on the session context?

We could do something like register_view(df: DataFrame) which would under the hood do exactly what you've got except not expose it back as a PyTableProvider and eliminate any possible confusion. Then we'd also save the user a step.

@matko would that solve your needs or do you need that view table provider exposed for other use?

Otherwise I think we have to plan for how we can have a common concept around two ways of creating table providers in python code. Also we would want to think about how we would handle the return type of a udtf, which we haven't even addressed.

skip the whole creating a view as a table provider and instead go straight to registering a view on the session context?

Sounds good.
Implemented.

timsaucer · 2025-02-15T19:29:05Z

src/dataframe.rs

@@ -156,6 +174,20 @@ impl PyDataFrame {
        PyArrowType(self.df.schema().into())
    }

+    /// Convert this DataFrame into a Table that can be used in register_table
+    fn _into_view(&self) -> PyDataFusionResult<PyTable> {


I recommend disabling that specific clippy warning.

timsaucer · 2025-02-15T19:29:50Z

README.md

+## Registering a DataFrame as a View
+
+You can use the `into_view` method to convert a DataFrame into a view and register it with the context.
+
+```python
+from datafusion import SessionContext, col, literal
+


I think this is very good, but would be more helpful if moved into the appropriate docs section so it goes into the online documentation rather than the readme.

I created a view.rst for this.

…ration

…r DataFrame registration

timsaucer

Looking good. I will try to run it later today.

timsaucer · 2025-03-07T13:13:15Z

src/dataframe.rs

+///
+/// # Methods
+///
+/// - `new`: Creates a new PyDataFrame.
+/// - `__getitem__`: Enable selection for `df[col]`, `df[col1, col2, col3]`, and `df[[col1, col2, col3]]`.
+/// - `__repr__`: Returns a string representation of the DataFrame.
+/// - `_repr_html_`: Returns an HTML representation of the DataFrame.
+/// - `describe`: Calculate summary statistics for a DataFrame.
+/// - `schema`: Returns the schema from the logical plan.
+/// - `into_view`: Convert this DataFrame into a Table that can be used in register_table. We have not finalized on PyTableProvider approach yet.
+/// - `select_columns`: Select columns from the DataFrame.
+/// - `select`: Select expressions from the DataFrame.
+/// - `drop`: Drop columns from the DataFrame.
+/// - `filter`: Filter the DataFrame based on a predicate.
+/// - `with_column`: Add a new column to the DataFrame.
+/// - `with_columns`: Add multiple new columns to the DataFrame.
+/// - `with_column_renamed`: Rename a column in the DataFrame.
+/// - `aggregate`: Aggregate the DataFrame based on group by and aggregation expressions.
+/// - `sort`: Sort the DataFrame based on expressions.
+/// - `limit`: Limit the number of rows in the DataFrame.
+/// - `collect`: Executes the plan, returning a list of `RecordBatch`es.


I don't think we need this whole section. The methods of the class should auto populate into the help and online documentation. Do you think this is necessary? It's another thing we'd have to manually maintain over time.

No, it's not necessary.
I deleted it.

kosiew added 8 commits February 6, 2025 12:59

add test_view

9d589a2

feat: add into_view method to register DataFrame as a view

648c185

add pytableprovider

e55ac9f

feat: add as_table method to PyTableProvider and update into_view to …

ca42449

…return PyTable

refactor: simplify as_table method and update documentation for into_…

d0c3163

…view

test: improve test_register_filtered_dataframe by removing redundant …

8578713

…comments and assertions

test: enhance test_register_filtered_dataframe with additional assert…

9cdd0dc

…ions for DataFrame results

ruff formatted

c207b6c

kosiew changed the title ~~View~~ Enable Dataframe to be converted into views which can be used in register_table Feb 6, 2025

cleanup: remove unused imports from test_view.py

20dbfe8

kosiew commented Feb 7, 2025

View reviewed changes

kosiew added 2 commits February 7, 2025 09:43

docs: add example for registering a DataFrame as a view in README.md

4b4c641

docs: update docstring for into_view method to clarify usage as ViewT…

12c4fe3

…able

kosiew marked this pull request as ready for review February 7, 2025 01:50

kosiew added 4 commits February 7, 2025 09:53

chore: add license header to test_view.py

15ead1f

ruff correction

48eb8db

refactor: rename into_view method to _into_view

f73eebb

ruff lint

6bba2e2

timsaucer reviewed Feb 15, 2025

View reviewed changes

kosiew added 5 commits February 18, 2025 12:41

refactor: simplify into_view method and update Rust binding convention

7b0cbf1

docs: add views section to user guide with example on registering views

f594b46

feat: add register_view method to SessionContext for DataFrame regist…

90a6a8b

…ration

Merge branch 'main' into view

c31395f

docs: update README and user guide to reflect register_view method fo…

f0837de

…r DataFrame registration

timsaucer reviewed Mar 7, 2025

View reviewed changes

docs: remove some documentation from PyDataFrame

9d8cdb5

timsaucer approved these changes Mar 8, 2025

View reviewed changes

timsaucer merged commit a80a788 into apache:main Mar 8, 2025
15 checks passed

Enable Dataframe to be converted into views which can be used in register_table #1016

Enable Dataframe to be converted into views which can be used in register_table #1016

Uh oh!

Conversation

kosiew commented Feb 6, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timsaucer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

timsaucer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!