Skip to content

Support for automatic replacement scans #1244

@kylebarron

Description

@kylebarron

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

DuckDB has a replacement scan feature in the Python API that allows it to directly access Python variables that are Arrow-based.

This would mean that you wouldn't necessarily need to perform a register_dataframe step in advance; you could just query a variable directly. This would also be temporary; it wouldn't persist the dataframe as a registered object in the session.

Describe the solution you'd like

If a table name is not found among the registered

Is there some way to implement an order of operations for finding tables with a certain name? I.e. inject some code so that if a table x isn't found already-registered, then it can evaluate whether the Python variable x is Arrow-like?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

I could implement the Python side of this, but I don't know the DataFusion API well enough to know where to find the hooks or how to connect to them.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions