ARROW-10368 [Rust] [DataFusion] Refactor scan nodes to allow extensions

[rdettai]

Replace all specific XxxScan nodes with a generic SourceScan that dynamically dispatches to any source implementation.

The main goal is to make implementations of custom data sources possible (read from S3, custom file formats...). It might also make the LogicalPlan enum a bit more readable.

Initial discussion: https://issues.apache.org/jira/browse/ARROW-10368

[github-actions]

https://issues.apache.org/jira/browse/ARROW-10368

[rdettai]

@jorgecarleitao I have implemented the basic structure, with a SourceScanner trait proposal. I have shown how the SourceScan logical plan is mapped to an execution by just passing through the source implementation. This basically means that there is no "real" conversion from logical to physical plan, we are just passing the same implem from one to the other.

If this is "conceptually" okay with you, I will port all the currently implemented sources to this model.

[rdettai]

I just found out something! It seems that datafusion is full of interesting mysteries 😄 ! You actually already have the abstractions required to do what I want, but they are a little hidden.

With ExecutionContext::register_table(&mut self, name: &str, provider: Box<dyn TableProvider>) you can actually already directly register an implementation of a custom source exec. You have to implement the TableProvider trait that allows you to have the projection pushdown, then directly the ExecutionPlan trait. You can then run your queries on it, and you can even use the new source from the dataframe API with ExecutionContext::table(&mut self, table_name: &str) -> Result<Arc<dyn DataFrame>>.

Isn't that wonderful ?

Now the only thing that remains to be done I guess is:

• add a commodity function like ExecutionContext::read_provider(&mut self, provider: Box<dyn TableProvider>) -> Result<Arc<dyn DataFrame>> that shortcuts the two calls mentioned above. This is mainly meant to make this feature more explicit.
• add a new example ?
• enjoy... ❤️