arrow::Schema and DFSchema difference

[mustafasrepo]

In current codebase some places use arrow::Schema for its schema, some other places use DFSchema. These structs are quite similar, also there are some places where schema is converted from one type to other (such as create_memory_table inside context.rs).

I want to know more about difference between these structs, and the rationale for choosing in between them.

[alamb]

The difference is that DFSchema can contain a "relation name" -- aka a table name whereas a Schema only has a single string name

DF schema is currently used to know what table the different fields come from

The way they are implemented is pretty bad (I think I did the initial version 😢 ) -- if I were to do it again I would implement it as a wrapper:

struct DFSchema {
  relation_name: Arc<str>,
  inner: SchemaRef
}

Which would make conversion easier and more efficient I think

[phillipleblanc]

For anyone stumbling across this discussion like I did, this has been refactored in #9595

[mustafasrepo]

Thanks for the answer