feat: Use NativeType in get_example_type, information schema#21737
feat: Use NativeType in get_example_type, information schema#21737theirix wants to merge 7 commits intoapache:mainfrom
Conversation
github-actions
Bot
added
logical-expr
|
|
||
| #[deprecated(since = "46.0.0", note = "See get_example_types instead")] | ||
| pub fn get_possible_types(&self) -> Vec<Vec<DataType>> { | ||
| pub fn get_possible_types(&self) -> Vec<Vec<NativeType>> { |
There was a problem hiding this comment.
Hm. A public API break for a deprecated method ... Maybe it is time to just remove it ?!
There was a problem hiding this comment.
That's true. I will drop get_possible_types. However, get_example_types is also public - it is used in the information schema internally, and can be used by DF users.
Since it breaks the expr-common API, how about deprecating the get_example_types signature and adding a NativeType-based get_representative_types?
There was a problem hiding this comment.
@martin-g to avoid breaking changes in the get_example_types (my miss), I've introduced the new method, while deprecating get_example_types.
The Signature API is heavily DataType-based, so the get_example_types. Should we let them coexist, or even move the NativeType-based logic to the information schema? cc @jayzhan211
There was a problem hiding this comment.
It seems the NativeType is not powerful enough to fully replace DataType here.
For example https://github.com/apache/datafusion/pull/21737/changes#diff-52d29120f24c2a01793f6d729fdb7898abaa8d2320db75aba5cbb06cd714930eR505-R516 shows that there are no counterparts for Union's UnionMode and Map's keys_sorted and defaults should be used. And this may lead to confusions.
There was a problem hiding this comment.
Undoubtedly, it's not a replacement now. The question for this improvement is whether we'd like to provide changes only to the information-schema, which benefits from native types (then we move code there), or also provide a public API for the Signature to support NativeType alongside DataType (as now).
At the same time, UDFs are migrating to TypeSignature, abstracting from physical data types.
Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>
Instead, add the new `get_representative_types` API with NativeType. Deprecated old `get_example_types` and related helpers, left as-is to avoid breaking API change.
| /// | ||
| /// This is used for `information_schema` and can be used to generate | ||
| /// documentation or error messages. | ||
| /// Remove with `get_example_types` |
There was a problem hiding this comment.
This should be a comment (//) instead of rustdoc (///)
|
|
||
| #[deprecated(since = "46.0.0", note = "See get_example_types instead")] | ||
| pub fn get_possible_types(&self) -> Vec<Vec<DataType>> { | ||
| pub fn get_possible_types(&self) -> Vec<Vec<NativeType>> { |
There was a problem hiding this comment.
It seems the NativeType is not powerful enough to fully replace DataType here.
For example https://github.com/apache/datafusion/pull/21737/changes#diff-52d29120f24c2a01793f6d729fdb7898abaa8d2320db75aba5cbb06cd714930eR505-R516 shows that there are no counterparts for Union's UnionMode and Map's keys_sorted and defaults should be used. And this may lead to confusions.
2 participants
Which issue does this PR close?
NativeTypeinstead ofDataTypeforget_example_types#14761.Rationale for this change
Moving from physical types:
get_example_typesand the information schema use ArrowDataType, but it is sufficient to use Datafusion'sNativeTypeinstead.Let's introduce a new API
get_representative_types, based onNativeType, and deprecate publicget_example_typesAPIIt is a logical continuation of #15965
What changes are included in this PR?
get_representative_typesto provideNativeTypeNUMERICSfinally (a brush-up for Refactor away usage ofNUMERICS/INTEGERSindatafusion/expr-common/src/type_coercion/aggregates.rs#18092)get_data_typeshelperget_data_type_for_schemato resolve a NativeType back to the most appropriate DataType to show in schemaStruct,Union,Mapto this helperAre these changes tested?
Tests are passing
Are there any user-facing changes?
TypeSignature::get_example_typesAPITypeSignature::get_possible_types(since v46)