Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C++] Expose schema in named table provider #34278

Closed
icexelloss opened this issue Feb 21, 2023 · 2 comments
Closed

[C++] Expose schema in named table provider #34278

icexelloss opened this issue Feb 21, 2023 · 2 comments
Assignees
@icexelloss
Labels
Component: C++ Type: enhancement
Milestone
12.0.0

Comments

@icexelloss
Copy link
Contributor

Describe the enhancement requested

Currently, the named table provider takes "names" as input:

arrow/cpp/src/arrow/engine/substrait/options.h

Line 68 in 11ae62a

using NamedTableProvider =

In some cases, it is useful for the implementer of the provider to have access to the schema as well.
One case we have currently is that we are using named table to read from a custom remote storage and the file schema in the storage doesn't match 100% with the node schema (e.g., int32 vs int64 difference). If we have the node schema, we can add cast operation in the named table provider to avoid downstream schema/data mismatching issue.

Component(s)

C++
@icexelloss
Copy link
Contributor Author

take

icexelloss added a commit to icexelloss/arrow that referenced this issue Feb 21, 2023
@icexelloss
apacheGH-34278: [C++] Expose schema in named table provider
97c8776
@github-actions github-actions bot mentioned this issue Feb 21, 2023
Merged
icexelloss added a commit to icexelloss/arrow that referenced this issue Feb 21, 2023
@icexelloss
apacheGH-34278: [C++] Expose schema in named table provider
810a027
westonpace pushed a commit that referenced this issue Feb 22, 2023
@icexelloss
GH-34278: [C++] Expose schema in named table provider (#34279)
12c2b38 
### Rationale for this change

#34278
### What changes are included in this PR?
This PR adds an parameter "schema" in the `NamedTableProvider` interface

### Are these changes tested?
Added test to check schema in named table provider

### Are there any user-facing changes?
This is a developer API change.

Authored-by: Li Jin <ice.xelloss@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
@westonpace westonpace added this to the 12.0.0 milestone Feb 22, 2023
@westonpace
Copy link
Member

Issue resolved by pull request 34279
#34279

fatemehp pushed a commit to fatemehp/arrow that referenced this issue Feb 24, 2023
@icexelloss @fatemehp
apacheGH-34278: [C++] Expose schema in named table provider (apache#3…
02ce7ba 
…4279)

### Rationale for this change

apache#34278
### What changes are included in this PR?
This PR adds an parameter "schema" in the `NamedTableProvider` interface

### Are these changes tested?
Added test to check schema in named table provider

### Are there any user-facing changes?
This is a developer API change.

Authored-by: Li Jin <ice.xelloss@gmail.com>
Signed-off-by: Weston Pace <weston.pace@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@icexelloss icexelloss

Labels
Component: C++ Type: enhancement
Projects
None yet
Milestone
12.0.0
Development

No branches or pull requests
2 participants
@icexelloss @westonpace