Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[C#] Implement C Data Interface for C# #33856

Closed
1 of 3 tasks
wjones127 opened this issue Jan 25, 2023 · 0 comments · Fixed by #34133 or #35496
Closed
1 of 3 tasks

[C#] Implement C Data Interface for C# #33856

wjones127 opened this issue Jan 25, 2023 · 0 comments · Fixed by #34133 or #35496
Assignees
@wjones127
Labels
Component: C# Type: enhancement
Milestone
13.0.0

Comments

@wjones127
Copy link
Member

wjones127 commented Jan 25, 2023 •
edited
Loading

Describe the enhancement requested

C# is one of the few independent implementations of Arrow that doesn't support the C Data Interface.

Supporting this would allow the ParquetSharp library, which wraps the Parquet C++ implementation, to produce and consume Arrow arrays and record batches.

Subtasks

Component(s)

C#
@wjones127 wjones127 mentioned this issue Jan 25, 2023
Closed
@github-actions github-actions bot mentioned this issue Feb 10, 2023
Merged
eerhardt pushed a commit that referenced this issue Apr 4, 2023
@wjones127 @westonpace
GH-34737: [C#] C Data interface for schemas and types (#34133)
f02d351 
### Rationale for this change

This starts the C Data Interface implementation for C# with integration for `ArrowSchema`. `ArrowArray` will come in a follow-up PR.

### What changes are included in this PR?

* Adds classes `CArrowSchema` and `ImportedArrowSchema` which allow interacting with the `CArrowSchema`.
* Adds integration tests with PyArrow, inspired by the similar integration tests in [arrow-rs](https://github.com/apache/arrow-rs/blob/master/arrow/src/pyarrow.rs)

### Are these changes tested?

Yes, the PyArrow integration tests validate the functionality.

### Are there any user-facing changes?

This only adds new APIs, and doesn't change any existing ones.

* Closes: #33856
* Closes: #34737

Lead-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Eric Erhardt <eric.erhardt@microsoft.com>
@wjones127 wjones127 reopened this Apr 4, 2023
@CurtHagenlocher CurtHagenlocher mentioned this issue May 8, 2023
Merged
ArgusLi pushed a commit to Bit-Quill/arrow that referenced this issue May 15, 2023
@wjones127 @westonpace @ArgusLi
apacheGH-34737: [C#] C Data interface for schemas and types (apache#3…
5dbeb4f 
…4133)

### Rationale for this change

This starts the C Data Interface implementation for C# with integration for `ArrowSchema`. `ArrowArray` will come in a follow-up PR.

### What changes are included in this PR?

* Adds classes `CArrowSchema` and `ImportedArrowSchema` which allow interacting with the `CArrowSchema`.
* Adds integration tests with PyArrow, inspired by the similar integration tests in [arrow-rs](https://github.com/apache/arrow-rs/blob/master/arrow/src/pyarrow.rs)

### Are these changes tested?

Yes, the PyArrow integration tests validate the functionality.

### Are there any user-facing changes?

This only adds new APIs, and doesn't change any existing ones.

* Closes: apache#33856
* Closes: apache#34737

Lead-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Eric Erhardt <eric.erhardt@microsoft.com>
rtpsw pushed a commit to rtpsw/arrow that referenced this issue May 16, 2023
@wjones127 @westonpace @rtpsw
apacheGH-34737: [C#] C Data interface for schemas and types (apache#3…
fa9e04b 
…4133)

### Rationale for this change

This starts the C Data Interface implementation for C# with integration for `ArrowSchema`. `ArrowArray` will come in a follow-up PR.

### What changes are included in this PR?

* Adds classes `CArrowSchema` and `ImportedArrowSchema` which allow interacting with the `CArrowSchema`.
* Adds integration tests with PyArrow, inspired by the similar integration tests in [arrow-rs](https://github.com/apache/arrow-rs/blob/master/arrow/src/pyarrow.rs)

### Are these changes tested?

Yes, the PyArrow integration tests validate the functionality.

### Are there any user-facing changes?

This only adds new APIs, and doesn't change any existing ones.

* Closes: apache#33856
* Closes: apache#34737

Lead-authored-by: Will Jones <willjones127@gmail.com>
Co-authored-by: Weston Pace <weston.pace@gmail.com>
Signed-off-by: Eric Erhardt <eric.erhardt@microsoft.com>
eerhardt pushed a commit that referenced this issue May 22, 2023
@CurtHagenlocher
GH-33856: [C#] Implement C Data Interface for C# (#35496)
0dca449 
### Rationale for this change

This continues implementing the C Data Interface for C# with integration for `ArrowArray`, `RecordBatch` and streams.

### What changes are included in this PR?

- Adds classes `CArrowArray` and `CArrowStream` to represent the C API structures.
- Adds interface `IArrowArrayStream` to represent an array stream or record batch reader.
- Adds classes `CArrowArrayImporter`, `CArrowArrayExporter`, `CArrowArrayStreamImporter` and `CArrowArrayExporter` to marshal between C# and C representations.
- Augments the native memory representation to support (reasonably safe) ownership of memory by external code.

### Are these changes tested?

Yes. Testing is largely done via the Python C API interface.

### Are there any user-facing changes?

Yes, this adds new user-facing APIs to import and export C# structures using the C API.

**This PR includes breaking changes to public APIs.**

The default time unit for Time64Type was previously milliseconds. This does not appear to be valid, so it has been changed to nanoseconds.

- Closes: #33856
- Closes: #33857
* Closes: #33856

Authored-by: Curt Hagenlocher <curt@hagenlocher.org>
Signed-off-by: Eric Erhardt <eric.erhardt@microsoft.com>
@eerhardt eerhardt added this to the 13.0.0 milestone May 22, 2023
@adamreeve adamreeve mentioned this issue Jun 16, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wjones127 wjones127

Labels
Component: C# Type: enhancement
Projects
None yet
Milestone
13.0.0
2 participants
@wjones127 @eerhardt