GH-37710: [C++][Integration] Add C++ Utf8View implementation #37792

bkietz · 2023-09-19T17:50:37Z

Rationale for this change

After the PR changing the spec and schema ( #37526 ) is accepted, this PR will be undrafted. It adds the minimal addition of a C++ implementation and was extracted from the original C++ Utf8View pr ( #35628 ) for ease of review.

What changes are included in this PR?

The new types are available with new subclasses of DataType, Array, ArrayBuilder, ...
The values of string view arrays can be visited as std::string_view as with StringArray
String view arrays can be round tripped through IPC and integration JSON

Closes: [C++][Integration] Add C++ implementation of Utf8View #37710

Relevant mailing list discussions:

cpp/src/arrow/array/array_binary.h

cpp/src/arrow/array/builder_binary.cc

cpp/src/arrow/array/data.cc

cpp/src/arrow/array/validate.cc

cpp/src/arrow/builder.cc

cpp/src/arrow/type_traits.h

felipecrv · 2023-09-21T22:45:33Z

cpp/src/arrow/type_traits.h

+using is_binary_like_type = std::integral_constant<
+    bool, (is_base_binary_type<T>::value && !is_string_like_type<T>::value) ||
+              is_binary_view_type<T>::value || is_fixed_size_binary_type<T>::value>;


This changes the meaning of is_binary_like_type. For list-views I added new predicates and kept the existing predicates' semantics.

Illustrate in this graph https://gist.github.com/felipecrv/3c02f3784221d946dec1b031c6d400db

I agree with @felipecrv btw. This may silently produce incorrect binary-view handling in operations not touched by this PR.

These traits are used in very few places. I think the current arrangement is fine; "binary like" = has a byte array, "string like" = has a byte array which is utf8.

I've made a follow up to address this #38204

The question is whether these are used by third-party code. These are not internal APIs, and there is a risk that user code will then try to use binary view data in non-compatible routines, and get UB or data corruption.

I've reverted the change to is_binary_like_type and is_string_like_type

cpp/src/parquet/arrow/arrow_reader_writer_test.cc

pitrou · 2023-10-26T17:35:05Z

@bkietz Please make sure you rebase before merging, since these are rather large changes that could unwillingly affect other parts of the code.

paleolimbot · 2023-10-26T18:16:25Z

@assignUser It looks like at least one CI job is using the latest nightly a little too oportunistically and causing the CI to fail here! https://github.com/apache/arrow/actions/runs/6658040302/job/18094056322?pr=37792#step:6:21

paleolimbot · 2023-10-26T18:22:54Z

We'll add that to the PR cleaning up the edges on the (somewhat large) change to the R build system defaults. You can safely ignore that CI failure (or if you can't safely ignore it than I volunteer to fix it!)

bkietz · 2023-10-26T21:28:34Z

Thanks @paleolimbot ! I'll merge now

conbench-apache-arrow · 2023-10-27T18:37:12Z

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit f622139.

There were 2 benchmark results indicating a performance regression:

Commit Run on ursa-i9-9960x at 2023-10-26 22:16:07Z
- dataset-serialize (Python) with dataset=nyctaxi_multi_ipc_s3, format=csv, selectivity=10pc
- dataset-serialize (Python) with dataset=nyctaxi_multi_ipc_s3, format=csv, selectivity=100pc

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

alamb · 2023-10-27T20:46:33Z

Thank you for everyone who commented, reviewed and helped get this PR merged

This was introduced by apacheGH-37792.

### Rationale for this change We need explicit cast for `int64_t` to `size_t` conversion for i386 environment. This was introduced by GH-37792. ### What changes are included in this PR? Add explicit cast. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38556 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…he#38557) ### Rationale for this change We need explicit cast for `int64_t` to `size_t` conversion for i386 environment. This was introduced by apacheGH-37792. ### What changes are included in this PR? Add explicit cast. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38556 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…pache#37792) ### Rationale for this change After the PR changing the spec and schema ( apache#37526 ) is accepted, this PR will be undrafted. It adds the minimal addition of a C++ implementation and was extracted from the original C++ Utf8View pr ( apache#35628 ) for ease of review. ### What changes are included in this PR? - The new types are available with new subclasses of DataType, Array, ArrayBuilder, ... - The values of string view arrays can be visited as `std::string_view` as with StringArray - String view arrays can be round tripped through IPC and integration JSON * Closes: apache#37710 Relevant mailing list discussions: * https://lists.apache.org/thread/l8t1vj5x1wdf75mdw3wfjvnxrfy5xomy * https://lists.apache.org/thread/3qhkomvvc69v3gkotbwldyko7yk9cs9k Authored-by: Benjamin Kietzman <bengilgit@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>

…he#38557) ### Rationale for this change We need explicit cast for `int64_t` to `size_t` conversion for i386 environment. This was introduced by apacheGH-37792. ### What changes are included in this PR? Add explicit cast. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38556 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…pache#37792) ### Rationale for this change After the PR changing the spec and schema ( apache#37526 ) is accepted, this PR will be undrafted. It adds the minimal addition of a C++ implementation and was extracted from the original C++ Utf8View pr ( apache#35628 ) for ease of review. ### What changes are included in this PR? - The new types are available with new subclasses of DataType, Array, ArrayBuilder, ... - The values of string view arrays can be visited as `std::string_view` as with StringArray - String view arrays can be round tripped through IPC and integration JSON * Closes: apache#37710 Relevant mailing list discussions: * https://lists.apache.org/thread/l8t1vj5x1wdf75mdw3wfjvnxrfy5xomy * https://lists.apache.org/thread/3qhkomvvc69v3gkotbwldyko7yk9cs9k Authored-by: Benjamin Kietzman <bengilgit@gmail.com> Signed-off-by: Benjamin Kietzman <bengilgit@gmail.com>

…he#38557) ### Rationale for this change We need explicit cast for `int64_t` to `size_t` conversion for i386 environment. This was introduced by apacheGH-37792. ### What changes are included in this PR? Add explicit cast. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38556 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions bot added Component: Parquet Component: C++ Component: Documentation awaiting committer review Awaiting committer review labels Sep 19, 2023

bkietz force-pushed the 37710-cxx-impl-string-view branch 5 times, most recently from d468860 to 660008f Compare September 20, 2023 15:59

github-actions bot added the Component: Python label Sep 20, 2023

bkietz force-pushed the 37710-cxx-impl-string-view branch from 660008f to 614a411 Compare September 20, 2023 21:28

github-actions bot added the Component: R label Sep 20, 2023

bkietz force-pushed the 37710-cxx-impl-string-view branch from 614a411 to b62d539 Compare September 21, 2023 16:00

github-actions bot removed the Component: Documentation label Sep 21, 2023

bkietz marked this pull request as ready for review September 21, 2023 16:01

bkietz requested review from assignUser, kou, raulcd, paleolimbot, thisisnic, wgtmac and westonpace as code owners September 21, 2023 16:01

bkietz requested review from felipecrv and pitrou September 21, 2023 16:01

bkietz mentioned this pull request Sep 21, 2023

GH-35627: [C++][Format][Integration] Add string view to the arrow format #35628

Closed

felipecrv reviewed Sep 21, 2023

View reviewed changes

cpp/src/arrow/array/array_binary.h Outdated Show resolved Hide resolved

felipecrv reviewed Sep 21, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 22, 2023

bkietz mentioned this pull request Sep 22, 2023

[C++] [Parquet]: Add support for binary view and string view #37832

Open

bkietz added 5 commits October 26, 2023 14:05

apacheGH-37710: [C++][Integration] Add C++ Utf8View implementation

2e8f25d

review comments

0e14114

concatenate should ensure null slots contain an empty string view

ce925f6

correct value count

02d3385

revert binary_like, review comments

028bca2

bkietz force-pushed the 37710-cxx-impl-string-view branch from 2798197 to 028bca2 Compare October 26, 2023 18:06

paleolimbot mentioned this pull request Oct 26, 2023

[R] Add test for new nixlibs.R #38430

Closed

bkietz merged commit f622139 into apache:main Oct 26, 2023
32 of 35 checks passed

bkietz removed the awaiting change review Awaiting change review label Oct 26, 2023

bkietz deleted the 37710-cxx-impl-string-view branch October 26, 2023 21:29

kou mentioned this pull request Nov 2, 2023

[CI][C++] test-debian-11-cpp-i386 failed by long long int to size_t conversion #38556

Closed

kou added a commit to kou/arrow that referenced this pull request Nov 2, 2023

apacheGH-38556: [C++] Add missing explicit int64_t cast for i386

fc3ea1c

This was introduced by apacheGH-37792.

kou mentioned this pull request Nov 2, 2023

GH-38556: [C++] Add missing explicit size_t cast for i386 #38557

Merged

kou added a commit to kou/arrow that referenced this pull request Nov 2, 2023

apacheGH-38556: [C++] Add missing explicit int64_t cast for i386

f1bc62c

This was introduced by apacheGH-37792.

bkietz mentioned this pull request Nov 14, 2023

[Go][Integration] Add support for Utf8View to the go implementation #38718

Closed

urvishdesai mentioned this pull request Nov 28, 2023

Support for Utf8View in the Rust library jorgecarleitao/arrow2#1596

Open

This was referenced Jan 16, 2024

[Python] Support Binary/StringView in PyArrow #39633

Open

[C++] Implement basic functionality for new Binary/Utf8View types #39634

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-37710: [C++][Integration] Add C++ Utf8View implementation #37792

GH-37710: [C++][Integration] Add C++ Utf8View implementation #37792

bkietz commented Sep 19, 2023 •

edited by alamb

felipecrv Sep 21, 2023

pitrou Oct 10, 2023

bkietz Oct 10, 2023

bkietz Oct 11, 2023

pitrou Oct 25, 2023

bkietz Oct 26, 2023

pitrou commented Oct 26, 2023

paleolimbot commented Oct 26, 2023

paleolimbot commented Oct 26, 2023

bkietz commented Oct 26, 2023

conbench-apache-arrow bot commented Oct 27, 2023

alamb commented Oct 27, 2023

GH-37710: [C++][Integration] Add C++ Utf8View implementation #37792

GH-37710: [C++][Integration] Add C++ Utf8View implementation #37792

Conversation

bkietz commented Sep 19, 2023 • edited by alamb

Rationale for this change

What changes are included in this PR?

felipecrv Sep 21, 2023

Choose a reason for hiding this comment

pitrou Oct 10, 2023

Choose a reason for hiding this comment

bkietz Oct 10, 2023

Choose a reason for hiding this comment

bkietz Oct 11, 2023

Choose a reason for hiding this comment

pitrou Oct 25, 2023

Choose a reason for hiding this comment

bkietz Oct 26, 2023

Choose a reason for hiding this comment

pitrou commented Oct 26, 2023

paleolimbot commented Oct 26, 2023

paleolimbot commented Oct 26, 2023

bkietz commented Oct 26, 2023

conbench-apache-arrow bot commented Oct 27, 2023

alamb commented Oct 27, 2023

bkietz commented Sep 19, 2023 •

edited by alamb