GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

rok · 2022-10-07T04:25:43Z

This is to add DELTA_BYTE_ARRAY encoder.

Closes: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

github-actions · 2022-10-07T04:26:03Z

https://issues.apache.org/jira/browse/ARROW-17619

rok · 2022-10-07T04:27:12Z

This is not really review ready yet and needs #14191 and #14293 to merge first.

rok · 2023-01-26T12:35:54Z

note: this should support BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY. PARQUET-2231

github-actions · 2023-02-23T14:15:35Z

Closes: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

mapleFU · 2023-02-28T05:34:01Z

(Can this patch go ahead now?)

mapleFU

Though this patch is still draft, I run it for fun. Seems that here is some issue. May that helps

cpp/src/parquet/encoding.cc

…tor APIs

pitrou · 2023-08-21T17:02:18Z

I'm going to merge this now.

mapleFU · 2023-08-21T17:12:13Z

Thats a long time, bravo!

rok · 2023-08-21T19:25:04Z

Thanks all for helping this along! I'm very happy we got this in!

anjakefala · 2023-08-21T19:51:46Z

Congratulations, @rok!!!

AlenkaF · 2023-08-22T04:21:38Z

Great work everybody, congrats!!

conbench-apache-arrow · 2023-08-22T04:26:14Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 94bd0d2.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

jorisvandenbossche · 2023-08-22T10:00:35Z

We should probably update the python docstring as well:

arrow/python/pyarrow/parquet/core.py

Lines 822 to 827 in fe750ed

    
           column_encoding : string or dict, default None 
        
               Specify the encoding scheme on a per column basis. 
        
               Currently supported values: {'PLAIN', 'BYTE_STREAM_SPLIT'}. 
        
               Certain encodings are only compatible with certain data types. 
        
               Please refer to the encodings section of `Reading and writing Parquet 
        
               files <https://arrow.apache.org/docs/cpp/parquet.html#encodings>`_.

(which was clearly already outdated before this PR as well! ;))

pitrou · 2023-08-22T10:01:24Z

Feel free to open a PR :-)

rok · 2023-08-22T16:13:56Z

I created an issue :) #37312

mapleFU · 2023-11-08T15:00:39Z

cpp/src/parquet/encoding.cc

-            RETURN_NOT_OK(helper.builder->ReserveData(
-                std::min<int64_t>(len_, helper.chunk_space_remaining)));


Why previously here ReserveData for min(len_, helper.chunk_space_remaining) here, wouldn't len_ be too large @pitrou

I don't understand: are you commented on the removed code? I'd rather not try to understand code that was removed months ago...

See #38437 for context, where this code is being added back partially

I also don't understand them T_T.

Just confused by this change, so I tried to understand the origin code and find out why this cause the regression, what should I do to fix it

…t writer (apache#14341) This is to add DELTA_BYTE_ARRAY encoder. * Closes: apache#32863 Lead-authored-by: Rok Mihevc <rok@mihevc.org> Co-authored-by: Rok <rok@mihevc.org> Co-authored-by: Antoine Pitrou <antoine@python.org> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Co-authored-by: Gang Wu <ustcwg@gmail.com> Co-authored-by: mwish <1506118561@qq.com> Co-authored-by: Will Jones <willjones127@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

github-actions bot added Component: Documentation Component: C++ Component: Python Component: Parquet labels Oct 7, 2022

rok force-pushed the ARROW-17619 branch from 5ddafef to 8bdb4b9 Compare November 12, 2022 02:16

rok force-pushed the ARROW-17619 branch 2 times, most recently from 37077b0 to 8448815 Compare November 25, 2022 15:27

rok force-pushed the ARROW-17619 branch from 8448815 to 6b0761b Compare December 15, 2022 12:12

asfimport mentioned this pull request Jan 5, 2023

[C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #32863

Closed

rok force-pushed the ARROW-17619 branch from 6b0761b to 2cfa3cc Compare February 10, 2023 19:43

rok force-pushed the ARROW-17619 branch from 2cfa3cc to 7f33ecc Compare February 23, 2023 14:13

github-actions bot removed the Component: Documentation label Feb 23, 2023

rok changed the title ~~ARROW-17619: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer~~ GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer Feb 23, 2023

mapleFU reviewed Mar 1, 2023

View reviewed changes

cpp/src/parquet/encoding.cc Outdated Show resolved Hide resolved

cpp/src/parquet/encoding.cc Outdated Show resolved Hide resolved

github-actions bot added awaiting review Awaiting review awaiting committer review Awaiting committer review awaiting changes Awaiting changes and removed awaiting review Awaiting review awaiting committer review Awaiting committer review labels Mar 1, 2023

rok force-pushed the ARROW-17619 branch from 7f33ecc to 75c2fd3 Compare March 1, 2023 17:42

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 1, 2023

rok force-pushed the ARROW-17619 branch from 6062c17 to b677934 Compare March 1, 2023 19:19

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Mar 1, 2023

rok added 4 commits August 21, 2023 16:00

Review feedback

0aaa6b4

Review feedback

9f0cdbb

Review feedback

46aa303

Review feedback

acc40ed

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Aug 21, 2023

Rewrite Binary/FLBA specializations to correctly account for Accumula…

5039cf9

…tor APIs

pitrou force-pushed the ARROW-17619 branch from cbf01b6 to 5039cf9 Compare August 21, 2023 15:38

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 21, 2023

pitrou approved these changes Aug 21, 2023

View reviewed changes

pitrou merged commit 94bd0d2 into apache:main Aug 21, 2023
30 of 33 checks passed

pitrou removed the awaiting change review Awaiting change review label Aug 21, 2023

github-actions bot added the awaiting committer review Awaiting committer review label Aug 21, 2023

mapleFU mentioned this pull request Aug 22, 2023

[C++][Parquet] Encoding: Add Benchmark for DELTA_BYTE_ARRAY #37293

Closed

rok mentioned this pull request Aug 22, 2023

[Python][Docs] Update Python docstrings to reflect new parquet encoding option #37312

Closed

jorisvandenbossche mentioned this pull request Oct 24, 2023

[C++] Parquet reading performance regressions #38432

Closed

mapleFU reviewed Nov 8, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Nov 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

rok commented Oct 7, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 7, 2022

rok commented Oct 7, 2022

rok commented Jan 26, 2023

github-actions bot commented Feb 23, 2023

mapleFU commented Feb 28, 2023

mapleFU left a comment

pitrou commented Aug 21, 2023

mapleFU commented Aug 21, 2023

rok commented Aug 21, 2023

anjakefala commented Aug 21, 2023

AlenkaF commented Aug 22, 2023

conbench-apache-arrow bot commented Aug 22, 2023

jorisvandenbossche commented Aug 22, 2023

pitrou commented Aug 22, 2023

rok commented Aug 22, 2023

mapleFU Nov 8, 2023

pitrou Nov 8, 2023

jorisvandenbossche Nov 8, 2023 •

edited

Loading

mapleFU Nov 8, 2023

		RETURN_NOT_OK(helper.builder->ReserveData(
		std::min<int64_t>(len_, helper.chunk_space_remaining)));

GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

GH-32863: [C++][Parquet] Add DELTA_BYTE_ARRAY encoder to Parquet writer #14341

Conversation

rok commented Oct 7, 2022 • edited by github-actions bot Loading

github-actions bot commented Oct 7, 2022

rok commented Oct 7, 2022

rok commented Jan 26, 2023

github-actions bot commented Feb 23, 2023

mapleFU commented Feb 28, 2023

mapleFU left a comment

Choose a reason for hiding this comment

pitrou commented Aug 21, 2023

mapleFU commented Aug 21, 2023

rok commented Aug 21, 2023

anjakefala commented Aug 21, 2023

AlenkaF commented Aug 22, 2023

conbench-apache-arrow bot commented Aug 22, 2023

jorisvandenbossche commented Aug 22, 2023

pitrou commented Aug 22, 2023

rok commented Aug 22, 2023

mapleFU Nov 8, 2023

Choose a reason for hiding this comment

pitrou Nov 8, 2023

Choose a reason for hiding this comment

jorisvandenbossche Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

mapleFU Nov 8, 2023

Choose a reason for hiding this comment

rok commented Oct 7, 2022 •

edited by github-actions bot

Loading

jorisvandenbossche Nov 8, 2023 •

edited

Loading