Support for ZLIB compression in ORC writer by vuule · Pull Request #11036 · NVIDIA/cudf

vuule · 2022-06-03T01:46:51Z

Expands the nvcomp adapter to cover compression. Supports SNAPPY and ZLIB.
Moves the ORC writer nvcomp compression from stripe_enc.cu to the adapter and add ZLIB support.
Adds padding to compressed blocks in writer to ensure alignment of output pointer alignment that is required for nvcomp ZLIB compression.

Minor changes:

Make page/block/chunk naming consistent in the nvcomp adapter -> always use chunk (to match nvcomp).
Rename members of batched_args so it also makes sense for compression.

…fea-ocr-writer-zlib

codecov · 2022-06-03T23:16:41Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.08@b7c7adc). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-22.08   #11036   +/-   ##
===============================================
  Coverage                ?   86.34%           
===============================================
  Files                   ?      144           
  Lines                   ?    22738           
  Branches                ?        0           
===============================================
  Hits                    ?    19632           
  Misses                  ?     3106           
  Partials                ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b7c7adc...6945584. Read the comment docs.

vuule · 2022-06-04T03:52:30Z

                       : src_len;
      uint32_t blk_size24{};
-      if (dst_len >= src_len) {
+      if (statuses[ss.first_block + b].status == 0) {


Previous condition was implicit, led to wrong output when the compressed data is larger.

karthikeyann · 2022-06-14T20:57:06Z

I see 97 modified files that includes already merged changes as well. is this a merge issue or a github issue?

karthikeyann · 2022-06-15T04:10:51Z

I see 97 modified files that includes already merged changes as well. is this a merge issue or a github issue?

Fixed this issue when I switched to 22.06 and back to 22.08 as base. Also updated reviewers.

Co-authored-by: Karthikeyan <6488848+karthikeyann@users.noreply.github.com>

…nto fea-orc-writer-zlib

karthikeyann

LGTM 👍

vuule · 2022-06-15T19:28:40Z

rerun tests

galipremsagar · 2022-06-15T20:59:19Z

rerun tests

vuule · 2022-06-15T23:16:01Z

@gpucibot merge

Some recently merged PRs (#11018 + #11036) do not include enough header which may cause compile error in some systems (in particular, CUDA 11.7 + gcc-11.2). This PR adds the missing header (`<optional>`) to fix the compile issue. Authors: - Nghia Truong (https://github.com/ttnghia) Approvers: - Karthikeyan (https://github.com/karthikeyann) - Yunsong Wang (https://github.com/PointKernel) URL: #11126

vuule added 4 commits June 2, 2022 13:11

ORC writer ZLIB support POC

542dd38

Merge branch 'branch-22.08' of https://github.com/rapidsai/cudf into …

2146779

…fea-ocr-writer-zlib

Merge branch 'branch-22.08' of https://github.com/rapidsai/cudf into …

a518c6c

…fea-ocr-writer-zlib

padding; bug fixes

9966bb4

vuule added feature request New feature or request cuIO cuIO issue non-breaking Non-breaking change labels Jun 3, 2022

vuule self-assigned this Jun 3, 2022

github-actions Bot added Python Affects Python cuDF API. libcudf Affects libcudf (C++/CUDA) code. labels Jun 3, 2022

vuule added 4 commits June 2, 2022 20:52

implement nvcomp adapter for max_output_chunk_size

8ce06a3

partial adapter for batched compression

540657b

finish compression adapter

ae3458b

rename batched_args to make it applicable to compression

74f8ba1

vuule added 2 commits June 3, 2022 17:12

docs; consistent naming

927d86f

revert accidental change

7617fa8

vuule commented Jun 4, 2022

View reviewed changes

Comment thread python/cudf/cudf/_lib/cpp/io/types.pxd

vuule added 3 commits June 3, 2022 21:25

use different block size between cmpression types

ae4e86b

fix max ZLIB block size

e17281c

python test

15718f2

vuule marked this pull request as ready for review June 6, 2022 18:22

vuule requested review from a team as code owners June 6, 2022 18:22

vuule requested review from galipremsagar, isVoid, karthikeyann and robertmaynard June 6, 2022 18:22

vuule mentioned this pull request Jun 7, 2022

Use new nvCOMP API to optimize the decompression temp memory size #11064

Merged

karthikeyann changed the base branch from branch-22.08 to branch-22.06 June 15, 2022 04:08

karthikeyann requested review from a team as code owners June 15, 2022 04:08

karthikeyann changed the base branch from branch-22.06 to branch-22.08 June 15, 2022 04:08

karthikeyann requested review from a team and removed request for a team June 15, 2022 04:12

karthikeyann reviewed Jun 15, 2022

View reviewed changes

Comment thread python/cudf/cudf/tests/test_orc.py

Comment thread cpp/src/io/orc/writer_impl.cu Outdated

Comment thread cpp/src/io/comp/nvcomp_adapter.cu Outdated

Comment thread cpp/src/io/comp/nvcomp_adapter.cpp

use auto

1ac8815

Co-authored-by: Karthikeyan <6488848+karthikeyann@users.noreply.github.com>

github-actions Bot removed gpuCI CMake CMake build issue Java Affects Java cuDF API. labels Jun 15, 2022

vuule added 2 commits June 15, 2022 05:35

constexpr; missing switch cases

2be0095

Merge branch 'fea-orc-writer-zlib' of https://github.com/vuule/cudf i…

6945584

…nto fea-orc-writer-zlib

vuule requested a review from karthikeyann June 15, 2022 12:38

karthikeyann approved these changes Jun 15, 2022

View reviewed changes

vuule added the 5 - Ready to Merge Testing and reviews complete, ready to merge label Jun 15, 2022

isVoid removed their request for review June 15, 2022 17:16

rapids-bot Bot merged commit 10cf648 into NVIDIA:branch-22.08 Jun 15, 2022

vuule deleted the fea-orc-writer-zlib branch June 15, 2022 23:16

ttnghia mentioned this pull request Jun 20, 2022

Fix compile error due to missing header #11126

Merged

Navigation Menu

Uh oh!

Support for ZLIB compression in ORC writer - #11036

Support for ZLIB compression in ORC writer#11036
rapids-bot[bot] merged 19 commits into
NVIDIA:branch-22.08from
vuule:fea-orc-writer-zlib

vuule commented Jun 3, 2022 •

edited

Loading

Uh oh!

codecov Bot commented Jun 3, 2022 •

edited

Loading

Uh oh!

vuule Jun 4, 2022

Uh oh!

Uh oh!

karthikeyann commented Jun 14, 2022

Uh oh!

karthikeyann commented Jun 15, 2022 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karthikeyann left a comment

Uh oh!

vuule commented Jun 15, 2022

Uh oh!

galipremsagar commented Jun 15, 2022

Uh oh!

vuule commented Jun 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

vuule commented Jun 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vuule Jun 4, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karthikeyann commented Jun 14, 2022

Uh oh!

karthikeyann commented Jun 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

karthikeyann left a comment

Choose a reason for hiding this comment

Uh oh!

vuule commented Jun 15, 2022

Uh oh!

galipremsagar commented Jun 15, 2022

Uh oh!

vuule commented Jun 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vuule commented Jun 3, 2022 •

edited

Loading

codecov Bot commented Jun 3, 2022 •

edited

Loading

karthikeyann commented Jun 15, 2022 •

edited

Loading