Add `read_orc_metadata` to libcudf by vuule · Pull Request #11815 · NVIDIA/cudf

vuule · 2022-09-29T00:56:25Z

Description

Adds a C++ interface to get information about an ORC file. It is meant to be an efficient way to get information like column names and types, as well as file structure (e.g. number of stripes). The returned structure can be expanded to include more types of metadata, for now it only returns info that we found relevant internally.

The returned column hierarchy matches the one used in ORC (i.e. root struct column included), not the hierarchy of a cuDF dataframe that the file would be read as (root column children become top level cuDF columns).

This PR also includes improvements to ORC reader benchmarks, enabled by the new metadata API.

Checklist

I am familiar with the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

…fea-read_orc_metadata

vuule · 2022-09-29T01:03:39Z

Cool aside and an argument against num_rows - read_orc benchmark results :)
(with NROWS and STRIPES file is read in 8 equal-ish chunks)

| column_selection | row_selection | uses_index | uses_numpy_dtype | timestamp_type | Samples |  CPU Time  | Noise |  GPU Time  | Noise | bytes_per_second | peak_memory_usage | encoded_file_size |
|------------------|---------------|------------|------------------|----------------|---------|------------|-------|------------|-------|------------------|-------------------|-------------------|
|              ALL |           ALL |        YES |              YES |          EMPTY |     87x | 173.101 ms | 4.27% | 173.098 ms | 4.27% |       3101548373 |       771.392 MiB |       175.078 MiB |
|              ALL |       STRIPES |        YES |              YES |          EMPTY |     74x | 205.027 ms | 3.95% | 205.015 ms | 3.95% |       2618687346 |       104.208 MiB |       175.078 MiB |
|              ALL |         NROWS |        YES |              YES |          EMPTY |     26x | 581.968 ms | 1.59% | 581.961 ms | 1.59% |        922519967 |       117.808 MiB |       175.078 MiB |

hyperbolic2346

Looking good to me, some minor nits.

hyperbolic2346 · 2022-10-03T15:47:16Z

  cudf::rmm_pool_raii rmm_pool;

-  auto constexpr num_chunks = 1;
+  auto constexpr num_chunks = RowSelection == row_selection::ALL ? 1 : 8;


No specific reason, just a value that's high enough for any overhead from chunked reading to presumably show up.

…fea-read_orc_metadata

Co-authored-by: Mike Wilson <hyperbolic2346@users.noreply.github.com>

nvdbaranec

Just a couple small comments

hyperbolic2346

tiny nit, but otherwise this looks great to me.

vuule · 2022-10-15T07:08:25Z

Benchmarked the API. Takes between 60 and 250 microseconds for 512MB dataframes, and 25 to 150 for a 512KB dataframes. Reading the same 512MB files takes 100-200 milliseconds, so the perf benefit is definitely there :)

codecov · 2022-10-15T08:54:05Z

Codecov Report

Base: 87.40% // Head: 88.13% // Increases project coverage by +0.72% 🎉

Coverage data is based on head (bba422a) compared to base (f72c4ce).
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff                @@
##           branch-22.12   #11815      +/-   ##
================================================
+ Coverage         87.40%   88.13%   +0.72%     
================================================
  Files               133      133              
  Lines             21833    21982     +149     
================================================
+ Hits              19084    19373     +289     
+ Misses             2749     2609     -140

Impacted Files	Coverage Δ
python/strings_udf/strings_udf/__init__.py	`86.27% <0.00%> (-10.61%)`	⬇️
python/cudf/cudf/io/text.py	`91.66% <0.00%> (-8.34%)`	⬇️
python/cudf/cudf/core/_base_index.py	`81.28% <0.00%> (-4.27%)`	⬇️
python/strings_udf/strings_udf/_typing.py	`94.73% <0.00%> (-1.06%)`	⬇️
python/cudf/cudf/utils/utils.py	`89.91% <0.00%> (-0.69%)`	⬇️
python/cudf/cudf/testing/dataset_generator.py	`72.83% <0.00%> (-0.42%)`	⬇️
python/dask_cudf/dask_cudf/core.py	`73.72% <0.00%> (-0.41%)`	⬇️
python/dask_cudf/dask_cudf/backends.py	`84.90% <0.00%> (-0.37%)`	⬇️
python/cudf/cudf/core/dataframe.py	`93.67% <0.00%> (-0.10%)`	⬇️
python/cudf/cudf/io/orc.py	`92.94% <0.00%> (-0.09%)`	⬇️
... and 24 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

hyperbolic2346

Thanks for doing the transform. It is informative. I would consider it a wash at best overall, but it is the preferred style in cudf.

I like the performance wins here, good work!

ajschmidt8

Approving ops-codeowner file changes

nvdbaranec

One minor comment.

…fea-read_orc_metadata

vuule · 2022-11-01T17:34:49Z

@gpucibot merge

vuule added 15 commits September 9, 2022 14:18

basic version

b6d81dc

nested names

a20054b

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

2e6174e

…fea-read_orc_metadata

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

b51e4be

…fea-read_orc_metadata

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

947e5f3

…fea-read_orc_metadata

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

494d1ec

…fea-read_orc_metadata

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

a4575d7

…fea-read_orc_metadata

Merge branch 'branch-22.10' of https://github.com/rapidsai/cudf into …

57eff91

…fea-read_orc_metadata

rename; move header size to orc.hpp

5f39748

add orc type kind

b98ac4b

use read_orc_metadata in benchmarks

cf3a787

return ORC schema (incl root column)

7b2ffc5

test

f4c2374

Merge branch 'branch-22.12' of https://github.com/rapidsai/cudf into …

30cf0af

…fea-read_orc_metadata

docs; child getter

f764d8a

vuule added feature request New feature or request non-breaking Non-breaking change labels Sep 29, 2022

vuule self-assigned this Sep 29, 2022

github-actions Bot added conda libcudf Affects libcudf (C++/CUDA) code. labels Sep 29, 2022

vuule marked this pull request as ready for review September 29, 2022 19:21

vuule requested review from a team as code owners September 29, 2022 19:21

vuule requested review from hyperbolic2346 and nvdbaranec September 29, 2022 19:21

hyperbolic2346 requested changes Oct 3, 2022

View reviewed changes

vuule added 0 - Waiting on Author Waiting for author to respond to review cuIO cuIO issue labels Oct 3, 2022

Merge branch 'branch-22.12' of https://github.com/rapidsai/cudf into …

3e88668

…fea-read_orc_metadata

vuule and others added 4 commits October 11, 2022 13:22

refactor to better match pyarrow types

6b0ac73

Apply suggestions from code review

0adae03

Co-authored-by: Mike Wilson <hyperbolic2346@users.noreply.github.com>

encapsulation

2431bb8

namespace

83560db

vuule requested a review from hyperbolic2346 October 11, 2022 22:56

nvdbaranec reviewed Oct 12, 2022

View reviewed changes

Comment thread cpp/benchmarks/io/orc/orc_reader_options.cpp Outdated

Comment thread cpp/src/io/functions.cpp Outdated

hyperbolic2346 approved these changes Oct 12, 2022

View reviewed changes

Comment thread cpp/src/io/functions.cpp Outdated

Comment thread cpp/tests/io/orc_test.cpp Outdated

Comment thread cpp/include/cudf/io/orc_metadata.hpp

vuule added 3 commits October 14, 2022 13:22

transform

73aa3ee

update benchmark

f014479

rvalue getters

8125908

vuule requested a review from nvdbaranec October 17, 2022 03:47

hyperbolic2346 approved these changes Oct 17, 2022

View reviewed changes

ajschmidt8 approved these changes Oct 17, 2022

View reviewed changes

num chunks constant

b3d31e8

vuule added 4 - Needs cuIO Reviewer and removed 0 - Waiting on Author Waiting for author to respond to review labels Oct 21, 2022

hyperbolic2346 approved these changes Oct 26, 2022

View reviewed changes

nvdbaranec reviewed Oct 31, 2022

View reviewed changes

Comment thread cpp/benchmarks/io/orc/orc_reader_options.cpp Outdated

vuule added 3 commits October 31, 2022 12:52

Merge branch 'branch-22.12' of https://github.com/rapidsai/cudf into …

89ebe73

…fea-read_orc_metadata

default_stream_value in detail

a351d86

benchmark comments

bba422a

vuule requested a review from nvdbaranec October 31, 2022 21:18

nvdbaranec approved these changes Nov 1, 2022

View reviewed changes

rapids-bot Bot merged commit 41fca6e into NVIDIA:branch-22.12 Nov 1, 2022

vuule deleted the fea-read_orc_metadata branch November 1, 2022 17:34

vyasr added 4 - Needs Review Waiting for reviewer to review or respond and removed 4 - Needs cuIO Reviewer labels Feb 23, 2024

Uh oh!

Conversation

vuule commented Sep 29, 2022

Description

Checklist

Uh oh!

vuule commented Sep 29, 2022

Uh oh!

hyperbolic2346 left a comment

Choose a reason for hiding this comment

Uh oh!

hyperbolic2346 Oct 3, 2022

Choose a reason for hiding this comment

Uh oh!

vuule Oct 11, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nvdbaranec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hyperbolic2346 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vuule commented Oct 15, 2022

Uh oh!

codecov Bot commented Oct 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hyperbolic2346 left a comment

Choose a reason for hiding this comment

Uh oh!

ajschmidt8 left a comment

Choose a reason for hiding this comment

Uh oh!

nvdbaranec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vuule commented Nov 1, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented Oct 15, 2022 •

edited

Loading