[REVIEW] Update orc reader and writer fuzz tests #7357

galipremsagar · 2021-02-09T20:24:11Z

This PR introduces:

Fixes to some of the breakages introduced by the latest pyorc in using pyorc.Struct.
Adapt to list dtype parameter changes introduced previously.
Misc fixes required for proper fuzz test runs.

vuule

LGTM, but it's kind of hard to understand why some changes were made (without the detailed description).

vuule · 2021-02-09T21:43:03Z

python/cudf/cudf/_fuzz_testing/orc.py

@@ -67,6 +67,19 @@ def generate_input(self):
            dtypes_meta, num_rows, num_cols = _generate_rand_meta(
                self, dtypes_list
            )
+            if num_cols == 0:
+                """
+                If a dataframe has no columns, then pyorc writer will throw


I wondered what the desired behavior is here, i.e. whether ORC as a format supports having no columns.
I think we also have some issues writing empty dataframes to ORC.

I guess we need to write an empty struct. With pyorc we can do it this way:

>>> import pyorc >>> import pandas as pd >>> output = open("sample.orc", "wb") >>> writer = pyorc.Writer(output, pyorc.Struct()) >>> writer.close() >>> pd.read_orc('sample.orc') Empty DataFrame Columns: [] Index: []

Looks like I also need to make some code-changes in this PR. I'll update this PR.

galipremsagar · 2021-02-09T23:11:46Z

LGTM, but it's kind of hard to understand why some changes were made (without the detailed description).

Updated the code-changes with comments, this is ready for a re-review.

python/cudf/cudf/_fuzz_testing/orc.py

python/cudf/cudf/_fuzz_testing/tests/fuzz_test_orc.py

codecov · 2021-02-10T00:49:26Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-0.19@c0282e6). Click here to learn what that means.
The diff coverage is n/a.

@@              Coverage Diff               @@
##             branch-0.19    #7357   +/-   ##
==============================================
  Coverage               ?   82.21%           
==============================================
  Files                  ?      100           
  Lines                  ?    16971           
  Branches               ?        0           
==============================================
  Hits                   ?    13953           
  Misses                 ?     3018           
  Partials               ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0282e6...b5f8165. Read the comment docs.

Co-authored-by: Ram (Ramakrishna Prabhu) <42624703+rgsl888prabhu@users.noreply.github.com>

vuule

found a typo, LGTM otherwise

python/cudf/cudf/_fuzz_testing/utils.py

Co-authored-by: Vukasin Milovanovic <vukasin.milovanovic.87@gmail.com>

galipremsagar · 2021-02-11T18:32:30Z

@gpucibot merge

updating orc fuzz tests

147f57b

galipremsagar added 3 - Ready for Review Ready for review by team 4 - Needs cuDF (Python) Reviewer improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Feb 9, 2021

galipremsagar self-assigned this Feb 9, 2021

galipremsagar requested a review from a team as a code owner February 9, 2021 20:24

galipremsagar requested review from shwina and rgsl888prabhu February 9, 2021 20:24

github-actions bot added the Python Affects Python cuDF API. label Feb 9, 2021

galipremsagar requested a review from vuule February 9, 2021 20:24

vuule approved these changes Feb 9, 2021

View reviewed changes

galipremsagar added 2 commits February 9, 2021 14:59

fix empty struct writing error

537e3aa

add comments

794f421

galipremsagar requested a review from vuule February 9, 2021 23:11

rgsl888prabhu reviewed Feb 10, 2021

View reviewed changes

Apply suggestions from code review

c567808

Co-authored-by: Ram (Ramakrishna Prabhu) <42624703+rgsl888prabhu@users.noreply.github.com>

rgsl888prabhu approved these changes Feb 11, 2021

View reviewed changes

galipremsagar removed the 4 - Needs cuDF (Python) Reviewer label Feb 11, 2021

vuule approved these changes Feb 11, 2021

View reviewed changes

python/cudf/cudf/_fuzz_testing/utils.py Outdated Show resolved Hide resolved

Update python/cudf/cudf/_fuzz_testing/utils.py

b5f8165

Co-authored-by: Vukasin Milovanovic <vukasin.milovanovic.87@gmail.com>

galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Needs cuIO Reviewer labels Feb 11, 2021

vuule removed the 3 - Ready for Review Ready for review by team label Feb 11, 2021

rapids-bot bot merged commit ebe307e into rapidsai:branch-0.19 Feb 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REVIEW] Update orc reader and writer fuzz tests #7357

[REVIEW] Update orc reader and writer fuzz tests #7357

galipremsagar commented Feb 9, 2021

vuule left a comment •

edited

Loading

vuule Feb 9, 2021

galipremsagar Feb 9, 2021

galipremsagar commented Feb 9, 2021

codecov bot commented Feb 10, 2021 •

edited

Loading

vuule left a comment

galipremsagar commented Feb 11, 2021

[REVIEW] Update orc reader and writer fuzz tests #7357

[REVIEW] Update orc reader and writer fuzz tests #7357

Conversation

galipremsagar commented Feb 9, 2021

vuule left a comment • edited Loading

Choose a reason for hiding this comment

vuule Feb 9, 2021

Choose a reason for hiding this comment

galipremsagar Feb 9, 2021

Choose a reason for hiding this comment

galipremsagar commented Feb 9, 2021

codecov bot commented Feb 10, 2021 • edited Loading

Codecov Report

vuule left a comment

Choose a reason for hiding this comment

galipremsagar commented Feb 11, 2021

vuule left a comment •

edited

Loading

codecov bot commented Feb 10, 2021 •

edited

Loading