Add `cudf.options` #11193

isVoid · 2022-07-01T23:21:21Z

This PR adds cudf.options, a global dictionary to store configurations. A set of helper functions to manage the registries are also included. See documentation included in the PR for detail.

See demonstration use in: #11272

Closes #5311

vyasr · 2022-07-02T01:47:33Z

At a glance, this design looks solid. Before we move forward too far with this, though, given how many different times something like this has been requested it would be good to see how well this works in practice for the use cases that we've consider (offhand, things like requiring stable sorts or forcing exact pandas compatibility for certain APIs come to mind).

isVoid · 2022-07-02T21:41:07Z

@vyasr certainly. The direct motivation of this PR is #10558. It's currently blocked by #11182 to fully implement what's needed. The stable sort idea sounds useful but I recall the way pandas config those is by choosing the sorting method in the API, not through config options. Is there a more specific stable sort use case you are referring to that I missed?

vyasr · 2022-07-05T17:10:28Z

The stable sort was just an arbitrary example. I was referring to the fact that APIs like cudf.merge and DataFrame.drop_duplicates order the results nondeterministically, and we have received many requests to change that. In at least some cases we could achieve deterministic ordering by doing some extra work, but we currently do not. Making that configurable is probably the most common reason that I've heard us discuss something like cudf.config in the past. If #10558 is the immediate goal, I'm just asking to see an example (perhaps in another branch based on this one) of how cudf.config would work for such a use case. It doesn't have to be sorting.

bdice

A few comments. I'm confused about why this deviates so strongly from the design of Pandas' options. I would try to align with that if at all possible, rather than coming up with an entirely different structure and naming scheme.

docs/cudf/source/api_docs/config.rst

docs/cudf/source/developer_guide/config.md

python/cudf/cudf/config.py

python/cudf/cudf/tests/test_config.py

python/cudf/cudf/config.py

vyasr · 2022-07-07T16:17:50Z

A few comments. I'm confused about why this deviates so strongly from the design of Pandas' options. I would try to align with that if at all possible, rather than coming up with an entirely different structure and naming scheme.

Part of me agrees with this, but just to play devil's advocate: do we anticipate having any overlap in the actual options that we offer? If not, it might be rather confusing to have something that looks like the same API but actually has no shared behavior since it's a completely different set of configuration options.

shwina · 2022-07-07T16:24:03Z

do we anticipate having any overlap in the actual options that we offer?

Potentially? For example, we currently piggyback off of Pandas' options context to control display behavior. There's no reason the user couldn't control the beahvior of Pandas and cuDF separately.

isVoid · 2022-07-13T19:49:50Z

Summarizing offline sync: We want to model cudf configuration interface close to pandas options interface to provide a more coherent experience for people who transfer from pandas. As of today, the options that pandas support for compute and I/O engines are incompatible with the nature of a GPU dataframe library. Thus we don't model cudf.options to be overlapping with pandas options set, but as an independent set. The first iteration will aim to provide a minimalistic (yet coherent with pandas) user interface.

codecov · 2022-07-14T03:08:09Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-22.08@ae1b581). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-22.08   #11193   +/-   ##
===============================================
  Coverage                ?   86.43%           
===============================================
  Files                   ?      144           
  Lines                   ?    22808           
  Branches                ?        0           
===============================================
  Hits                    ?    19714           
  Misses                  ?     3094           
  Partials                ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae1b581...438003c. Read the comment docs.

bdice

Feedback attached. I am glad the decision was made to stick close to what Pandas provides here - I have a few suggestions for improved alignment.

docs/cudf/source/developer_guide/index.md

docs/cudf/source/user_guide/config.md

python/cudf/cudf/__init__.py

python/cudf/cudf/options.py

python/cudf/cudf/tests/test_options.py

wence- · 2022-07-19T14:17:12Z

python/cudf/cudf/options.py

+def _build_option_description(name, opt):
+    return (
+        f"{name}:\n"
+        f"\t{opt.description}\n"
+        f"\t[Default: {opt.default}] [Current: {opt.value}]"
+    )


Does it make more sense to implement a custom __str__ on Option?

It might be alright to do that, but the Option doesn't know its own name, so it's missing a crucial piece of information. I'm happy with the current state here.

mroeschke

One feature of pandas options functions that I don't think it's publicized well is that you can pass a string that can regex match an option name e.g.

pd.get_option('my') would be equivalent to pd.get_option('my_option') if 'my' could only match 'my_option'

Personally, I am not fond of this feature as it's not explicit and not sure if there's a benefit for increased code complexity, but noting here if you want to include that functionality.

bdice · 2022-07-19T19:29:42Z

One feature of pandas options functions that I don't think it's publicized well is that you can pass a string that can regex match an option name [...]

I agree that's not necessary here, for simplicity's sake. We can align with the pandas design in lots of ways but regex matching feels like overkill unless/until we need it (e.g. if we have dozens of options and option namespaces with dot . separators).

wence-

Very minor wording, but looks good, thanks!

docs/cudf/source/developer_guide/options.md

docs/cudf/source/user_guide/options.md

Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com> Co-authored-by: Lawrence Mitchell <wence@gmx.li>

shwina · 2022-07-25T16:57:54Z

rerun tests

bdice

Some suggested refactors and typos/grammar fixes.

docs/cudf/source/user_guide/options.md

docs/cudf/source/developer_guide/options.md

docs/cudf/source/user_guide/options.md

python/cudf/cudf/__init__.py

python/cudf/cudf/options.py

python/cudf/cudf/tests/test_options.py

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

bdice

Nice work!

isVoid · 2022-07-26T23:24:54Z

Note that I have ported several review comments from @vyasr at #11272

isVoid · 2022-07-28T03:45:21Z

rerun tests

shwina · 2022-07-28T10:41:47Z

@gpucibot merge

jakirkham · 2022-07-28T16:57:01Z

Thanks all! 🙏

This PR introduces a cudf option to allow user to control the default bitwidth for integer and floating types. The first iteration only plans to provide three options: `None`, 32bit and 64bit. When set as `None`, that means the result dtype will align with what pandas constructs. Otherwise, default to what user specifies. "Default" implies that it should only affects places that requires type inference, that includes: - CSV/JSON readers when dtypes are not specified - cuDF constructors - Materializing a range index. This PR is the first demonstration use of `cudf.option`, depending on #11193. Diff will reduce once it's merged. closes #11182 #10318 Authors: - Michael Wang (https://github.com/isVoid) Approvers: - Ashwin Srinath (https://github.com/shwina) - Vyas Ramasubramani (https://github.com/vyasr) URL: #11272

isVoid added 3 commits July 1, 2022 16:13

Config functions and importing to top level namespace

7acc583

Add config tests

c25240c

add documentation

bfa10b6

github-actions bot added the Python Affects Python cuDF API. label Jul 1, 2022

isVoid added feature request New feature or request non-breaking Non-breaking change labels Jul 1, 2022

isVoid marked this pull request as ready for review July 1, 2022 23:22

isVoid requested a review from a team as a code owner July 1, 2022 23:22

isVoid requested review from bdice and charlesbluca July 1, 2022 23:22

bdice requested changes Jul 6, 2022

View reviewed changes

bdice reviewed Jul 6, 2022

View reviewed changes

python/cudf/cudf/config.py Outdated Show resolved Hide resolved

vyasr mentioned this pull request Jul 13, 2022

[FEA] Mixed precision Decimal math support in cudf Python #7680

Open

Update to options and conform to pandas API

68878cb

documentation updates

e093eb6

isVoid requested a review from bdice July 14, 2022 05:11

More docs update

bbd9cd4

isVoid changed the title ~~Add cudf.config~~ Add cudf.options Jul 14, 2022

isVoid mentioned this pull request Jul 15, 2022

Provide an Option for Default Integer and Floating Bitwidth #11272

Merged

bdice reviewed Jul 18, 2022

View reviewed changes

Docs update

90cbe33

Address docs reviews

f68c303

isVoid requested a review from charlesbluca July 19, 2022 00:45

style fixes

99201ba

wence- reviewed Jul 19, 2022

View reviewed changes

mroeschke reviewed Jul 19, 2022

View reviewed changes

wence- approved these changes Jul 20, 2022

View reviewed changes

docs/cudf/source/developer_guide/options.md Outdated Show resolved Hide resolved

docs/cudf/source/developer_guide/options.md Outdated Show resolved Hide resolved

docs/cudf/source/developer_guide/options.md Outdated Show resolved Hide resolved

charlesbluca reviewed Jul 21, 2022

View reviewed changes

docs/cudf/source/developer_guide/options.md Outdated Show resolved Hide resolved

docs/cudf/source/user_guide/options.md Outdated Show resolved Hide resolved

Address docs review comments

b741f1b

Co-authored-by: Charles Blackmon-Luca <20627856+charlesbluca@users.noreply.github.com> Co-authored-by: Lawrence Mitchell <wence@gmx.li>

isVoid requested a review from charlesbluca July 22, 2022 22:50

mroeschke approved these changes Jul 25, 2022

View reviewed changes

bdice reviewed Jul 25, 2022

View reviewed changes

Apply review comments

4ac78be

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

bdice approved these changes Jul 26, 2022

View reviewed changes

isVoid added 4 commits July 26, 2022 16:17

Add try except blocks and documents errors raised

f872c28

minor grammar updates

7d018e1

removes try except block around validator

e0a2546

style fixes

6ab41b5

Add one liner in dev guide

438003c

rapids-bot bot merged commit e4ce301 into rapidsai:branch-22.08 Jul 28, 2022

vyasr mentioned this pull request Nov 23, 2022

[FEA] Add a new cuDF option stable_sort that provides ordering guarantees for otherwise nondeterministic APIs #12236

Closed

jakirkham mentioned this pull request Dec 5, 2022

Add packed serialization option for dataframes #8661

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `cudf.options` #11193

Add `cudf.options` #11193

isVoid commented Jul 1, 2022 •

edited

Loading

vyasr commented Jul 2, 2022

isVoid commented Jul 2, 2022

vyasr commented Jul 5, 2022

bdice left a comment •

edited

Loading

vyasr commented Jul 7, 2022

shwina commented Jul 7, 2022

isVoid commented Jul 13, 2022

codecov bot commented Jul 14, 2022 •

edited

Loading

bdice left a comment

wence- Jul 19, 2022

bdice Jul 19, 2022

mroeschke left a comment

bdice commented Jul 19, 2022

wence- left a comment

shwina commented Jul 25, 2022

bdice left a comment

bdice left a comment

isVoid commented Jul 26, 2022

isVoid commented Jul 28, 2022

shwina commented Jul 28, 2022

jakirkham commented Jul 28, 2022

Add cudf.options #11193

Add cudf.options #11193

Conversation

isVoid commented Jul 1, 2022 • edited Loading

vyasr commented Jul 2, 2022

isVoid commented Jul 2, 2022

vyasr commented Jul 5, 2022

bdice left a comment • edited Loading

Choose a reason for hiding this comment

vyasr commented Jul 7, 2022

shwina commented Jul 7, 2022

isVoid commented Jul 13, 2022

codecov bot commented Jul 14, 2022 • edited Loading

Codecov Report

bdice left a comment

Choose a reason for hiding this comment

wence- Jul 19, 2022

Choose a reason for hiding this comment

bdice Jul 19, 2022

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

bdice commented Jul 19, 2022

wence- left a comment

Choose a reason for hiding this comment

shwina commented Jul 25, 2022

bdice left a comment

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

isVoid commented Jul 26, 2022

isVoid commented Jul 28, 2022

shwina commented Jul 28, 2022

jakirkham commented Jul 28, 2022

Add `cudf.options` #11193

Add `cudf.options` #11193

isVoid commented Jul 1, 2022 •

edited

Loading

bdice left a comment •

edited

Loading

codecov bot commented Jul 14, 2022 •

edited

Loading