Skip to content

Eagerly populate the class dict for cudf.pandas proxy types - #14534

Merged
rapids-bot[bot] merged 95 commits into
NVIDIA:branch-24.06from
shwina:eagerly-populate-class-dict
May 17, 2024
Merged

Eagerly populate the class dict for cudf.pandas proxy types#14534
rapids-bot[bot] merged 95 commits into
NVIDIA:branch-24.06from
shwina:eagerly-populate-class-dict

Conversation

@shwina

@shwina shwina commented Nov 30, 2023

Copy link
Copy Markdown
Contributor

Rather than dynamically looking up class attributes (and methods), this PR makes it so that we eagerly populate the class with all known methods and attributes (by inspecting the "slow" class).

This solves a number of problems:

  • it makes getattr trivially inexpensive (no dynamic __getattr__ for each attribute access)
  • it ensures the same object is returned every time you do, e.g., DataFrame.max
  • it makes tab completion fast because the attributes don't have to be computed each time
  • it no longer exposes attributes that are specific to cuDF - for example Series.list
  • it allows subclassing of proxy types to work better. For example, derived types can now call super(). to access attributes of base types

@github-actions github-actions Bot added the Python Affects Python cuDF API. label Nov 30, 2023
@copy-pr-bot

copy-pr-bot Bot commented Feb 22, 2024

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@shwina
shwina changed the base branch from branch-24.02 to branch-24.04 February 22, 2024 19:50
@github-actions github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue conda and removed Python Affects Python cuDF API. labels Feb 22, 2024
@shwina shwina removed libcudf Affects libcudf (C++/CUDA) code. CMake CMake build issue conda labels Feb 23, 2024
Comment thread ci/cudf_pandas_scripts/pandas-tests/run.sh Outdated
@github-actions github-actions Bot removed the ci label May 16, 2024
Comment thread python/cudf/cudf/pandas/scripts/run-pandas-tests.sh Outdated
@galipremsagar

Copy link
Copy Markdown
Contributor

/okay to test

@galipremsagar

Copy link
Copy Markdown
Contributor

/okay to test

@galipremsagar

galipremsagar commented May 17, 2024

Copy link
Copy Markdown
Contributor

@wence- I've reverted the vast majority of the changes I made to this PR and this is now ready for review. Since I've made a couple of minor fixes to this pr I'd like to get it reviewed by someone other than me before merging.

Something to note, the drop in test pass rate is expected and are valid failures that are not in the scope of fixing in this PR.

Comment thread python/cudf/cudf/pandas/fast_slow_proxy.py Outdated
Comment thread python/cudf/cudf/pandas/_wrappers/numpy.py Outdated
galipremsagar and others added 2 commits May 17, 2024 15:59
Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>
if instance is not None:
return _maybe_wrap_result(
getattr(instance._fsproxy_slow, self._name),
None, # type: ignore

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be good to adjust the func typing in _maybe_wrap_result as func: Callable | None and raise an exception in is_final_type and is_intermediate_type branches if func is None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to be straight-forward, while testing this change I noticed 4k additional failures. Maybe we can revisit this in another PR.

"_constructor": _FastSlowAttribute("_constructor"),
"__array_ufunc__": _FastSlowAttribute("__array_ufunc__"),
"_accessors": set(),
"_data": _FastSlowAttribute("_data", private=True),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So these attributes we are wanting the fast path to have a chance to evaluate and not automatically be _Unusable()?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah so in this case since it's private we won't try to operate on e.g. cudf.Index._data which is a ColumnAccessor and instead reach for cudf.Index._data which is a numpy ndarray which we can wrap.

This part of the eager population I'm not too thrilled about. These private variables are not guaranteed to be stable. It would be nice if any slow attribute was attempted to return a proxy object so we didn't have to specify attributes like this but I suppose that can be looked at in a follow up

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, there are still such cases we can address in follow-ups. At this point the scope of this PR has kept on expanding, that's why I haven't addressed additional issues in this PR.

Comment thread python/cudf/cudf/pandas/scripts/run-pandas-tests.sh
@mroeschke

Copy link
Copy Markdown
Contributor

/okay to test

@mroeschke mroeschke left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My typing request and concern about having to define private attributes in our proxies can be addressed in a follow up

@galipremsagar

Copy link
Copy Markdown
Contributor

/merge

@rapids-bot
rapids-bot Bot merged commit e6e6761 into NVIDIA:branch-24.06 May 17, 2024
@galipremsagar galipremsagar added the 5 - Ready to Merge Testing and reviews complete, ready to merge label May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

5 - Ready to Merge Testing and reviews complete, ready to merge improvement Improvement / enhancement to an existing function non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Tab completion broken in ipython

7 participants