New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEAT-#7001: Do not force materialization in MetaList.__getitem__() #7006
Conversation
2f1b6a2
to
d9c4ead
Compare
modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py
Fixed
Show fixed
Hide fixed
d9c4ead
to
3edf4b6
Compare
…etitem__() Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
3edf4b6
to
ae05a5a
Compare
modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py
Outdated
Show resolved
Hide resolved
obj = self.meta._obj | ||
return obj[self.idx] if isinstance(obj, list) else obj | ||
|
||
def map(self, materialized): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am kind of confused why this method is named map. Can you elaborate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main idea is to apply some calculation to a single materialized object and get one or multiple values. List with lengths is mapped to multiple lengths with MetaListMapper. Length is mapped to a different length with SlicedLenMapper.
If it's confusing, I let's rename it fo, for example, transfrom.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Renamed to MaterializationHook.
@dchigarev, any comments? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
left minor comments, otherwise LGTM, will check performance and respond with an approval
UPD: checked performance for the reproducer from #6948 and everything is fine
modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py
Show resolved
Hide resolved
Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
…etitem__() (modin-project#7006) Co-authored-by: Dmitry Chigarev <dmitry.chigarev@intel.com> Signed-off-by: Andrey Pavlenko <andrey.a.pavlenko@gmail.com>
What do these changes do?
flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
git commit -s
docs/development/architecture.rst
is up-to-date