This repository was archived by the owner on Feb 2, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 62
Implement initial version of rewrite for df.getitem as attr #601
Merged
AlexanderKalistratov
merged 3 commits into
IntelPython:master
from
densmirn:feature/df_getitem_attr
Feb 14, 2020
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # ***************************************************************************** | ||
| # Copyright (c) 2020, Intel Corporation All rights reserved. | ||
| # | ||
| # Redistribution and use in source and binary forms, with or without | ||
| # modification, are permitted provided that the following conditions are met: | ||
| # | ||
| # Redistributions of source code must retain the above copyright notice, | ||
| # this list of conditions and the following disclaimer. | ||
| # | ||
| # Redistributions in binary form must reproduce the above copyright notice, | ||
| # this list of conditions and the following disclaimer in the documentation | ||
| # and/or other materials provided with the distribution. | ||
| # | ||
| # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" | ||
| # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, | ||
| # THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR | ||
| # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR | ||
| # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, | ||
| # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, | ||
| # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; | ||
| # OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, | ||
| # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR | ||
| # OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, | ||
| # EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
| # ***************************************************************************** | ||
|
|
||
| from numba.ir import Assign, Const, Expr, Var | ||
| from numba.ir_utils import mk_unique_var | ||
| from numba.rewrites import register_rewrite, Rewrite | ||
| from numba.types import StringLiteral | ||
| from numba.typing import signature | ||
|
|
||
| from sdc.config import config_pipeline_hpat_default | ||
| from sdc.hiframes.pd_dataframe_type import DataFrameType | ||
|
|
||
|
|
||
| if not config_pipeline_hpat_default: | ||
| @register_rewrite('after-inference') | ||
| class RewriteDataFrameGetItemAttr(Rewrite): | ||
| """ | ||
| Search for calls of df.attr and replace it with calls of df['attr']: | ||
| $0.2 = getattr(value=df, attr=A) -> $const0.0 = const(str, A) | ||
| $0.2 = static_getitem(value=df, index=A, index_var=$const0.0) | ||
| """ | ||
|
|
||
| def match(self, func_ir, block, typemap, calltypes): | ||
| self.func_ir = func_ir | ||
| self.block = block | ||
| self.typemap = typemap | ||
| self.calltypes = calltypes | ||
| self.getattrs = getattrs = set() | ||
| for expr in block.find_exprs(op='getattr'): | ||
| obj = typemap[expr.value.name] | ||
| if not isinstance(obj, DataFrameType): | ||
| continue | ||
| if expr.attr in obj.columns: | ||
| getattrs.add(expr) | ||
|
|
||
| return len(getattrs) > 0 | ||
|
|
||
| def apply(self): | ||
| new_block = self.block.copy() | ||
| new_block.clear() | ||
AlexanderKalistratov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| for inst in self.block.body: | ||
AlexanderKalistratov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| if isinstance(inst, Assign) and inst.value in self.getattrs: | ||
| const_assign = self._assign_const(inst) | ||
| new_block.append(const_assign) | ||
|
|
||
| inst = self._assign_getitem(inst, index=const_assign.target) | ||
|
|
||
| new_block.append(inst) | ||
|
|
||
| return new_block | ||
|
|
||
| def _assign_const(self, inst, prefix='$const0'): | ||
| """Create constant from attribute of the instruction.""" | ||
| const_node = Const(inst.value.attr, inst.loc) | ||
| const_var = Var(inst.target.scope, mk_unique_var(prefix), inst.loc) | ||
AlexanderKalistratov marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| self.func_ir._definitions[const_var.name] = [const_node] | ||
| self.typemap[const_var.name] = StringLiteral(inst.value.attr) | ||
|
|
||
| return Assign(const_node, const_var, inst.loc) | ||
|
|
||
| def _assign_getitem(self, inst, index): | ||
| """Create getitem instruction from the getattr instruction.""" | ||
| new_expr = Expr.getitem(inst.value.value, index, inst.loc) | ||
| new_inst = Assign(value=new_expr, target=inst.target, loc=inst.loc) | ||
|
|
||
| self.func_ir._definitions[inst.target] = [new_expr] | ||
| self.calltypes[new_expr] = signature( | ||
| self.typemap[inst.target.name], | ||
| self.typemap[new_expr.value.name], | ||
| self.typemap[new_expr.index.name] | ||
| ) | ||
|
|
||
| return new_inst | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So we won't rewrite if we didn't find valid column name, right?
This means we won't raise AttributeError as pandas does if we use an invalid attribute name. We should probably state that in limitations, or think if we can look for all_supported_dataframe_methods in this rewrite and if we actually call a method skip rewriting, which aligns to pandas behavior:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we need to think about it, but in next PR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right now, the main issue us the case when column is named like '_data', '_columns' or '_index'. This will break all uses of dataframe