Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare entities by ID in DFS #637

Merged
merged 13 commits into from
Jul 8, 2019
Merged

Compare entities by ID in DFS #637

merged 13 commits into from
Jul 8, 2019

Conversation

CJStadler
Copy link
Contributor

This is much faster than fully comparing the entities.

This is much faster than fully comparing the entities.
@codecov
Copy link

codecov bot commented Jul 3, 2019

Codecov Report

Merging #637 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #637      +/-   ##
==========================================
- Coverage   97.43%   97.42%   -0.02%     
==========================================
  Files         118      118              
  Lines        9538     9539       +1     
==========================================
  Hits         9293     9293              
- Misses        245      246       +1
Impacted Files Coverage Δ
featuretools/feature_base/feature_base.py 97.63% <100%> (ø) ⬆️
featuretools/entityset/entity.py 96.09% <100%> (-0.41%) ⬇️
featuretools/variable_types/variable.py 98.23% <100%> (+0.02%) ⬆️
featuretools/entityset/relationship.py 98.68% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a990858...39c3819. Read the comment docs.

@@ -108,7 +108,8 @@ def _is_unique(self):
"""Is there any other relationship with same parent and child entities?"""
es = self.child_entity.entityset
relationships = es.get_forward_relationships(self._child_entity_id)
n = len([r for r in relationships if r.parent_entity == self.parent_entity])
n = len([r for r in relationships
if r._parent_entity_id == self._parent_entity_id])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case wasn't causing the observed performance drop, but I also don't see any reason to compare the actual entities here.

@CJStadler
Copy link
Contributor Author

Even though __eq__ wasn't being called with deep=True before, so it wasn't comparing dataframes, the comparison of variables is non-trivial (quadratic in the number of variables).

https://github.com/Featuretools/featuretools/blob/e67fe588ba3636a5b602b9c55229fe7da1214e2c/featuretools/entityset/entity.py#L120-L122

for v in self.variables:
if v not in other.variables:
return False
if set(self.variables) != set(other.variables):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed most of the places where this would be hit, but I thought I might as well make this comparison faster while we're looking at it.

kmax12
kmax12 previously approved these changes Jul 7, 2019
Copy link
Contributor

@kmax12 kmax12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -14,6 +14,7 @@ Changelog
* Keep dataframe sorted by time during feature calculation (:pr:`626`)
* Fix bug in encode_features that created duplicate columns of
features with multiple outputs (:pr:`622`)
* Fix performance regression in DFS (:pr:`637`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move to Future Release

kmax12
kmax12 previously approved these changes Jul 8, 2019
Copy link
Contributor

@kmax12 kmax12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@CJStadler CJStadler merged commit 4add6be into master Jul 8, 2019
@CJStadler CJStadler deleted the compare-entity-ids branch July 8, 2019 14:58
@rwedge rwedge mentioned this pull request Aug 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants