New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update calculating features to handle multiple paths #572
Commits on May 31, 2019
-
Refactor PandasBackend and FeatureTree to use paths
This is just a refactor, it does not change existing functionality.
-
Update calculating features to handle multiple paths
calculate_feature_matrix can now calculate features on entitysets where there are multiple paths between two entities ("diamond graphs"). The basic algorithm: 1. Construct a trie where the edges are relationships and each node contains a set of features. Also construct a trie with the same structure for storing dataframes. 2. Traverse the trie using depth first search. For each node: 1. Get a dataframe for the entity (of instances related to the parent). 2. Add variables linking the entity to its ancestors. 3. Recurse on children. After this is done all feature dependencies will have been calculated and stored in the dataframe trie. 4. Group the features for this node in the trie. 5. Calculate the features of each group. - Add Trie class. - Make Relationship hashable. - Rename FeatureTree to FeatureSet - Use feature.unique_name instead of hash in dicts. Features should not be used in dictionaries and sets because they do not support equality. The equality operator is overloaded to produce a new feature instead of comparing features. We have instead been using feature hash values, but this could potentially lead to bugs due to hash collisions. This commit instead uses the feature's unique_name where the hash had been used. The unique name contains the same information that is used to generate the hash, so it should not lead to any collisions.
-
Change DirectFeature to take a single relationship
Instead of a path, since we currently only support paths of length 1. Updates features JSON SCHEMA_VERSION.
-
Test calculate features on es with cycle
This currently fails because we do not detect that paths with cycles are not unique. This means that two features with different length paths through the same cycle will be given the same name, and will be treated as the same feature in sets and dictionaries.
-
Commits on Jun 3, 2019
Commits on Jun 4, 2019
-
-
Use dataframe from given entityset in PandasBackend
Instead of accessing it through the entity of the relationship, as this may have been deserialized and so may only have the entityset "metadata" and no actual data.
-
-
-
Calculate necessary columns for entity on demand
Instead of storing them ahead of time in a trie.
-
Commits on Jun 5, 2019
-
-
-
So that time_last, training_window, etc. can be shared across recursive calls without passing them.
-
I'm not sure how this was created (in addition to the normal .DS_Store)
-
Remove feature grouping by output frame type
This is no longer necessary since we use full frames for an entity if any features require it.
-
-
-
Only store df in trie after calculation
This way it is clear where the trie gets updated.
-
-
-
-
Merge branch 'feature-trie' of github.com:Featuretools/featuretools i…
…nto feature-trie
-
Commits on Jun 6, 2019
-
Get rid of FeatureTrie[] methods
Instead use .get_node and .value Hopefully this is more readable than trie[[]]
-
-
Replace 3 params with parent_data tuple
Since these will all either be present or not.
-
-
Don't call get_node with empty path
Because it will just return the object it was called on.
-
Represents a series of relationships, and the directions in which they are traversed.
-
-
Commits on Jun 11, 2019
-
-
-
Track approximated features by path
Instead of entity id. Changes precalculated_features to be a Trie. Updates Trie iterator to yield RelationshipPaths instead of lists.
-
-
-
Remove an extra column, removing the need for suffixes in the merge.
Commits on Jun 12, 2019
Commits on Jun 14, 2019
-
Merge branch 'master' into feature-trie
One test needed to be updated which attempted to make a direct feature of a grandparent.
-
Store full entity dfs in separate trie (#598)
If there is an aggregation of a uses_full_entity feature the aggregation should not be calculated on the full entity. Previously because we only had one set of dfs this was not possible. Now, we always store the "filtered" df in df_trie, and if necessary also store a full df in large_df_trie. The values of feature_trie are now a 3-tuple of (needs full entity, needs full entity features, rest of features). The main issue this solves is when there is a node without any features we might still need the full entity. For example: if the only target feature is "PERCENTILE(MEAN(customers.transactions))" there are no features on "customers", but we need the full entity so that the every row in "transactions" can get its relationship variable to the target entity. If any features at a given node require a full df we first calculate them. Then we extract the filtered df from the result and use this to calculate any remaining features.
-
Instead, calculate_feature_matrix constructs a FeatureSet when first calls, and then uses FeaturesCalculator for sets of instance_ids and cutoffs.