Create EntitySet with Koalas #1031
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
frances-h
changed the title
Tagar
reviewed
Jul 29, 2020
| if any(isinstance(entity.df, dd.DataFrame) for entity in mock_customer.entities): | ||
| pytest.xfail('Dask version fails, returned feature matrix is empty') | ||
| if any(isinstance(entity.df, (dd.DataFrame, ks.DataFrame)) for entity in mock_customer.entities): | ||
| pytest.xfail("Dask fails because returned feature matrix is empty; Koalas doesn't support custom agg functions") |
There was a problem hiding this comment.
well.. actually Koalas support custom aggregate functions through PySpark - it's way more flexible than Dask in this regard
https://spark.apache.org/docs/3.0.0/sql-ref-functions-udf-aggregate.html
There was a problem hiding this comment.
I think we can use .groupby(...).apply(...), see https://koalas.readthedocs.io/en/latest/reference/api/databricks.koalas.groupby.GroupBy.apply.html#databricks.koalas.groupby.GroupBy.apply
frances-h
force-pushed
the
koalas-entity
branch
from
4270e04 to
fec2da0
Compare
frances-h
force-pushed
the
koalas-entity
branch
from
5f27a0c to
6141607
Compare
kmax12
reviewed
Aug 13, 2020
kmax12
reviewed
Aug 13, 2020
| lti_df['last_time'] = ks.to_datetime(lti_df['last_time']) | ||
| lti_df['last_time_old'] = ks.to_datetime(lti_df['last_time_old']) | ||
| # TODO: Figure out a workaround for fillna and replace | ||
| lti_df = lti_df.max(axis=1) |
There was a problem hiding this comment.
In testing on the instacart dataset for the predict next purchase problem, this line appears to be causing a performance issue.
featuretools/entityset/entity.py
Outdated
Comment on lines
253
to
256
| if isinstance(instance_vals, dd.Series) or isinstance(instance_vals, ks.Series): | ||
| df = self.df.merge(instance_vals.to_frame(), how="inner", on=variable_id) | ||
| else: | ||
| df = self.df[self.df[variable_id].isin(instance_vals)] |
There was a problem hiding this comment.
In testing on the instacart dataset for the predict next purchase problem, this code block appears to be causing a performance issue.
* fix entity creation performance issue * fix test
* initial pass of making koalas optional * add optional for make_ecommerce_entityset * add koalas to extra_require, add error, revert accidental change * bump koalas version * make koalas optional for tests, add minimal circleci test runs * lint and update circleci config * update circleci config * fix spark defaults in conftest * update circleci config * use pytest.importorskip * update missing koalas error message * update koalas import error message * rename circleci tests
* updates to improve Entity.query_by_values perf * update for optional koalas install
* test coverage updates * test coverage updates * use set for explored to speed up test * misc test cleanup
* update docs * fix typo * add java to circleci docs build * fix typos * remove whitespace and add java requirement info
* update flags and get_function for agg primitives * update list-primitives * use enum instead of string keywords * update enum name to Library * fix typo * revert accidental changes * add pandas to compatibility * add comment * update bad_primitives check * split line
frances-h
force-pushed
the
koalas-entity
branch
from
b65a00b to
969cd3f
Compare
rwedge
reviewed
Sep 4, 2020
featuretools/computational_backends/calculate_feature_matrix.py
Outdated
Show resolved
Hide resolved
rwedge
changed the title
rwedge
reviewed
Sep 4, 2020
frances-h
force-pushed
the
koalas-entity
branch
from
c30f132 to
252995e
Compare
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Add support for creating entitysets with Koalas dataframes and using dfs
After creating the pull request: in order to pass the changelog_updated check you will need to update the "Future Release" section of
docs/source/changelog.rstto include this pull request.