FEAT: Implement support for elementwise UDF that returns multiple col… #2473

icexelloss · 2020-10-15T22:12:32Z

What is this change

This PR adds a supports for add multiple columns with a single elementwise UDF:

Example:

@elementwise(
    input_type=[dt.double],
    output_type=dt.Struct(['col1', 'col2'], [dt.double, dt.double]),
)
def foo1(v):
    return pd.DataFrame({'col1': v + 1, 'col2': v + 2})

result = alltypes.mutate(foo1(alltypes['double_col']).destructure()).execute()

expected = alltypes.mutate(
    col1=alltypes['double_col'] + 1, col2=alltypes['double_col'] + 2,
).execute()

This PR adds support in both pandas and pyspark backennd

Tests

Added test_elementwise_udf_struct and test_elementwise_udf_destruct

icexelloss · 2020-10-15T22:19:39Z

setup.cfg

@@ -16,7 +16,7 @@ inherit = false
 convention = numpy

 [isort]
-known_third_party = asv,click,clickhouse_driver,dateutil,google,graphviz,impala,kudu,mock,multipledispatch,numpy,pandas,pkg_resources,plumbum,psycopg2,pyarrow,pydata_google_auth,pygit2,pymapd,pymysql,pyspark,pytest,pytz,regex,requests,setuptools,sphinx_rtd_theme,sqlalchemy,thrift,toolz


For some reason seed-isort-config is changing this. Investigating.

This can be reverted after #2474 merged

icexelloss · 2020-10-16T15:10:13Z

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
  - repo: https://github.com/asottile/seed-isort-config
-    rev: v1.9.2
+    rev: v2.2.0


Will revert after #2474 merged

jreback

some questions & can you add a release note

jreback · 2020-10-16T20:46:12Z

ibis/backends/pandas/execution/selection.py

    if np.isscalar(result):
        return pd.Series(
            np.repeat(result, len(data.index)),
            index=data.index,
            name=result_name,
        )
-    return result.rename(result_name)
+
+    if expr.has_name():


could e an elif

ibis/backends/pandas/execution/selection.py

ibis/expr/api.py

jreback · 2020-10-16T20:47:43Z

ibis/expr/operations.py

-            if isinstance(projection, ir.ValueExpr):
+            if (
+                isinstance(projection, ir.StructValue)
+                and not projection.has_name()


why does the name matter?

If this is a named struct, we want to create a struct column instead of "flatten" the struct. e.g

This creates a struct column "new_struct_col" with col1, col2

table = table.mutate(new_struct_col=struct_udf(dt['v']))

This creates two columns, col1 and col2

table = table.mutate(struct_udf(dt['v']))

Also added test_elementwise_udf_struct to test adding a named struct column

hmm, is this the api we really want, e.g. this implicit naming behavior? it seems sensible but no easy way to turn this on/off or error check it.

jreback · 2020-10-16T20:48:09Z

ibis/file/csv.py

+                    None,
+                    [
+                        getattr(s.op(), 'name', None)
+                        or (s.get_name() if s.has_name() else None)


hmm see above

…umns

icexelloss · 2020-10-19T16:23:05Z

Per discussion offline with @jreback, changing get_name for return None instead of throwing exception has wide impact (fails many tests) and I will address that change in a separate PR.

Opened #2484 to track

jreback · 2020-10-19T23:39:56Z

ibis/expr/operations.py

-            if isinstance(projection, ir.ValueExpr):
+            if (
+                isinstance(projection, ir.StructValue)
+                and not projection.has_name()


hmm, is this the api we really want, e.g. this implicit naming behavior? it seems sensible but no easy way to turn this on/off or error check it.

jreback · 2020-10-19T23:41:01Z

ibis/udf/vectorized.py

+    ...         [date.str.slice(0, 4), date.str.slice(4, 8)],
+    ...         axis=1
+    ...     )
+    ...     result.columns = ['year', 'monthday']


what happens if the result is NOT named, do I get an exception?

Actually, results are not required to be named. I updated the test to reflect this. Column labels will be assign based on output_type.

jreback · 2020-10-21T00:22:30Z

thanks @icexelloss

icexelloss requested a review from jreback October 15, 2020 22:16

icexelloss added pandas The pandas backend pyspark The Apache PySpark backend udf Issues related to user-defined functions labels Oct 15, 2020

icexelloss force-pushed the udf-multiple-return-cols branch from e00e277 to 74aa88c Compare October 15, 2020 22:18

icexelloss commented Oct 15, 2020

View reviewed changes

icexelloss commented Oct 16, 2020

View reviewed changes

icexelloss force-pushed the udf-multiple-return-cols branch from f4ad892 to 1d42727 Compare October 16, 2020 18:10

jreback suggested changes Oct 16, 2020

View reviewed changes

icexelloss added 6 commits October 19, 2020 11:10

FEAT: Implement support for elementwise UDF that returns multiple col…

2fd97b7

…umns

Fix csv pre_execute rule for Selection

83fb7ec

Add doc examples

31450dc

Fix setup.cfg

5af8430

Address comments

7053364

Add release notes

62173a2

icexelloss force-pushed the udf-multiple-return-cols branch from 5de0b8c to 62173a2 Compare October 19, 2020 15:11

icexelloss added 4 commits October 19, 2020 11:41

Check unnamed expressions in Aggregation node

5056da9

Revert change to get_name

7acb320

Revert changes to Aggregation node

077f335

Revert changes to setup.cfg

8e3bd39

icexelloss mentioned this pull request Oct 19, 2020

Consider changing get_name to return None instead of throwing exception for unnamed expressions #2484

Closed

Fix csv backend

8b5f917

jreback suggested changes Oct 19, 2020

View reviewed changes

icexelloss added 5 commits October 20, 2020 14:43

Add destructure

603087a

clean up

b900911

Allow returned value to be unnamed

6af45f2

Clean up

9be5f7d

Fix tests

e664232

jreback added this to the Next Feature Release milestone Oct 20, 2020

jreback approved these changes Oct 20, 2020

View reviewed changes

jreback merged commit 41cfc4d into ibis-project:master Oct 21, 2020

icexelloss mentioned this pull request Oct 21, 2020

FEAT: Add destructure support for analtyic and reduction UDF #2487

Merged

timothydijamco mentioned this pull request May 3, 2021

ENH: Use ndarray as array representation in Pandas backend #2753

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FEAT: Implement support for elementwise UDF that returns multiple col… #2473

FEAT: Implement support for elementwise UDF that returns multiple col… #2473

icexelloss commented Oct 15, 2020 •

edited

Loading

icexelloss Oct 15, 2020

icexelloss Oct 16, 2020

icexelloss Oct 16, 2020

jreback left a comment

jreback Oct 16, 2020

icexelloss Oct 19, 2020

jreback Oct 16, 2020

icexelloss Oct 19, 2020 •

edited

Loading

icexelloss Oct 19, 2020 •

edited

Loading

jreback Oct 19, 2020

jreback Oct 16, 2020

icexelloss Oct 19, 2020

icexelloss commented Oct 19, 2020

jreback Oct 19, 2020

jreback Oct 19, 2020

icexelloss Oct 20, 2020

jreback commented Oct 21, 2020

FEAT: Implement support for elementwise UDF that returns multiple col… #2473

FEAT: Implement support for elementwise UDF that returns multiple col… #2473

Conversation

icexelloss commented Oct 15, 2020 • edited Loading

What is this change

Tests

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

icexelloss Oct 19, 2020 • edited Loading

Choose a reason for hiding this comment

icexelloss Oct 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

icexelloss commented Oct 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Oct 21, 2020

icexelloss commented Oct 15, 2020 •

edited

Loading

icexelloss Oct 19, 2020 •

edited

Loading

icexelloss Oct 19, 2020 •

edited

Loading