[SPARK-34544][PYTHON] Convert PandasDataFrameLike and PandasSeriesLike to aliases of Pandas types by zero323 · Pull Request #34927 · apache/spark

zero323 · 2021-12-17T00:16:28Z

What changes were proposed in this pull request?

This PR proposes replacing currently used Protocols:

PandasDataFrameLike
PandasSeriesLike

with simple aliases of upstream types.

This exposed a number of typing issues, pirmairly around pyspark.pandas API, which will be resolved in this PR.

Additionally it adds VirtusLab/pandas-stubs to CI dependencies.

Why are the changes needed?

Currently used Protocols where a workaround, included to improve typing coverage until Pandas exposes their type hints.

In a meantime, relatively stable stubs sources emerged (with a lot of ongoing discussions around) and pandas-stubs package (available on PyPI and conda-forge) provides better coverage without adding maintenance overhead on our side.

Does this PR introduce any user-facing change?

Better typing experience around Pandas UDFs.

How was this patch tested?

Existing typecheck pipeline and unit tests.

zero323 · 2021-12-17T00:18:02Z

FYI @HyukjinKwon

There is still a lot of work to done here (reduced typing errors from ~150 to <50 so far).

(Also, commit messages contain explanations why certain fixes are needed, I'll include these here once I am closer to completion).

SparkQA · 2021-12-17T00:53:43Z

Test build #146302 has finished for PR 34927 at commit 5958791.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-12-17T01:01:43Z

I like this!!

HyukjinKwon · 2021-12-17T01:01:54Z

cc @ueshin @itholic @xinrong-databricks FYI

SparkQA · 2021-12-17T01:22:23Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50774/

itholic · 2021-12-17T05:54:34Z

I've brief reviewed of the work, and so far looks pretty nice to me!!

Let me revisit and take a closer look when it's ready to review.

SparkQA · 2021-12-17T13:56:48Z

Test build #146332 has finished for PR 34927 at commit 3c5c894.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-17T14:19:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50807/

SparkQA · 2021-12-17T15:03:54Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50807/

SparkQA · 2021-12-17T19:54:06Z

Test build #146339 has finished for PR 34927 at commit e5043a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-17T20:18:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50813/

SparkQA · 2021-12-17T21:21:43Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50813/

zero323 · 2021-12-17T21:28:50Z

I've brief reviewed of the work, and so far looks pretty nice to me!!

Let me revisit and take a closer look when it's ready to review.

I'll ping you once I am closer to completion ‒ I resolved most of the minor problems, but hit a bigger issue with pandas_udf annotations ‒ seems like we'll have to rethink how this stuff is done, which is something I am not very happy about.

SparkQA · 2021-12-17T21:54:28Z

Test build #146341 has finished for PR 34927 at commit d81bfee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2021-12-17T22:15:21Z

We can remove the pandas entry from mypy.ini?
https://github.com/apache/spark/blob/d81bfee6b56ffffa3deac2ab14e8a88821998e77/python/mypy.ini#L119-L120

SparkQA · 2021-12-17T22:16:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50815/

SparkQA · 2021-12-17T23:01:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50815/

SparkQA · 2021-12-18T10:58:51Z

Test build #146356 has finished for PR 34927 at commit 045d587.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-18T11:07:44Z

Test build #146357 has finished for PR 34927 at commit 111b6cc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-18T11:16:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50830/

SparkQA · 2021-12-18T11:16:55Z

Test build #146358 has finished for PR 34927 at commit 535e3fb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-18T11:18:08Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50831/

SparkQA · 2021-12-18T11:31:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50832/

SparkQA · 2021-12-18T12:11:32Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50830/

Type cannot be more precise, because hole param is unbound

Issue with upstream annotations

Internal, not exposed in upstream hints

SparkQA · 2021-12-19T00:49:16Z

Test build #146370 has finished for PR 34927 at commit 47a727d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-12-19T01:19:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50844/

SparkQA · 2021-12-19T02:16:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50844/

SparkQA · 2021-12-21T15:49:07Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50920/

SparkQA · 2021-12-21T16:10:39Z

Test build #146445 has finished for PR 34927 at commit 1224d1e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait ExposesMetadataColumns extends LogicalPlan

SparkQA · 2021-12-21T16:44:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50920/

.github/workflows/build_and_test.yml

itholic

Thanks for the bunch of cleanup!

zero323 · 2021-12-23T00:16:31Z

Merged into master.

Thanks all!

### What changes were proposed in this pull request? This PR is a minor followup of #34927 that adds `pandas-stubs` dependency into `dev/requirements.txt`. ### Why are the changes needed? For easier development setup. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually tested as below: ```bash pip install -r dev/requirements.txt ``` Closes #35029 from HyukjinKwon/SPARK-34544. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

github-actions bot added CORE INFRA PYTHON SQL labels Dec 17, 2021

zero323 force-pushed the SPARK-34544 branch from e5043a5 to d81bfee Compare December 17, 2021 21:25

github-actions bot added the EXAMPLES label Dec 18, 2021

zero323 added 11 commits December 19, 2021 01:17

Ignore on SPARK-37678

94082c3

Type ignore Index.spark

d2b1960

Type cannot be more precise, because hole param is unbound

Ignore parse_dates incompatibility

a85868d

Issue with upstream annotations

Ignore columns reasign

35b3b1d

Ignore pandas __class_getitem__ assignments

41d5fc9

Ignore _all_kinds access

e05f7bf

Internal, not exposed in upstream hints

Drop Pandas Protocols tests

cdf15db

Fix udf annotations and examples

31d0ff0

Drop pandas ignores from mypy.ini

05eabdd

Add short description of purpose to pandas protocols

0e06a8d

Drop ignore[import] on pandas imports

47a727d

zero323 force-pushed the SPARK-34544 branch from 28684cd to 47a727d Compare December 19, 2021 00:18

zero323 marked this pull request as ready for review December 19, 2021 05:33

zero323 changed the title ~~[WIP][SPARK-34544][PYTHON] Convert PandasDataFrameLike and PandasSeriesLike to aliases of Pandas types~~ [SPARK-34544][PYTHON] Convert PandasDataFrameLike and PandasSeriesLike to aliases of Pandas types Dec 19, 2021

Merge branch 'master' into SPARK-34544

1224d1e

HyukjinKwon approved these changes Dec 22, 2021

View reviewed changes

xinrong-meng approved these changes Dec 22, 2021

View reviewed changes

itholic reviewed Dec 22, 2021

View reviewed changes

.github/workflows/build_and_test.yml Show resolved Hide resolved

itholic approved these changes Dec 22, 2021

View reviewed changes

zero323 closed this in a70006d Dec 23, 2021

zero323 deleted the SPARK-34544 branch December 23, 2021 00:16

HyukjinKwon mentioned this pull request Dec 27, 2021

[SPARK-34544][PYTHON][FOLLOW-UP] Add pandas-stubs in dev/requirement.txt #35029

Closed

Conversation

zero323 commented Dec 17, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

zero323 commented Dec 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

HyukjinKwon commented Dec 17, 2021

Uh oh!

HyukjinKwon commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

itholic commented Dec 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

zero323 commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

ueshin commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 17, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 18, 2021

Uh oh!

SparkQA commented Dec 19, 2021

Uh oh!

SparkQA commented Dec 19, 2021

Uh oh!

SparkQA commented Dec 19, 2021

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

SparkQA commented Dec 21, 2021

Uh oh!

Uh oh!

itholic left a comment

Choose a reason for hiding this comment

Uh oh!

zero323 commented Dec 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

zero323 commented Dec 17, 2021 •

edited

Loading

itholic commented Dec 17, 2021 •

edited

Loading