Skip to content

Comments

[SPARK-37415][PYTHON][ML] Inline type hints for pyspark.ml.util#35367

Closed
zero323 wants to merge 3 commits intoapache:masterfrom
zero323:SPARK-37415
Closed

[SPARK-37415][PYTHON][ML] Inline type hints for pyspark.ml.util#35367
zero323 wants to merge 3 commits intoapache:masterfrom
zero323:SPARK-37415

Conversation

@zero323
Copy link
Member

@zero323 zero323 commented Jan 30, 2022

What changes were proposed in this pull request?

This PR migrates type pyspark.ml.util annotations from stub file to inline type hints.

Why are the changes needed?

Part of ongoing migration of type hints.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

@zero323
Copy link
Member Author

zero323 commented Jan 30, 2022

This is just a rough draft ‒ I'll revisit it once SPARK-37397 (and revise it to address any possible issues detected during parallel work on SPARK-37416).

@zero323
Copy link
Member Author

zero323 commented Feb 3, 2022

Thanks for the review @HyukjinKwon! I am still working on this one (there was an issue with some of the Generic positions, that caused MRO failures) and won't make it final until I have time to finalize ml.wrapper ‒ because we have crazy inheritance hierarchies with a lot of abstract things, making small changes tends to escalate all over the place (ml.wrapper, which is not big by itself, might require changes in stubs for all ml modules 😰, because of the abstract _things exposed to type checkers)

@zero323 zero323 changed the title [WIP][SPARK-37415][PYTHON][ML] Inline type hints for pyspark.ml.util [SPARK-37415][PYTHON][ML] Inline type hints for pyspark.ml.util Feb 4, 2022
@zero323 zero323 marked this pull request as ready for review February 4, 2022 22:15
@zero323
Copy link
Member Author

zero323 commented Feb 4, 2022

Run the test run with #35399 and both seem to work well together.

@zero323 zero323 closed this in 0084a86 Feb 6, 2022
@zero323
Copy link
Member Author

zero323 commented Feb 6, 2022

Merged into master.

Thanks!

@zero323 zero323 deleted the SPARK-37415 branch February 6, 2022 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants