New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-41222][CONNECT][PYTHON] Unify the typing definitions #38757
Conversation
d16476b
to
84ff08b
Compare
python/pyspark/sql/connect/column.py
Outdated
|
||
import json | ||
import decimal | ||
import datetime | ||
|
||
import pyspark.sql.connect.proto as proto | ||
from pyspark.sql.connect._typing import PrimitiveType | ||
|
||
primitive_types = (bool, float, int, str) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
list the types here to avoid a circular import
, since _typing
also import column
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can import this in the function in that case. e.g.)
def _bin_op(
name: str, doc: str = "binary function", reverse: bool = False
) -> Callable[["Column", Any], "Expression"]:
+ from pyspark.sql.connect._typing import PrimitiveType
def _(self: "Column", other: Any) -> "Expression":
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks, let me update
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM thanks!
merged into master |
### What changes were proposed in this pull request? 1, remove `__init__.py` 2, rename `ColumnOrString ` as `ColumnOrName` to be the same as pyspark ### Why are the changes needed? 1, there are two typing files now: `_typing.py` and `__init__.py`, they are used in different files, which is very confusing; 2, the definitions of `LiteralType` are different, the old one in `_typing.py` was never used 3, both `ColumnOrString ` and `ColumnOrName` are used now; ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing UTs Closes apache#38757 from zhengruifeng/connect_typing. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
### What changes were proposed in this pull request? 1, remove `__init__.py` 2, rename `ColumnOrString ` as `ColumnOrName` to be the same as pyspark ### Why are the changes needed? 1, there are two typing files now: `_typing.py` and `__init__.py`, they are used in different files, which is very confusing; 2, the definitions of `LiteralType` are different, the old one in `_typing.py` was never used 3, both `ColumnOrString ` and `ColumnOrName` are used now; ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? existing UTs Closes apache#38757 from zhengruifeng/connect_typing. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
What changes were proposed in this pull request?
1, remove
__init__.py
2, rename
ColumnOrString
asColumnOrName
to be the same as pysparkWhy are the changes needed?
1, there are two typing files now:
_typing.py
and__init__.py
, they are used in different files, which is very confusing;2, the definitions of
LiteralType
are different, the old one in_typing.py
was never used3, both
ColumnOrString
andColumnOrName
are used now;Does this PR introduce any user-facing change?
No
How was this patch tested?
existing UTs