-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-48068][PYTHON] mypy
should have --python-executable
parameter
#46314
Conversation
Could you review this, @HyukjinKwon ? |
mypy
failure in Python 3.10mypy
failure in Python 3.10+
mypy
failure in Python 3.10+mypy
failure in Python 3.10 and 3.11
Although I verified it locally with Python 3.9, CI seems to behave differently. Let me check more.
|
I have the same problem in my local but I think it's because of the Python version we're using |
Maybe we should just upgrade our Python version in CI. |
Thank you, @HyukjinKwon . Ya, let me upgrade it, @HyukjinKwon . |
Since Infra Image is using
|
mypy
failure in Python 3.10 and 3.11mypy
should have --python-executable
parameter
All linters passed. Could you review this PR, @viirya ? |
Could you review this PR, @huaxingao ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @dongjoon-hyun
Thank you, @huaxingao ! |
### What changes were proposed in this pull request? This PR aims to fix `mypy` failure by propagating `lint-python`'s `PYTHON_EXECUTABLE` to `mypy`'s parameter correctly. ### Why are the changes needed? We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not always guaranteed. We need to use `mypy`'s parameter to make it sure. https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705 This patch is useful whose `python3` chooses one of multiple Python installation like our CI environment. ``` $ docker run -it --rm ghcr.io/apache/apache-spark-ci-image:master-8905641334 bash WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested root2ef6ce08d2c4:/# python3 --version Python 3.10.12 root2ef6ce08d2c4:/# python3.9 --version Python 3.9.19 ``` For example, the following shows that `PYTHON_EXECUTABLE` is not considered by `mypy`. ``` root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 3428 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46314 from dongjoon-hyun/SPARK-48068. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 26c871f) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
### What changes were proposed in this pull request? This PR aims to fix `mypy` failure by propagating `lint-python`'s `PYTHON_EXECUTABLE` to `mypy`'s parameter correctly. ### Why are the changes needed? We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not always guaranteed. We need to use `mypy`'s parameter to make it sure. https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705 This patch is useful whose `python3` chooses one of multiple Python installation like our CI environment. ``` $ docker run -it --rm ghcr.io/apache/apache-spark-ci-image:master-8905641334 bash WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested root2ef6ce08d2c4:/# python3 --version Python 3.10.12 root2ef6ce08d2c4:/# python3.9 --version Python 3.9.19 ``` For example, the following shows that `PYTHON_EXECUTABLE` is not considered by `mypy`. ``` root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 3428 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46314 from dongjoon-hyun/SPARK-48068. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> (cherry picked from commit 26c871f) Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Looks good to me. Thanks @dongjoon-hyun |
Thank you, @viirya ! |
### What changes were proposed in this pull request? This PR aims to fix `mypy` failure by propagating `lint-python`'s `PYTHON_EXECUTABLE` to `mypy`'s parameter correctly. ### Why are the changes needed? We assumed that `PYTHON_EXECUTABLE` is used for `dev/lint-python` like the following. That's not always guaranteed. We need to use `mypy`'s parameter to make it sure. https://github.com/apache/spark/blob/ff401dde50343c9bbc1c49a0294272f2da7d01e2/.github/workflows/build_and_test.yml#L705 This patch is useful whose `python3` chooses one of multiple Python installation like our CI environment. ``` $ docker run -it --rm ghcr.io/apache/apache-spark-ci-image:master-8905641334 bash WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested root2ef6ce08d2c4:/# python3 --version Python 3.10.12 root2ef6ce08d2c4:/# python3.9 --version Python 3.9.19 ``` For example, the following shows that `PYTHON_EXECUTABLE` is not considered by `mypy`. ``` root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --python-executable=python3.11 --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 3428 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.9 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 root18c8eae5791e:/spark# PYTHON_EXECUTABLE=python3.11 mypy --namespace-packages --config-file python/mypy.ini python/pyspark | wc -l 1 ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46314 from dongjoon-hyun/SPARK-48068. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
What changes were proposed in this pull request?
This PR aims to fix
mypy
failure by propagatinglint-python
'sPYTHON_EXECUTABLE
tomypy
's parameter correctly.Why are the changes needed?
We assumed that
PYTHON_EXECUTABLE
is used fordev/lint-python
like the following. That's not always guaranteed. We need to usemypy
's parameter to make it sure.spark/.github/workflows/build_and_test.yml
Line 705 in ff401dd
This patch is useful whose
python3
chooses one of multiple Python installation like our CI environment.For example, the following shows that
PYTHON_EXECUTABLE
is not considered bymypy
.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.
Was this patch authored or co-authored using generative AI tooling?
No.