Skip to content

[SPARK-56765][PYTHON] Suppress pyarrow.compute mypy attr-defined errors#55742

Closed
201573 wants to merge 1 commit into
apache:masterfrom
201573:codex/56765-pyarrow-compute-mypy
Closed

[SPARK-56765][PYTHON] Suppress pyarrow.compute mypy attr-defined errors#55742
201573 wants to merge 1 commit into
apache:masterfrom
201573:codex/56765-pyarrow-compute-mypy

Conversation

@201573
Copy link
Copy Markdown

@201573 201573 commented May 7, 2026

What changes were proposed in this pull request?

This PR adds narrow # type: ignore[attr-defined] annotations for pyarrow.compute calls whose symbols are missing from the PyArrow 24.0.0 stubs: floor_temporal, assume_timezone, and local_timestamp.

Why are the changes needed?

PyArrow 24.0.0 still exposes these compute kernels at runtime, but its type stubs do not expose them to mypy. This currently breaks PySpark lint with attr-defined errors.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

  • PYTHON_EXECUTABLE=/tmp/spark-56765-mypy-venv/bin/python ./dev/lint-python --compile
  • /tmp/spark-56765-mypy-venv/bin/ruff format --check python/pyspark/sql/pandas/types.py python/pyspark/sql/conversion.py
  • git diff --check
  • Targeted mypy with pyarrow==24.0.0 confirmed these pyarrow.compute attr-defined errors are suppressed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

@LuciferYang LuciferYang requested a review from zhengruifeng May 7, 2026 16:37
@LuciferYang
Copy link
Copy Markdown
Contributor

cc @zhengruifeng as discussed offline, the daily test failures on branch-4.x/branch-4.2/branch-4.1 are likely related to this issue.

@201573 201573 force-pushed the codex/56765-pyarrow-compute-mypy branch from dfb4378 to 6009085 Compare May 7, 2026 17:04
Copy link
Copy Markdown
Contributor

@devin-petersohn devin-petersohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gaogaotiantian
Copy link
Copy Markdown
Contributor

gaogaotiantian commented May 7, 2026

I think the solution in #55744 is better. We should try to avoid ignoring unused-ignore. This is an issue for pyarrow.compute and we should just ignore that specific module.

@LuciferYang
Copy link
Copy Markdown
Contributor

+1, agree with @gaogaotiantian

@201573
Copy link
Copy Markdown
Author

201573 commented May 8, 2026

Thanks for the feedback. I agree that #55744 is cleaner since it keeps the suppression scoped to pyarrow.compute in mypy.ini instead of adding unused-ignore at the call sites.

I'll close this in favor of #55744.

@201573 201573 closed this May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants