Skip to content

[SPARK-48646][PYTHON] Refine Python data source API docstring and type hints#47003

Closed
allisonwang-db wants to merge 2 commits intoapache:masterfrom
allisonwang-db:spark-48646-ds-typing
Closed

[SPARK-48646][PYTHON] Refine Python data source API docstring and type hints#47003
allisonwang-db wants to merge 2 commits intoapache:masterfrom
allisonwang-db:spark-48646-ds-typing

Conversation

@allisonwang-db
Copy link
Contributor

What changes were proposed in this pull request?

This PR improves the docstring and type hints for classes and methods in datasource.py.

Why are the changes needed?

To improve type hint and docstring.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing docs test.

Was this patch authored or co-authored using generative AI tooling?

No

@allisonwang-db
Copy link
Contributor Author

cc @HyukjinKwon

@HyukjinKwon
Copy link
Member

Let's fix MyPy failures though:

annotations failed mypy checks:
python/pyspark/sql/worker/python_streaming_sink_runner.py:118: error: Unused "type: ignore" comment  [unused-ignore]
python/pyspark/sql/worker/python_streaming_sink_runner.py:120: error: Unused "type: ignore" comment  [unused-ignore]
python/pyspark/sql/worker/commit_data_source_write.py:93: error: Unused "type: ignore" comment  [unused-ignore]
python/pyspark/sql/worker/commit_data_source_write.py:95: error: Unused "type: ignore" comment  [unused-ignore]

@xinrong-meng
Copy link
Member

LGTM after tests passed, thank you!

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants