[SPARK-50311][PYTHON] Support (add|remove|get|clear)Tag(s) APIs for PySpark
#48843
Conversation
itholic
changed the title
(add|remove|get|clear)Tag(s) for PySpark(add|remove|get|clear)Tag(s) APIs for PySpark
| from pyspark.testing.connectutils import ReusedConnectTestCase | ||
|
|
||
|
|
||
| class JobCancellationParityTests(JobCancellationTestsMixin, ReusedConnectTestCase): |
There was a problem hiding this comment.
Moved this suite from pyspark.sql.tests.connect.test_session and integrated with JobCancellationTests
|
LGTM thank you! |
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Actually I think you should directly set local properties via calling addTag etc instead of having Python thread local.
Python thread <> JVM thread mappings are maintained so using this approach should b efine.
you can throw an exception if PYSPARK_PIN_THREAD is enabled.
Oh, if we can just throw an exception for |
|
I think we might need resolve the bug from Scala side first. |
|
Merged to master. |
### What changes were proposed in this pull request? This PR followups #48843 to remove remote_only decorator from supported API and update the test ### Why are the changes needed? To check Connect parity properly ### Does this PR introduce _any_ user-facing change? No, it's test-only ### How was this patch tested? Updated the existing test ### Was this patch authored or co-authored using generative AI tooling? No Closes #49012 from itholic/SPARK-50311-followup. Authored-by: Haejoon Lee <haejoon.lee@databricks.com> Signed-off-by: Haejoon Lee <haejoon.lee@databricks.com>
### What changes were proposed in this pull request? This is minor followup for #48843 to fix test name properly from Python file ### Why are the changes needed? Use proper test name ### Does this PR introduce _any_ user-facing change? No, it's test only ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No Closes #49199 from itholic/SPARK-50311-followup-2. Authored-by: Haejoon Lee <haejoon.lee@databricks.com> Signed-off-by: Haejoon Lee <haejoon.lee@databricks.com>
3 participants
What changes were proposed in this pull request?
This PR proposes to support
(add|remove|get|clear)Tag(s)for PySparkWhy are the changes needed?
For improving the compatibility between Spark Connect and Spark Classic. These tag-related APIs are currently only supported from Spark Connect.
Does this PR introduce any user-facing change?
New APIs are added
How was this patch tested?
Added UTs
Was this patch authored or co-authored using generative AI tooling?
No