-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-43928][SQL][PYTHON][CONNECT] Add bit operations to Scala, Python and Connect API #41608
Conversation
python/pyspark/sql/functions.py
Outdated
| 1| | ||
| 1| | ||
+------------+ | ||
<BLANKLINE> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's don't add <BLANKLINE>
see #41610
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK
@@ -1829,6 +1829,112 @@ def bitwise_not(col: "ColumnOrName") -> Column: | |||
return _invoke_function_over_columns("bitwise_not", col) | |||
|
|||
|
|||
@try_remote_functions | |||
def bit_count(col: "ColumnOrName") -> Column: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should also add them to *.rst files for Python API reference documentation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry. I forgot it.
@beliefer oh, please rebase to resolve the conflicts. |
@beliefer would you mind re-triggering the failed test? |
the failed tests should be unrelated |
merged to master, thank you @beliefer so much! |
@zhengruifeng @HyukjinKwon Thank you! |
…on and Connect API ### What changes were proposed in this pull request? This PR want add bit operations to Scala, Python and Connect API. These API show below. - bit_and - bit_count - bit_get - bit_or - bit_xor - getbit ### Why are the changes needed? Add bit operations to Scala, Python and Connect API ### Does this PR introduce _any_ user-facing change? 'No'. New feature. ### How was this patch tested? New test cases. Closes apache#41608 from beliefer/SPARK-43928. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
https://github.com/apache/spark/actions/runs/5296358122/jobs/9587419094
|
@LuciferYang Let me fix it. |
…tions` of `DataFrameFunctionsSuite` ### What changes were proposed in this pull request? #41608 add bit operations to Scala, Python and Connect API, but missing the test for `DataFrameFunctionsSuite`. ### Why are the changes needed? Fix missing tests ### Does this PR introduce _any_ user-facing change? 'No'. New feature. ### How was this patch tested? N/A Closes #41640 from beliefer/SPARK-43928_followup. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
…on and Connect API ### What changes were proposed in this pull request? This PR want add bit operations to Scala, Python and Connect API. These API show below. - bit_and - bit_count - bit_get - bit_or - bit_xor - getbit ### Why are the changes needed? Add bit operations to Scala, Python and Connect API ### Does this PR introduce _any_ user-facing change? 'No'. New feature. ### How was this patch tested? New test cases. Closes apache#41608 from beliefer/SPARK-43928. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
…tions` of `DataFrameFunctionsSuite` ### What changes were proposed in this pull request? apache#41608 add bit operations to Scala, Python and Connect API, but missing the test for `DataFrameFunctionsSuite`. ### Why are the changes needed? Fix missing tests ### Does this PR introduce _any_ user-facing change? 'No'. New feature. ### How was this patch tested? N/A Closes apache#41640 from beliefer/SPARK-43928_followup. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Ruifeng Zheng <ruifengz@apache.org>
What changes were proposed in this pull request?
This PR want add bit operations to Scala, Python and Connect API.
These API show below.
Why are the changes needed?
Add bit operations to Scala, Python and Connect API
Does this PR introduce any user-facing change?
'No'.
New feature.
How was this patch tested?
New test cases.