[SPARK-56518][PYTHON] Add current_path to PySpark functions#55379
Closed
srielau wants to merge 2 commits into
Closed
[SPARK-56518][PYTHON] Add current_path to PySpark functions#55379srielau wants to merge 2 commits into
srielau wants to merge 2 commits into
Conversation
srielau
changed the title
HyukjinKwon
changed the title
HyukjinKwon
reviewed
Apr 16, 2026
| "bitmap_count", | ||
| "current_catalog", | ||
| "current_database", | ||
| "current_path", |
Member
There was a problem hiding this comment.
would also have to be added in https://github.com/apache/spark/blob/master/python/docs/source/reference/pyspark.sql/functions.rst
Contributor
|
We need this to recover our CI. |
Member
|
Merged to master. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add the
current_path()function to PySpark to match the JVM-sidecurrent_path()added in SPARK-56489.The following files were updated:
python/pyspark/sql/functions/builtin.py— added thecurrent_path()function definition with docstring.python/pyspark/sql/functions/__init__.py— addedcurrent_pathto__all__.python/pyspark/sql/connect/functions/builtin.py— added the Spark Connect stub.Why are the changes needed?
SPARK-56489 added
CURRENT_PATH()as a builtin SQL expression and registered it inorg.apache.spark.sql.functions, but the corresponding PySpark function was not added. This causestest_function_parityintest_functions.pyto fail because the test checks that every function in the JVMfunctionsobject has a corresponding PySpark function (or is explicitly excluded).Does this PR introduce any user-facing change?
Yes.
pyspark.sql.functions.current_path()is now available and returns the current SQL path as a comma-separated list of qualified schema names (e.g.system.builtin,system.session,spark_catalog.default). This is a new function on the unreleased master branch only.How was this patch tested?
The existing
test_function_paritytest inpython/pyspark/sql/tests/test_functions.pycovers this change — it will now pass sincecurrent_pathis no longer missing from PySpark.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.6)