[SPARK-50220][PYTHON] Support listagg in PySpark #49231
Conversation
mikhailnik-db
changed the title
|
please also add new function into |
| ---------- | ||
| col : :class:`~pyspark.sql.Column` or column name | ||
| target column to compute on. | ||
| delimiter : :class:`~pyspark.sql.Column` or str, optional |
There was a problem hiding this comment.
| delimiter : :class:`~pyspark.sql.Column` or str, optional | |
| delimiter : :class:`~pyspark.sql.Column`, literal string or bytes, optional |
| ---------- | ||
| col : :class:`~pyspark.sql.Column` or column name | ||
| target column to compute on. | ||
| delimiter : :class:`~pyspark.sql.Column` or str, optional |
There was a problem hiding this comment.
| delimiter : :class:`~pyspark.sql.Column` or str, optional | |
| delimiter : :class:`~pyspark.sql.Column`, literal string or bytes, optional |
| # TODO(SPARK-50220): listagg functions will soon be added and removed from this list | ||
| "listagg_distinct", | ||
| "listagg", | ||
| "string_agg", |
There was a problem hiding this comment.
do we plan to add the aliases string_agg and string_agg_distinct?
There was a problem hiding this comment.
Yes, ofc. Just wanted to achieve green ci for listagg first
| col: "ColumnOrName", delimiter: Optional[Union[Column, str, bytes]] = None | ||
| ) -> Column: | ||
| if delimiter is None: | ||
| return _invoke_function_over_columns("listagg_distinct", col) |
There was a problem hiding this comment.
I suspect this won't work, since we don't have listagg_distinct in FunctionRegistry.
In spark connect, _invoke_function_over_columns just build a unresolved function.
There was a problem hiding this comment.
I think you can refer to
spark/python/pyspark/sql/connect/functions/builtin.py
Lines 1090 to 1096 in 05750de
mikhailnik-db
changed the title
mikhailnik-db
changed the title
|
thanks, merged to master |
What changes were proposed in this pull request?
Added new function
listaggto pyspark.Follow-up of #48748.
Why are the changes needed?
Allows to use native Python functions to write queries with
listagg. E.g.,df.select(F.listagg(df.value, ",").alias("r")).Does this PR introduce any user-facing change?
Yes, new functions
listaggandlistagg_distinct(with aliasesstring_aggandstring_agg_distinct) in pyspark.How was this patch tested?
Unit tests
Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot