[SPARK-47211][CONNECT][PYTHON] Fix ignored PySpark Connect string collation #45316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -3397,6 +3397,14 @@ def test_df_caache(self): | |||
| self.assert_eq(10, df.count()) | |||
| self.assertTrue(df.is_cached) | |||
|
|
|||
| def test_collated_string(self): | |||
There was a problem hiding this comment.
Shall we add the test to pyspark.sql.tests.test_types.TypesTestsMixin, so that pyspark.sql.tests.connect.test_parity_types can inherit automatically?
There was a problem hiding this comment.
@xinrong-meng Moved the test and added to test_parity_types, please check.
There was a problem hiding this comment.
Just fyi we don't need to add the test to test_parity_types, it will inherit automatically.
Thanks @HyukjinKwon for the fix.
amaliujia
reviewed
Feb 28, 2024
| @@ -139,7 +139,7 @@ def pyspark_types_to_proto_types(data_type: DataType) -> pb2.DataType: | |||
| if isinstance(data_type, NullType): | |||
| ret.null.CopyFrom(pb2.DataType.NULL()) | |||
| elif isinstance(data_type, StringType): | |||
| ret.string.CopyFrom(pb2.DataType.String()) | |||
| ret.string.collation_id = data_type.collationId | |||
There was a problem hiding this comment.
Is this change tested? The added UT seems to test proto -> type conversion but not vice verse?
There was a problem hiding this comment.
@amaliujia New changes should cover conversions in both directions (both get invoked during testing), please check.
HyukjinKwon
reviewed
Feb 29, 2024
| @@ -86,6 +86,9 @@ def test_rdd_with_udt(self): | |||
| def test_udt(self): | |||
| super().test_udt() | |||
|
|
|||
| def test_collated_string(self): | |||
There was a problem hiding this comment.
We can actually just remove this. It will automatically run the tests (by inheritance)
HyukjinKwon
approved these changes
Feb 29, 2024
HyukjinKwon
reviewed
Feb 29, 2024
|
@nikolamand-db seems you need to enable the Github Action? |
I'm having trouble with Github Actions, it's disabled for my account. Already filed a ticket with Github support. |
xinrong-meng
approved these changes
Feb 29, 2024
|
Merged to master. |
TakawaAkirayo
pushed a commit
to TakawaAkirayo/spark
that referenced
this pull request
Mar 4, 2024
…lation
### What changes were proposed in this pull request?
When using Connect with PySpark, string collation silently gets dropped:
```
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> spark.sql("select 'abc' collate 'UNICODE'")
DataFrame[collate(abc): string]
>>> from pyspark.sql.types import StructType, StringType, StructField
>>> spark.createDataFrame([], StructType([StructField('id', StringType(2))]))
DataFrame[id: string]
```
Instead of `string` type in dataframe, we should be seeing `string COLLATE 'UNICODE'`.
Fix the issue by adding collation info to conversions under connect `UnparsedDataType`.
### Why are the changes needed?
To enable correct behavior with PySpark connect collated strings.
### Does this PR introduce _any_ user-facing change?
Yes, with these changes the user gets correct string collation from PySpark connect session.
### How was this patch tested?
Added test which simulates the described scenario.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes apache#45316 from nikolamand-db/nikolamand-db/SPARK-47211.
Lead-authored-by: Nikola Mandic <nikola.mandic@databricks.com>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
ericm-db
pushed a commit
to ericm-db/spark
that referenced
this pull request
Mar 5, 2024
…lation
### What changes were proposed in this pull request?
When using Connect with PySpark, string collation silently gets dropped:
```
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> spark.sql("select 'abc' collate 'UNICODE'")
DataFrame[collate(abc): string]
>>> from pyspark.sql.types import StructType, StringType, StructField
>>> spark.createDataFrame([], StructType([StructField('id', StringType(2))]))
DataFrame[id: string]
```
Instead of `string` type in dataframe, we should be seeing `string COLLATE 'UNICODE'`.
Fix the issue by adding collation info to conversions under connect `UnparsedDataType`.
### Why are the changes needed?
To enable correct behavior with PySpark connect collated strings.
### Does this PR introduce _any_ user-facing change?
Yes, with these changes the user gets correct string collation from PySpark connect session.
### How was this patch tested?
Added test which simulates the described scenario.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes apache#45316 from nikolamand-db/nikolamand-db/SPARK-47211.
Lead-authored-by: Nikola Mandic <nikola.mandic@databricks.com>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
jpcorreia99
pushed a commit
to jpcorreia99/spark
that referenced
this pull request
Mar 12, 2024
…lation
### What changes were proposed in this pull request?
When using Connect with PySpark, string collation silently gets dropped:
```
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> spark.sql("select 'abc' collate 'UNICODE'")
DataFrame[collate(abc): string]
>>> from pyspark.sql.types import StructType, StringType, StructField
>>> spark.createDataFrame([], StructType([StructField('id', StringType(2))]))
DataFrame[id: string]
```
Instead of `string` type in dataframe, we should be seeing `string COLLATE 'UNICODE'`.
Fix the issue by adding collation info to conversions under connect `UnparsedDataType`.
### Why are the changes needed?
To enable correct behavior with PySpark connect collated strings.
### Does this PR introduce _any_ user-facing change?
Yes, with these changes the user gets correct string collation from PySpark connect session.
### How was this patch tested?
Added test which simulates the described scenario.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes apache#45316 from nikolamand-db/nikolamand-db/SPARK-47211.
Lead-authored-by: Nikola Mandic <nikola.mandic@databricks.com>
Co-authored-by: Hyukjin Kwon <gurwls223@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When using Connect with PySpark, string collation silently gets dropped:
Instead of
stringtype in dataframe, we should be seeingstring COLLATE 'UNICODE'.Fix the issue by adding collation info to conversions under connect
UnparsedDataType.Why are the changes needed?
To enable correct behavior with PySpark connect collated strings.
Does this PR introduce any user-facing change?
Yes, with these changes the user gets correct string collation from PySpark connect session.
How was this patch tested?
Added test which simulates the described scenario.
Was this patch authored or co-authored using generative AI tooling?
No.