-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-40820][PYTHON][SQL] Creating StructType from Json #43474
Conversation
b907846
to
9301bf0
Compare
9301bf0
to
0f5431b
Compare
@HyukjinKwon I have re-opened the PR, could you check please? |
@@ -0,0 +1,68 @@ | |||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's move the tests to DataTypeSuite.scala
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
moved!
@@ -1579,6 +1579,12 @@ def test_row_without_field_sorting(self): | |||
self.assertEqual(r, expected) | |||
self.assertEqual(repr(r), "Row(b=1, a=2)") | |||
|
|||
def test_struct_field_from_json(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def test_struct_field_from_json(self): | |
def test_struct_field_from_json(self): | |
# SPARK-40820: fromJson with only name and type |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed!
What changes were proposed in this pull request? Read schema from json without nullable and metadata Why are the changes needed? In order to read schema from json and avoid having to set implicit values Does this PR introduce any user-facing change? Yes, avoiding filling json with implicit values How was this patch tested? Unit tests
0f5431b
to
9b1dded
Compare
@HyukjinKwon could you check this, please? |
Merged to master. |
What changes were proposed in this pull request?
Read schema from json without nullable and metadata
Why are the changes needed?
In order to read schema from json and avoid having to set implicit values
Does this PR introduce any user-facing change?
Yes, avoiding filling json with implicit values
How was this patch tested?
Unit tests
When create a StructType from a Python dictionary you use StructType.fromJson or in scala DataType.fromJson
To create a schema can be created as follows from the code below, but it requires to put inside the json: Nullable and Metadata, this is inconsistent because within the DataType class this by default.
Python Error:
Scala Error: