Add a build in CI for pyspark 3.0 #521
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #521 +/- ##
==========================================
+ Coverage 86.18% 86.24% +0.05%
==========================================
Files 81 81
Lines 4467 4471 +4
Branches 717 718 +1
==========================================
+ Hits 3850 3856 +6
+ Misses 505 503 -2
Partials 112 112
Continue to review full report at Codecov.
|
|
@liangz1 |
mengxr
suggested changes
Mar 30, 2020
liangz1
changed the title
petastorm/unischema.py
Outdated
Comment on lines
393
to
395
| schema_field_indices = {field_name: i for i, field_name in enumerate(unischema.fields)} | ||
| sorted_dict = OrderedDict(sorted(encoded_dict.items(), key=lambda item: schema_field_indices[item[0]])) | ||
| return pyspark.Row(**sorted_dict) |
There was a problem hiding this comment.
Sorted_dict still cannot guarantee the result correct. Python<3.6 does not guarantee the kwargs key order.
See https://issues.apache.org/jira/browse/SPARK-29748
Replace the 3 lines with:
field_list = list(unischema.fields.keys())
# generate a value list which match the schema column order.
value_list = [encoded_dict[name] for name in field_list]
# create a row by value list
row = pyspark.Row(*value_list)
# set row fields
row.__fields__ = field_list
return rowCC @mengxr
|
Summary:
|
WeichenXu123
approved these changes
Mar 31, 2020
tkakantousis
pushed a commit
to logicalclocks/petastorm
that referenced
this pull request
Sep 16, 2020
* This PR add latest spark master version test in CI. * Fix `dict_to_spark_row` method, See uber#521 (comment) * For spark<3.0 and pyarrow>=0.15, add ARROW_PRE_0_15_IPC_FORMAT=1 in CI, and remove unnecessary skipif in test. See explanation in https://spark.apache.org/docs/3.0.0-preview/sql-pyspark-pandas-with-arrow.html#compatibiliy-setting-for-pyarrow--0150-and-spark-23x-24x Co-authored-by: WeichenXu <weichen.xu@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The pypi package is not released yet. Once it is available, I'll update the code.