Adding map type read support #914
Conversation
|
/gcbrun |
![sonatype-lift[bot]](https://avatars.githubusercontent.com/in/9843?s=60&v=4){kind=small}
...d_party/apache-spark/src/main/java/com/google/cloud/spark/bigquery/ArrowSchemaConverter.java
Outdated
Show resolved
Hide resolved
...igquery-connector-common/src/main/java/com/google/cloud/spark/bigquery/SchemaConverters.java
Show resolved
Hide resolved
...d_party/apache-spark/src/main/java/com/google/cloud/spark/bigquery/ArrowSchemaConverter.java
Outdated
Show resolved
Hide resolved
🛠 Lift Auto-fixSome of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.1 # Download the patch
curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/spark-bigquery-connector/914.diff -o lift-autofixes.diff
# Apply the patch with git
git apply lift-autofixes.diff
# Review the changes
git diffWant it all in a single command? Open a terminal in your project's directory and copy and paste the following command: curl https://lift.sonatype.com/api/patch/github.com/GoogleCloudDataproc/spark-bigquery-connector/914.diff | git applyOnce you're satisfied, commit and push your changes in your project. Footnotes |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
|
/gcbrun |
1 similar comment
|
/gcbrun |
|
/gcbrun |
| Field key = subFields.get("key"); | ||
| Field value = subFields.get("value"); | ||
| MapType mapType = DataTypes.createMapType(convert(key).dataType(), convert(value).dataType()); | ||
| return Optional.of(new StructField(field.getName(), mapType, /* nullable */ false, metadata)); |
There was a problem hiding this comment.
Why is nullable always false?
There was a problem hiding this comment.
Because it is a REPEATED field, not NULLABLE. There are some compromises we need to do as BigQuery has no native MAP type
| MapType mapType = DataTypes.createMapType(convert(key).dataType(), convert(value).dataType()); | ||
| return Optional.of(new StructField(field.getName(), mapType, /* nullable */ false, metadata)); | ||
| } catch (IllegalArgumentException e) { | ||
| // no "key" or "value" fields |
There was a problem hiding this comment.
Can we check if subfields contains "key" and "value" instead of try..catch?
There was a problem hiding this comment.
Good point, fixed that
|
/gcbrun |
|
/gcbrun |
This reverts commit fb1586e.
Currently reading ARRAY<STRUCT<key,value>> fields into Spark Map is supported. Write and documentation will be added in additional PR, in order to keep them (relatively) small