Ensure null values can be converted to BsonNull #89
Conversation
Fixes issue saving arrays / maps containing nulls Top level fields containing null are included when writing the document. This mirrors nulls and creating of rows. Users should filter data containing nulls, should they not want to save that data. SPARK-384 SPARK-351
|
A side effect of the fix for this ticket is that any fields with The benefit of this is now data can fully round trip when using the connector. Note: In 10.0.x null fields were automatically excluded and null values in lists / maps threw an error. |
| @@ -138,7 +140,7 @@ public BsonValue toBsonValue(final DataType dataType, final Object data) { | |||
| || DataTypes.TimestampType.acceptsType(dataType)) { | |||
| return new BsonDateTime(((Date) data).getTime()); | |||
| } else if (DataTypes.NullType.acceptsType(dataType)) { | |||
There was a problem hiding this comment.
So it wouldn't have fallen into this conditional, but instead would throw DataException below?
There was a problem hiding this comment.
In the case of data type Array<String> eg: ["a", null] the string value a would have been fine but it would have tried to convert a null value to string eg: new BsonString((String) data). That causes the BsonString constructor to throw an exception that is then caught and wrapped in a DataException below.
Fixes issue saving arrays / maps containing nulls
Top level fields containing null are included when writing the document. This mirrors nulls and creating of rows.
Users should filter data containing nulls, should they not want to save that data.
SPARK-384
SPARK-351