[BUG][JSON] from_json to Map type should produce null for invalid entries

[andygrove]

Describe the bug
When converting JSON to Map<String,String> using from_json, invalid entries are returned as empty structs instead of nulls.

Steps/Code to reproduce bug

val df = Seq("{}", "BAD", "{\"A\": 100}").toDF.repartition(2)
df.selectExpr("from_json(value, 'MAP<STRING,STRING>')").show()

CPU Output

+----------+                                                                    
|   entries|
+----------+
|        {}|
|      null|
|{A -> 100}|
+----------+

GPU Output

+----------+
|   entries|
+----------+
|        {}|
|        {}|
|{A -> 100}|
+----------+

Expected behavior
Results should match CPU.

Environment details (please complete the following information)
N/A

Additional context
None

[andygrove]

I ran a test in spark-rapids-jni:

ColumnVector input = ColumnVector.fromStrings("{}", "BAD", "{\"A\": 100}");
ColumnVector outputMap = MapUtils.extractRawMapFromJsonString(input);
TableDebug.get().debug("outputMap", outputMap);

This shows the following output:

GPU COLUMN outputMap - NC: 0 DATA: null VAL: null
GPU COLUMN outputMap:DATA - NC: 0 DATA: null VAL: null
GPU COLUMN outputMap:DATA:CHILD_0 - NC: 0 DATA: DeviceMemoryBufferView{address=0x7f48a3c01200, length=1, id=-1} VAL: null
GPU COLUMN outputMap:DATA:CHILD_1 - NC: 0 DATA: DeviceMemoryBufferView{address=0x7f48a3c01a00, length=3, id=-1} VAL: null
COLUMN outputMap - LIST
OFFSETS
0 [0 - 0)
1 [0 - 0)
2 [0 - 1)
COLUMN outputMap:DATA - STRUCT
COLUMN outputMap:DATA:CHILD_0 - STRING
0 "A" 41
COLUMN outputMap:DATA:CHILD_1 - STRING
0 "100" 313030

There is no differentiation between the source strings {} and BAD. None of the values are null.

@ttnghia Would it be possible to detect the invalid item and set the corresponding entry to null?

[ttnghia]

Yes that is possible but it seems not trivial at this point because the underlying JNI code doesn't work based on lines JSON (it concatenates all the input rows into one giant JSON string).

We need to rework it to have this fixed.