[FEA] Add support for reading JSON containing structs where rows are not consistent

[andygrove]

Is your feature request related to a problem? Please describe.

Given this input file:

{ "teacher": "Bob" }
{ "student": { "name": "Carol", "age":  21 } }
{ "teacher": "Bob", "student": { "name": "Carol", "age": 21 } }

Given this test for GpuJsonScan (in PR #10245 which adds support for reading structs in GpuJsonScan):

@pytest.mark.parametrize('filename', ['nested-structs.ndjson'])
@pytest.mark.parametrize('schema', [
    StructType([StructField('teacher', StringType())]),
    StructType([
        StructField('student', StructType([
            StructField('name', StringType()),
            StructField('age', IntegerType())
        ]))
    ]),
    StructType([
        StructField('teacher', StringType()),
        StructField('student', StructType([
            StructField('name', StringType()), 
            StructField('age', IntegerType())
        ]))
    ]),
])
@pytest.mark.parametrize('read_func', [read_json_df, read_json_sql])
@pytest.mark.parametrize('v1_enabled_list', ["", "json"])
def test_read_nested_struct(spark_tmp_table_factory, std_input_path, read_func, filename, schema, v1_enabled_list):
    conf = copy_and_update(_enable_all_types_conf, {'spark.sql.sources.useV1SourceList': v1_enabled_list})
    assert_gpu_and_cpu_are_equal_collect(
        read_func(std_input_path + '/' + filename,
                  schema,
                  spark_tmp_table_factory,
                  {}),
        conf=conf)

I see errors such as this:

Caused by: java.lang.AssertionError: Type conversion is not allowed from Table{columns=[ColumnVector{rows=3, type=STRING, nullCount=Optional.empty, offHeap=(ID: 22 7fc165d1cda0)}, ColumnVector{rows=3, type=STRUCT, nullCount=Optional.empty, offHeap=(ID: 23 7fc1640ea610)}], cudfTable=140468586241024, rows=3} to [StringType, StructType(StructField(name,StringType,true),StructField(age,IntegerType,true))] columns 0 to 2
	at com.nvidia.spark.rapids.GpuColumnVector.from(GpuColumnVector.java:674)
	at com.nvidia.spark.rapids.GpuColumnVector.from(GpuColumnVector.java:555)
	at com.nvidia.spark.rapids.GpuTextBasedPartitionReader.$anonfun$readBatch$3(GpuTextBasedPartitionReader.scala:279)
	at scala.Option.map(Option.scala:230)
	at com.nvidia.spark.rapids.GpuTextBasedPartitionReader.$anonfun$readBatch$1(GpuTextBasedPartitionReader.scala:279)

This is a similar issue with GpuJsonToStruct:

@pytest.mark.parametrize('schema', [
    'struct<teacher:string>',
    'struct<student:struct<name:string,age:int>>',
    'struct<teacher:string,student:struct<name:string,age:int>>'
])
@allow_non_gpu(*non_utc_allow)
def test_from_json_struct_of_struct(schema):
    json_string_gen = StringGen(r'{"teacher": "[A-Z]{1}[a-z]{2,5}",' \
                                r'"student": {"name": "[A-Z]{1}[a-z]{2,5}", "age": 1\d}}') \
        .with_special_pattern('', weight=50) \
        .with_special_pattern('null', weight=50) \
        .with_special_pattern('invalid_entry', weight=50)
    assert_gpu_and_cpu_are_equal_collect(
        lambda spark : unary_op_df(spark, json_string_gen) \
            .select(f.from_json('a', schema)),
        conf={"spark.rapids.sql.expression.JsonToStructs": True})

fails with:

Type conversion is not allowed from STRUCT(STRING,STRUCT(STRING,INT64)) to StructType(StructField(teacher,StringType,true),StructField(student,StructType(StructField(name,StringType,true),StructField(age,IntegerType,true)),true))

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.

[andygrove]

Root cause seems to be rapidsai/cudf#14864

[revans2]

I do not see these errors when I run the tests locally. These tests have been enabled for a long time too, so I suspect that #10542 fixed this, or at least worked around it enough that it is no longer a problem.