Fixing parquet list of struct interpretation

[hyperbolic2346]

Description

This change alters how we interpret non-annotated data in a parquet file. Most modern parquet writers would produce something like:

message spark_schema {
  required int32 id;
  optional group phoneNumbers (LIST) {
    repeated group phone {
      required int64 number;
      optional binary kind (STRING);
    }
  }
}

But the list annotation isn't required. If it didn't exist, we would incorrectly interpret this schema as a struct of struct and not a list of struct. This change alters the code to look at the child and see if it is repeated. If it is, this indicates a list.

closes #13664

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[jlowe]

I tried testing this PR with the Spark plugin, but Apache Spark thinks the schema of the repeated_no_annotation.parquet file is:

root                                                                            
 |-- id: integer (nullable = true)
 |-- phoneNumbers: struct (nullable = true)
 |    |-- phone: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- number: long (nullable = true)
 |    |    |    |-- kind: string (nullable = true)

I think the issue is that there's no logical type annotation on phoneNumbers so we can't assume it's a list even though the type pattern underneath it looks list-like.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[hyperbolic2346]

This is really close now, but not 100%.

expected:

cudf::struct_view<cudf::list_view<cudf::struct_view<int64_t,cudf::string_view,>>,>:
Length : 6:
Null count: 2
111100
   cudf::list_view<cudf::struct_view<int64_t,cudf::string_view,>>:
   Length : 6
   Offsets(0x7f253c000600) : 0, 0, 0, 0, 1, 2, 5
   Null count: 2
   111100
      cudf::struct_view<int64_t,cudf::string_view,>:
      Length : 5:
         5555555555, 1111111111, 1111111111, 2222222222, 3333333333
         10110
         NULL, home, home, NULL, mobile

produced:

cudf::struct_view<cudf::list_view<cudf::struct_view<int64_t,cudf::string_view,>>,>:
Length : 6:
Null count: 2
111100
   cudf::list_view<cudf::struct_view<int64_t,cudf::string_view,>>:
   Length : 6
   Offsets(0x7f253c007000) : 0, 0, 0, 0, 1, 2, 5
   Null count: 3
   111000
      cudf::struct_view<int64_t,cudf::string_view,>:
      Length : 5:
         5555555555, 1111111111, 1111111111, 2222222222, 3333333333
         10110
         NULL, home, home, NULL, mobile

Note the null difference on the inner list.

[hyperbolic2346]

Spark integration tests show a few errors like java.lang.IllegalStateException: Logical error: no valid casts are found STRUCT to ByteType, Type conversion is not allowed from Table{columns=[ColumnVector{rows=1, type=LIST, nullCount=Optional.empty, offHeap=(ID: 435757 7f52640e7fa0)}], cudfTable=139991842717840, rows=1} to [MapType(StructType(StructField(first,StringType,true),StructField(middle,StringType,true),StructField(last,StringType,true)),StringType,true)] columns 0 to 1, and java.lang.IllegalStateException: Logical error: no valid casts are found STRUCT to BooleanType which make me think there may be some changes to how parquet normal files are being interpreted by cudf and not just the very old format.

More investigation is warranted here.

[hyperbolic2346]

Integration tests for spark are passing now and this is ready for review.

[harrism]

/ok to test

[hyperbolic2346]

/ok to test

[hyperbolic2346]

/ok to test

[vuule]

Looks good; few questions.

[vuule]

neat approach, localizes the madness 👍

[hyperbolic2346]

I think this could ultimately fix things like single-level list as well. I like this approach too.

[hyperbolic2346]

/ok to test

[hyperbolic2346]

/merge