"out of order projection is not supported" after Fix Parquet Arrow Schema Inference

[alamb]

Describe the bug
After #1682 from @tustvold some tests in datafusion begin to fail with

"out of order projection is not supported" after Fix Parquet Arrow Schema Inference

Note this code is not included in 14.0.0

To Reproduce
See the reproduction instructions on apache/datafusion#2530

Expected behavior
tests should pass

Additional context
Add any other context about the problem here.

[tustvold]

This is expected behaviour, DataFusion currently passed out of order projection to the parquet reader despite the ArrayReader having never supported it. We now complain explicitly that the reader does not support this, but imo this is better than silently ignoring it?

[alamb]

despite the ArrayReader having never supported it.

What happened if the reader was passed out of order projections? Was data ignored?

[tustvold]

It silently ignored the out-of-order projection, i.e.

#[test]
    fn test_out_of_order_projection() {
        let testdata = arrow::util::test_util::parquet_test_data();
        let path = format!("{}/alltypes_plain.parquet", testdata);
        let file = File::open(&path).unwrap();
        let reader = SerializedFileReader::try_from(file).unwrap();
        let expected_rows = reader.metadata().file_metadata().num_rows() as usize;

        let mut arrow_reader = ParquetFileArrowReader::new(Arc::new(reader));
        let b1 = arrow_reader.get_record_reader_by_columns([0, 1], 2).unwrap();

        let b2 = arrow_reader.get_record_reader_by_columns([1, 0], 2).unwrap();

        assert_eq!(b1.schema, b2.schema);
    }

Would pass.

To add to the mess

#[test]
    fn test_out_of_order_projection() {
        let testdata = arrow::util::test_util::parquet_test_data();
        let path = format!("{}/alltypes_plain.parquet", testdata);
        let file = File::open(&path).unwrap();
        let reader = SerializedFileReader::try_from(file).unwrap();
        let expected_rows = reader.metadata().file_metadata().num_rows() as usize;

        let mut arrow_reader = ParquetFileArrowReader::new(Arc::new(reader));
        let b1 = arrow_reader.get_schema_by_columns([0, 1], true).unwrap();
        let b2 = arrow_reader.get_schema_by_columns([1, 0], true).unwrap();

        assert_eq!(b1, b2);
    }

Would not pass, as it sort of understood column projection - although it was completely broken for nested types

[alamb]

Do we need to make any changes to DataFusion?

[tustvold]

Not beyond updating it to use the new ProjectionMask within ParquetExec, so changing from column_reader_with_columns(iter) to column_reader_with_columns(ProjectionMask::roots(iter))

[alamb]

Ok, Maybe I can find time to do so as part of preparing the arrow 15 release

[tustvold]

I'd be happy to help if you run into any roadblocks, it should be straightforward

[alamb]

Update: @tustvold has kindly offered to take a stab at preparing a datafusion PR to upgrade to latest arrow, prior to #1727

[tustvold]

PR here - apache/datafusion#2631