Add schema to parquet writer options

[JkSelf]

Here is the schema information for the TPCH supplier table.

root
 |-- s_suppkey: long (nullable = true)  
 |-- s_name: string (nullable = true) 
 |-- s_address: string (nullable = true)  
 |-- s_nationkey: long (nullable = true)  
 |-- s_phone: string (nullable = true)  
 |-- s_acctbal: double (nullable = true)  
 |-- s_comment: string (nullable = true)  

When we write the supplier table into a parquet file using velox parquet writer in Gluten, we encounter the incorrect schema issue.

root
 |-- n0_0: long (nullable = true)
 |-- n0_1: string (nullable = true)
 |-- n0_2: string (nullable = true)
 |-- n0_3: long (nullable = true)
 |-- n0_4: string (nullable = true)
 |-- n0_5: double (nullable = true)
 |-- n0_6: string (nullable = true)

The reason for this is that the schema is inferred from the record batch. To address this, this PR introduces a schema into the WriterOption. This allows us to pass the appropriate schema to the Parquet writer.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	5cec6cf
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/6580e83cdb740a0008820fe1

[majetideepak]

WriterOptions has the Velox schema. We should be able to convert that to Arrow schema in the Parquet Writer.

[JkSelf]

WriterOptions has the Velox schema. We should be able to convert that to Arrow schema in the Parquet Writer.

@majetideepak It appears that the WriterOptions in the DWRF writer contains the Velox schema, while the Parquet writer does not.

[majetideepak]

@JkSelf Please add a Velox schema to Parquet writer Options.

[JkSelf]

@JkSelf Please add a Velox schema to Parquet writer Options.

@majetideepak Updated. Can you help to review again?

[JkSelf]

@majetideepak Thanks for your review. I have fixed your comments. And can you help to review again? Thanks.

[JkSelf]

@majetideepak Can you help to review again?

[JkSelf]

@majetideepak Please help to review again. Thanks.

[JkSelf]

@majetideepak @mbasmanova Please help to review again. Thanks.

[majetideepak]

@JkSelf Sorry, I missed your comments. I think the PR description should clarify that this is an improvement from the current approach of inferring the schema from the input data.

[JkSelf]

@majetideepak I have fixed your comments. Can you help to review again?

[JkSelf]

Seeing test failures:

Note: Google Test filter = E2EFilterTest.shortDecimalDirect
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from E2EFilterTest
[ RUN      ] E2EFilterTest.shortDecimalDirect

=================================================================
==10221==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f02d037f808 at pc 0x7f02da58211a bp 0x7ffcf1082290 sp 0x7ffcf1082288
READ of size 8 at 0x7f02d037f808 thread T0
SCARINESS: 33 (8-byte-read-heap-buffer-overflow-far-from-bounds)
    #0 0x7f02da582119 in facebook::velox::parquet::arrow::FixedLenByteArray facebook::velox::parquet::arrow::SerializeFunctor<facebook::velox::parquet::arrow::PhysicalType<(facebook::velox::parquet::arrow::Type::type)7>, arrow::Decimal128Type, void>::FixDecimalEndianess<16>(unsigned char const*, long) fbcode/velox/dwio/parquet/writer/arrow/ColumnWriter.cpp:2848
    #1 0x7f02da581759 in facebook::velox::parquet::arrow::SerializeFunctor<facebook::velox::parquet::arrow::PhysicalType<(facebook::velox::parquet::arrow::Type::type)7>, arrow::Decimal128Type, void>::Serialize(arrow::Decimal128Array const&, facebook::velox::parquet::arrow::ArrowWriteContext*, facebook::velox::parquet::arrow::FixedLenByteArray*) fbcode/velox/dwio/parquet/writer/arrow/ColumnWriter.cpp:2806

In this pull request, we only modify the schema. I'm not sure why this issue is occurring. Could it be a bug in Arrow? @majetideepak @mbasmanova Do you have any input? Thanks for your help.

@mbasmanova @majetideepak The issue seems to be caused by not using dictionary encoding in the vector in Writer::Writer(vector) API to get the arrow schema if creating a empty vector. Currently, I have placed the schema update inside the Writer::Writer(vector) API so that we can retrieve the Arrow schema based on the vector. Then, update the name in the Arrow schema based on the name in the schema_. And also added complex type unit tests. Please help to review again. Thanks.

[facebook-github-bot]

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mbasmanova]

@JkSelf Thank you for iterating. Looks good % couple of nits.

[JkSelf]

@mbasmanova Resolved all your comments. Can you help to review again? Thanks.

[mbasmanova]

Thanks.

[facebook-github-bot]

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@mbasmanova merged this pull request in 2ada2ec.