Incorrect (but ignored) metadata written after ColumnChunk

[liukun4515]

TLDR: is we wrote the wrong thing after the column chunk, but no implementations actually read this data as it is already present in the footer, and so we never noticed

Describe the bug
I'm working on the #1935, and go through the write path of rust-version.

I find that parquet-write write some error data which is ColumnChunk to the file, you can find the logic from

arrow-rs/parquet/src/file/writer.rs

Line 441 in 9f7b600

fn serialize_column_chunk(&mut self, chunk: parquet::ColumnChunk) -> Result<()> {

From my knowledge about parquet format in this https://github.com/apache/parquet-format/blob/54e53e5d7794d383529dd30746378f19a12afd58/src/main/thrift/parquet.thrift#L789 and https://github.com/apache/parquet-format#file-format, the data after column data is ColumnMetaData, not ColumnChunk

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

[liukun4515]

I will try to fix the this point.

[liukun4515]

@sunchao @alamb PTAL

[liukun4515]

@tustvold
I read the java version of parquet write in the https://github.com/apache/parquet-mr/blob/e06384455567c56d5906fc3a152ab00fd8dfdf33/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L916, and it do't follow the format https://github.com/apache/parquet-mr/blob/e06384455567c56d5906fc3a152ab00fd8dfdf33/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java#L916 writing the Column Metadata behind each column chunk.
Is there any missing information for me?
I'm confused about the logic about write path between rust-parquet and java-parquet.

[liukun4515]

I created a ticket in the Parquet-MR for java version
https://issues.apache.org/jira/browse/PARQUET-2162

[alamb]

@liukun4515 and @tustvold I tried to summarize the issue in the title of this ticket (which is included in the CHANGELOG) but I am not sure I totally understand -- can you please verify my title change is correct?

[tustvold]

The TLDR is we wrote the wrong thing after the column chunk, but no implementations actually read this data as it is already present in the footer, and so we never noticed

[alamb]

Thank you -- added to the original description