[ARROW-17904] add flatten data files with checksums on datapage v1

[mapleFU]

Here I add 3 files for checking checksums, there schema are all:

message m {
    required int32 a;
    required int32 b;
} 

And there contents are equal:

  private static final int PAGE_SIZE = 1024 * 10; // 10KB
  private static final byte[] colAPage1Bytes = new byte[PAGE_SIZE];
  private static final byte[] colAPage2Bytes = new byte[PAGE_SIZE];
  private static final byte[] colBPage1Bytes = new byte[PAGE_SIZE];
  private static final byte[] colBPage2Bytes = new byte[PAGE_SIZE];
  private static final int numRecordsLargeFile = (2 * PAGE_SIZE) / Integer.BYTES;

  /** Write out sample Parquet file using ColumnChunkPageWriteStore directly, return path to file */
  private Path writeSimpleParquetFile(Configuration conf, CompressionCodecName compression)
    throws IOException {
    File file = tempFolder.newFile();
    file.delete();
    String originFileName = file.getPath();
    Path path = new Path(file.toURI());
    System.out.println("writeSimpleParquetFile: " + path);

    for (int i = 0; i < PAGE_SIZE; i++) {
      colAPage1Bytes[i] = (byte) i;
      colAPage2Bytes[i] = (byte) -i;
      colBPage1Bytes[i] = (byte) (i + 100);
      colBPage2Bytes[i] = (byte) (i - 100);
    }

    ParquetFileWriter writer =  new ParquetFileWriter(conf, schemaSimple, path,
      ParquetWriter.DEFAULT_BLOCK_SIZE, ParquetWriter.MAX_PADDING_SIZE_DEFAULT);

    writer.start();
    writer.startBlock(numRecordsLargeFile);

    CodecFactory codecFactory = new CodecFactory(conf, PAGE_SIZE);
    CodecFactory.BytesCompressor compressor = codecFactory.getCompressor(compression);

    ColumnChunkPageWriteStore writeStore = new ColumnChunkPageWriteStore(
      compressor, schemaSimple, new HeapByteBufferAllocator(),
      Integer.MAX_VALUE, ParquetOutputFormat.getPageWriteChecksumEnabled(conf));

    PageWriter pageWriter = writeStore.getPageWriter(colADesc);
    pageWriter.writePage(BytesInput.from(colAPage1Bytes), numRecordsLargeFile / 2,
      numRecordsLargeFile / 2, EMPTY_STATS_INT32, Encoding.RLE, Encoding.RLE, Encoding.PLAIN);
    pageWriter.writePage(BytesInput.from(colAPage2Bytes), numRecordsLargeFile / 2,
      numRecordsLargeFile / 2, EMPTY_STATS_INT32, Encoding.RLE, Encoding.RLE, Encoding.PLAIN);

    pageWriter = writeStore.getPageWriter(colBDesc);
    pageWriter.writePage(BytesInput.from(colBPage1Bytes), numRecordsLargeFile / 2,
      numRecordsLargeFile / 2, EMPTY_STATS_INT32, Encoding.RLE, Encoding.RLE, Encoding.PLAIN);
    pageWriter.writePage(BytesInput.from(colBPage2Bytes), numRecordsLargeFile / 2,
      numRecordsLargeFile / 2, EMPTY_STATS_INT32, Encoding.RLE, Encoding.RLE, Encoding.PLAIN);

    writeStore.flushToFileWriter(writer);

    writer.endBlock();

The code is borrow and modified from parquet-mr's TestDataPageV1Checksums, because seems that only parquet-mr implements crc on data page v1. Each file has two columns, and each column has two pages, the size of each page would tried to be 10KB.

And:

• data/datapage_v1-corrupt-checksum.parquet tests corrput crc
• data/datapage_v1-snappy-compressed-checksum.parquet is generated with snappy compressed
• data/datapage_v1-uncompressed-checksum.parquet is page with correct crc and no compression. Specifically, the first page of column a and the second page of column b is corrupt.

[mapleFU]

This will be used in apache/arrow#14351
@pitrou

[pitrou]

• data/datapage_v1-corrupt-checksum.parquet tests corrput crc

  • data/datapage_v1-snappy-compressed-checksum.parquet is generated with snappy compressed

  • data/datapage_v1-uncompressed-checksum.parquet is page with correct crc and no compression. Specifically, the first page of column a and the second page of column b is corrupt.

I'm not sure I understand. Is there one file or two files with corrupt CRCs?

[mapleFU]

I'm not sure I understand. Is there one file or two files with corrupt CRCs?

Lets draw some pictures here:

data/datapage_v1-uncompressed-checksum.parquet:

| Column "a" | Page 0 [bad crc ] | Uncompressed Contents | Page 1 [correct crc ] | Uncompressed Contents |
| Column "b" | Page 0 [correct crc ] | Uncompressed Contents | Page 1 [bad crc ] | Uncompressed Contents |

data/datapage_v1-corrupt-checksum.parquet:

| Column "a" | Page 0 [correct crc ] | Uncompressed Contents | Page 1 [correct crc ] | Uncompressed Contents |
| Column "b" | Page 0 [correct crc ] | Uncompressed Contents | Page 1 [correct crc ] | Uncompressed Contents |

data/datapage_v1-snappy-compressed-checksum.parquet:

| Column "a" | Page 0 [correct crc ] | Snappy Contents | Page 1 [correct crc ] | Snappy Contents |
| Column "b" | Page 0 [correct crc ] | Snappy Contents | Page 1 [correct crc ] | Snappy Contents |

And each page has 10KB / sizeof(int32_t) records

[pitrou]

Thanks @mapleFU , can you add that information to https://github.com/apache/parquet-testing/blob/master/data/README.md ?

[mapleFU]

can you add that information to https://github.com/apache/parquet-testing/blob/master/data/README.md ?

I've upload a basic description. But perhaps I'm not able to written a good document, so welcome to edit it directly. @pitrou

[pitrou]

Thanks @mapleFU . This looks good to me.