Keep original schema and metadata

[hombit]

I propose to keep original metadata and schema with the parquet file. It also includes keeping all the columns in the file, which may be achieved with replacing their values with nulls.

[Schwarzam]

What are the benefits of keeping all columns?

Should we then remove the columns that the user did not choose on the client side?

[hombit]

The benefit is to keep the data to be consistent with the original schema, so users of the server would be confused by the mismatch of the actual data with 1) individual parquet file key-value metadata, such as pandas metadata, 2) _metadata and _common_metadata of the parquet dataset.

You still could skip reading this data from the disk, and just create null-filled columns which would occupy very small space in parquet files

import pyarrow as pa
import pyarrow.parquet

pa.parquet.write_table(pa.table({'a': np.random.default_rng(0).random(1<<20)}), 'test_missed.parquet')
pa.parquet.write_table(pa.table({'a': np.random.default_rng(0).random(1<<20), 'b': pa.array([None] * (1 << 20), type=pa.float64())}), 'test_nulls.parquet')

> ls -lk test_*.parquet
-rw-r--r--@ 1 hombit  staff  8668973 Jun  3 12:40 test_missed.parquet
-rw-r--r--@ 1 hombit  staff  8669228 Jun  3 12:40 test_nulls.parquet

[Schwarzam]

Implemented the null columns in the commit b6ae52b.
(will change to only PRs on the main soon)

The metadata still an issue. ParquetWriter does not have any params to pass the metadata.