Support custom_metadata= argument in to_parquet

[rjzamora]

Adds a custom_metadata= argument to DataFrame.to_parquet. If a dictionary is passed in by the user, those key/values will be included in all footer metadata, and in the global _metadata file, along with the usual b"pandas" metadata.

Note that this PR only adds custom_metadata= support for the "pyarrow" engine. @martindurant - Do you know if something similar can be accomplished in the "fastparquet" engine? I have not investigated that end just yet. [EDIT: Both pyarrow and fastparquet engines are now supported.]

• Closes Define Column Metadata with to_parquet #7343
• Tests added / passed
• Passes black dask / flake8 dask

[martindurant]

fastparquet itself does not support this, but it would be trivial to add.

[martindurant]

You may want to check for the pandas key, since that could render a dataset unreadable.

[rjzamora]

@jorisvandenbossche - I saw that this method is labeled as "Experimental," but it has been around for years now (as far as I can tell). Is there any reason not to use it here?

[jorisvandenbossche]

I think that's fine to use (we should update the docstring)

[jorisvandenbossche]

-> https://issues.apache.org/jira/browse/ARROW-11980

[rjzamora]

You may want to check for the pandas key, since that could render a dataset unreadable.

Can you clarify what you mean. Are you saying that we shouldn't allow the user to modify the "pandas" metadata (i.e. add their own b"pandas" key)?

[martindurant]

In FastParquetEngine.initialize_write we create an fmd object, which is then used for the rest of the writes. We could insert metadata directly into this:

fmd.key_value_metadata.extend([
fastparquet.parquet_thrift.KeyValue(key=key, value=value)
for key, value in custom_metadata.items()
])

[martindurant]

Can you clarify what you mean. Are you saying that we shouldn't allow the user to modify the "pandas" metadata (i.e. add their own b"pandas" key)?

Actually, the order matters - either we hand the user the ability to overwrite the pandas tag, or we would overwrite whatever they provide. In either case, I think a warning would be warranted.

[rjzamora]

In FastParquetEngine.initialize_write we create an fmd object, which is then used for the rest of the writes. We could insert metadata directly into this

Nice! Good idea - I'll experiment with that :)

Actually, the order matters

Just want to clarify... In the current solution, we are "updating" the key-value metadata with the user-specified metadata. Therefore, we would be replacing an existing b"pandas" key if the user decided to pass in custom pandas metadata. You are saying this is fine, but we want to add a warning, since changing the pandas metadata in the wrong way can render the data completely unreadable - correct?

[martindurant]

Correct. We could choose to error (you mustn't mess it up!) or ignore. Is there a genuine reason to want to update that tag? In our case, b"pandas" should always be present.

[rjzamora]

Is there a genuine reason to want to update that tag?

It's hard to imagine a user wanting to open that can of worms :) I'm sure there exists a "power-user use case" where dask.dataframe is just not able produce the ideal pandas metadata for a specific workflow (likely related to index preservation, etc), and a possible solution is to override the key/value metadata with an option like this. With that said, I am totally fine raising an error here until we actually observe a motivating use case in the wild.

[rjzamora]

Thanks for the quick feedback here @martindurant! This PR now includes support for fastparquet, and an error is raised if the custom metadata includes a b'pandas' key.

[jsignell]

Good to merge then?

[martindurant]

The one failure is indeed for parquet, so should be investigated

[rjzamora]

The one failure is indeed for parquet, so should be investigated

I couldn't reproduce locally, and it doesn't seem that this PR was the cause (although I am not completely sure).

[jsignell]

Sorry this sat for so long @rjzamora