Change to_parquet default to write_metadata_file=None

[jcrist]

This changes to_parquet to default to write_metadata_file=None. If None, a _metadata file is only written when append=True and the dataset has an existing _metadata file, otherwise it defaults to False.

If a _metadata file doesn't exist when appending, the last file in the dataset is used to validate schema and divisions.

There are also some adjacent changes to simplify the generated to_parquet graph (in particular, the full dataset metadata isn't actually needed in each to_parquet task, but was previously included, bloating graph size).

The majority of this PR is modifying the tests to not fail after this change, since many of them were implicitly relying on the existence of a _metadata file.

Fixes #8901.

• Closes #xxxx
• Tests added / passed
• Passes pre-commit run --all-files

[jcrist]

cc @rjzamora

[rjzamora]

Thanks for working on this @jcrist!

I didn't look very carefully at the tests yet, but I only have some minor questions so far.

[rjzamora]

Does this just need a gather_statistics=True to avoid the compute()? (same question for other usage of sol below)

[jcrist]

Or a check_divisions=False. The main change is that without a _metadata file read_parquet will result in a dataframe without divisions by default, and most of the tests are calling assert_eq on two dask dataframes (so divisions are compared). Since there are tests for explicitly checking divisions, in most cases checking divisions like this was accidental and unimportant, I used a mishmash of:

• Using a pandas dataframe for the intended solution
• Passing in gather_statistics=True to get a dataframe wit hknown divisions
• Passing in check_divisions=False

to resolve these failures.

[rjzamora]

That makes sense - In my branch to deprecate gather_statistics (which I didn't submit yet, and will likely conflict with this PR's test changes in many places), I did end up adding an explicit calculate_divisions=True in many places.

It is somewhat true that we are "accidentally" checking divisions, but my slight preference is to avoid computing before the assert_eq unless we really need to. I'd say check_divisions=False is a bit better, but I may end up rolling back some of these sol changes in the gather_statistics PR.

[rjzamora]

LGTM @jcrist

I do think it is important that we update the Metadata section in the new parquet doc page, but I'm okay with that happening in a separate PR if you'd prefer.

[rjzamora]

I'm okay with removing this, but if you specifiy gather_statistics=True (soon to becalculate_divisions=True), we will filter out the empty partitions. The original test was checking that we don't get these empty partitions.