Add transparent compression for on-disk shuffle with partd

[ChrWesp]

Shoud close #5704

This change adds transparent file compression for on-disk shuffle via partd.
The compression algorithm can be specified by dask config (keyword shuffle_disk_compression).
It can be any algorithm which is exposed by partd.compressed. In case an algorithm is absent, it silently falls back to no compression.

Please check one thing - I am not sure about the naming schema and where to document the config parameter.

• Tests added / passed
• Passes black dask / flake8 dask

[quasiben]

Thanks for submitting a PR for #5704 . The default config file can be found here: https://github.com/dask/dask/blob/master/dask/dask.yaml . Can I ask that you also add to the config test here:

dask/dask/tests/test_config.py

Lines 417 to 418 in 5db6dad

	def test_core_file():
	assert "temporary-directory" in dask.config.config

[mrocklin]

What do we think about the name? Two things come to mind:

1. We've started namespacing most of our configuration names. Maybe we should do the same here?
2. We tend to use hyphens rather than underscores. Maybe we should do the same here.

[ChrWesp]

@quasiben : Thanks, I will add the config test.
@mrocklin : The current naming is definitly not optimal but I do not know the general naming convention or what would fit best here. Please make a proposal

[TomAugspurger]

I'd suggest dataframe.shuffle-comperssion.

[TomAugspurger]

Thanks for the PR. Can you also add tests?

[TomAugspurger]

There's a linting issue too: https://travis-ci.org/dask/dask/jobs/637027038#L1437-L1440. Contributing guidelines are at https://docs.dask.org/en/latest/develop.html#code-formatting. You might want to install and use pre-commit.

[TomAugspurger]

Thanks @ChrWesp!