Add release candidates to release procedure

[hammer]

Dask 2.26.0 (side note: anchor tags change with each release, so this link will be incorrect soon) included dask/dask#6393, a major change to how casting and delegation works in Dask.

Unfortunately this change was incomplete and testing did not catch things like dask/dask#6611 and dask/dask#6631.

Has the Dask community considered adding a step to the release procedure in which a release candidate is published and possibly voted upon before becoming an official release?

This discussion is related but separate from the semantic versioning discussion happening in #93.

[kkraus14]

Similar discussion is also happening in #84

[TomAugspurger]

Unfortunately this change was incomplete and testing did not catch things like dask/dask#6611 and dask/dask#6631.

The regression reported in #6611 was caught before the release, thanks to a downstream project testing against dask master. However it wasn't elevated to a release blocker. And I'm not entirely sure it should have been. The Array.__contains__ isn't sufficiently useful to block a release IMO. I think the report in dask/dask#6631 probably would have warranted blocking the release.

As to release candidates, @hammer what benefit does an RC have over downstream projects testing against nightlies / master? My experience with pandas RCs is not positive. The most recent pandas RC had something like 50-100 downloads on conda-forge (https://anaconda.org/conda-forge/pandas/files?version=1.1.0rc0), and I think 0 reported issues (despite there being a number of regressions). For a pure-python dependency like dask that typically keeps master in a releasable state, testing against master (or nightly) seems preferable to me.

[jakirkham]

We’ve also been discussing nightlies ( #76 ). Not quite the same thing, but may help for downstream testing as well.