Release changelogs

[fjetter]

I took the changelog of the distributed 2021.10.0 and 2021.09.0 releases and tried to classify the changes into a few categories.

The 2021.09.0 changelog had 22 changes of which 14 (that's >60%) are maintenance related fixes as far as I can tell. Only very few actual user relevant fixes are in this release.

I see two issues

1. I'm wondering if our release cadence is actually quite short. I don't think that particular release actually offers an incredible amount of value
2. The changelog is incredibly hard to parse. Even for me as a core maintainer it is virtually impossible to make sense of the changelog. With the default changelog, I do not know what are bugfixes, enhancements, doc changes, etc. Even for actual enhancements, I'm having a hard time figuring out what this is about, e.g. what is GH#5248 actually doing? I can guess but the changelog doesn't really tell me

Generic enhancements and regressions:   3
Dashboards:                             1
Documentation:                          4
Tests, Maintenance, Refactoring, etc.:  14
------------------------------------------
Total                                   22

Changelog categorized distributed 2021.09.0

# Generic enhancements and regression
- Overwrite worker plugins (GH#5248) Matthew Rocklin
- Improved IPv6 dask-worker support (GH#5197) Walt Woods
- Add support for diskless machines to system monitor (GH#5257) James Bourbeau

# Active Memory Managment
# Stability

# Dashboard
- Use non-histogram plots up to 100 workers (GH#5249) Matthew Rocklin

# Documentation
- Fix “then” -> “than” typo in docs (GH#5247) David Chudzicki
- Fix typo (remove extra verb “creates”) in docs (GH#5244) David Chudzicki
- Fix “fractiom” -> “fraction” typo in docstring (GH#5245) David Chudzicki
- Fix “schedulers” -> “scheduler” typo in docs (GH#5246) David Chudzicki
-
# Tests, Maintenance, refactoring
- Avoid during-iteration scheduler plugin changes (GH#5259) Doug Davis
- Properly check for ipv6 availability (GH#5255) crusaderky
- Replace atop with blockwise (GH#5289) James Bourbeau
- Add pytest color to CI (GH#5276) James Bourbeau
- Fix test_map and others (GH#5278) crusaderky
- Use name argument with Scheduler.remove_plugin calls (GH#5260) Doug Davis
- Downgrade to jupyter_client 6 (GH#5273) crusaderky
- Migrate Security HTML repr to Jinja2 (GH#5264) Jacob Tomlinson
- Migrate ProcessInterface HTML repr to Jinja2 (GH#5263) Jacob Tomlinson
- Fix add_plugin warnings (GH#5267) Doug Davis
- Add list around iterator in handle_missing_dep (GH#5285) Matthew Rocklin
- Jupyter-client 7 compatiblity (GH#5286) Min RK
- Remove GroupProgress scheduler plugin (GH#5256) James Bourbeau
- Refactor scheduler plugins; store in a dictionary (GH#5120) Doug Davis

Moving on to the 2021.10.0 release which was actually delayed for stability reasons, the summary looks like

Generic enhancements and regressions:   8
Stability:                              5
Dashboards:                             8
Documentation:                          5
Security:                               1
Tests, Maintenance, Refactoring, etc.:  27
------------------------------------------
Total                                   55

This feels like a much more meaningful release, however, the changelog is a bit of a mess, I likely misclassified a lot.

Changelog categorized distributed 2021.10.0

# Generic enhancements and regressions (8 / 14.5%)

- Increase latency for stealing (GH#5390) Florian Jetter
- Log original exception upon compute failure (GH#5387) Florian Jetter
- Add support for partial functions to iscoroutinefunction util (GH#5344) Michael Adkins
- Use dask-spec for SSHCluster (GH#5191) Charles Blackmon-Luca
- Update _cluster_info dict in __init__ (GH#5305) Jacob Tomlinson
- Pickle exception and traceback immediately (GH#5338) Mads R. B. Kristensen
- Ensure dask-worker and dask-scheduler pick up preload configuration values (GH#5365) James Bourbeau
- Handle UCXNotConnected error (GH#5449) Peter Andreas Entschev

# Active Memory Managment (5 / 9%)
- Run multiple AMMs in parallel (GH#5339) crusaderky
- Reinstate: AMM ReduceReplicas to iterate only on replicated tasks (GH#5341) crusaderky
- Don’t schedule tasks to paused workers (GH#5431) crusaderky
- Active Memory Manager to use bulk comms (GH#5357) crusaderky
- Sync worker status to the scheduler; new ‘paused’ status (GH#5330) crusaderky


# Stability (8 / 14.5%)
- Ensure resumed flight tasks are still fetched (GH#5426) Florian Jetter
- Ensure reconnecting workers do not loose required data (GH#5436) Florian Jetter
- Resolve work stealing deadlock caused by race in move_task_confirm (GH#5379) 
- Fix regression where unknown tasks were allowed to be stolen (GH#5392) 
- Do no attempt to fetch keys which are no longer in flight (GH#5160) Florian 
- Fix zombie worker tasks after missing transition (GH#5316) Florian Jetter
- Workers submit a reply to the scheduler if replica removal was rejected (GH#5356) Florian Jetter
- Worker state machine refactor (GH#5046) Florian Jetter

# Dashboard (1/ 1.8%)
- Add scroll to dashboard dropdown (GH#5418) Jacob Tomlinson
  
# Security (1 / 1.8%)
- Pass host through LocalCluster to workers (GH#5427) Jim Crist-Harif

# Documentation (5 / 9%)
- AMM high level documentation (GH#5456) crusaderky
- Simple SSHCluster example (GH#5349) Ray Bell
- Use new Dask docs theme (GH#5391) Jacob Tomlinson
- Enhance AMM docstrings (GH#5340) crusaderky
- Fix typo in client side example foundations.rst (GH#5336) Genevieve Buckley

# Tests, Maintenance, refactoring (27 / 49%)
- Revisit Scheduler.add_plugin / Scheduler.remove_plugin (GH#5394) crusaderky
- Provide stack for suspended coro in test timeout (GH#5446) Florian Jetter
- Use pip install . instead of calling setup.py (GH#5442) Matthias Bussonnier
- Type annotations for Worker and gen_cluster (GH#5438) crusaderky
- Mark test_gather_dep* as xfail (GH#5432) crusaderky
- Remove zict-related skips (GH#5429) James Bourbeau
- Fixes async warnings in UCX tests (GH#5396) Peter Andreas Entschev
- Enable mypy in CI 2/2 (GH#5348) crusaderky
- Rewrite test_client_timeout (GH#5397) crusaderky
- Fix flaky test_WorkerPlugin_overwrite (GH#5398) crusaderky
- Add coverage badge to README (GH#5382) James Bourbeau
- Mark test_stress_creation_and_deletion as xfail (GH#5393) James Bourbeau
- Mark test_worker_reconnects_mid_compute* tests as flaky (GH#5378) James Bourbeau
- Remove pytest.mark.repeat from test_prometheus_collect_task_states (GH#5376) James Bourbeau
- Add code coverage (GH#4670) James Bourbeau
- Mark distributed/tests/test_client.py::test_profile_server as flaky (GH#5375) James Bourbeau
- Enable mypy in CI 1/2 (GH#5328) crusaderky
- Use Dask temporary file utility (GH#5361) James Bourbeau
- Avoid deprecated random set sampling (GH#5360) James Bourbeau
- Add check for unsupported NVML metrics (GH#5343) Charles Blackmon-Luca
- Add pre-commit to environments (GH#5362) Ray Bell
- Generate Cython HTML annotations (GH#5321) crusaderky
- Bump RAPIDS_VER for gpuCI (GH#5358) Charles Blackmon-Luca
- fsspec and s3fs git tips are incompatible (GH#5346) crusaderky
- Worker State Machine Refactor: clean up dead handlers (GH#5359) crusaderky
- Fix test_many_Progress and others (GH#5329) crusaderky
- Run pyupgrade in CI (GH#5327) crusaderky

Original changelogs, see http://distributed.dask.org/en/stable/changelog.html#id2

I didn't go through the dask/dask changelog but it is similarly unorganized and I am struggling to decipher what is what and what would be relevant for me if I was a user.

I would like to propose

A) We start categorizing our changelog in this or a similar way. I believe we could easily do this via a semi-automated process using labels. We might even consider dropping the entire maintenance section.

Alternatively, we could maintain a manual changelog, e.g. similar to pandas. That's more effort and a frequent source for merge conflicts in my experience but that would also allow us to provide more context information for a change.

Either way, I believe a more organized changelog would provide a lot of value for users.

B) This will be a more lengthy discussion but I would like to start a conversation about our release cadence. In particular with stability problems all over the place, the two weeks cadence feels a bit rushed sometimes to fix known regressions and properly test them.

Edit: Let's move discussion around B to #200

[jsignell]

A) I really like the idea of an organized changelog and I agree that we can use PR labels to achieve that in a way that does not significantly increase the burden on maintainers.

B) I don't understand how the release cadence affects our ability to fix regressions. In my mind, we agreed to just release where we are at every 2 weeks. With that approach, if there is a regression and it takes a month to fix, then that regression would just be in two releases. I know that is not exactly what we have been doing, but it feels like a fine approach to me. If people want to be able to pin to a release that they know is a pretty good one, that takes us back to the conversation about "blessed" releases.

[fjetter]

don't understand how the release cadence affects our ability to fix regressions. In my mind, we agreed to just release where we are at every 2 weeks

I believe two weeks are not sufficient to find and fix the regressions. Due to stability problems in dask/distributed we recently needed to delay multiple releases because we didn't manage to fix them in the given time.

[jameslamb]

I wanted to add some examples from how other Python projects handle characterizing release notes. Hope it helps.

We start categorizing our changelog in this or a similar way. I believe we could easily do this via a semi-automated process using labels.

We do this in LightGBM using the release-drafter bot.

• docs: https://probot.github.io/apps/release-drafter/
• example config: https://github.com/microsoft/LightGBM/blob/99f0f3ecf1de7f6f4f81434680c4ddb63800c567/.github/release-drafter.yml
• example release notes: https://github.com/microsoft/LightGBM/releases/tag/v3.3.0

Release-drafter automatically creates a draft release (only visible to maintainers), and then you have an opportunity to manually edit it before publishing it as an official release.

Alternatively, we could maintain a manual changelog

My personal opinion...asking individual contributors to update a changelog in files adds friction for not a lot of benefit. This is what cuml does, for example, and I recall needing to fix merge conflicts in that project's CHANGELOG.md 4 times while I patiently waited for a small PR to be reviewed.

For a sort of middle ground between manual and automatic, prefect asks contributors to add a small YAML file to a directory called changes/. On each release, prefect has some automation that sucks up these files and creates release notes, then deletes the files.

• docs on how they update: https://github.com/PrefectHQ/prefect/tree/0388c1691f69046091381011a678084877d83bea/changes
• script they use: https://github.com/PrefectHQ/prefect/blob/0388c1691f69046091381011a678084877d83bea/update_changelog.py

That approach is nice because it doesn't result in merge conflicts, but it's still a burden on maintainers to remind contributors to add such a file.

[fjetter]

I like the above idea about the release-drafter bot. However, so far, we're not actually using githubs release feature but I guess this is not a deal breaker. Either we start using it or we simply use the bots to generate the markdown. IIUC, this would mostly be based on labels so we should start labeling everyhing properly either way.

My personal opinion...asking individual contributors to update a changelog in files adds friction for not a lot of benefit.

Yes, I pointed out pandas because they are using this and I am aware of that process. It is quite tedious, the results are OK, definitely better than what dask has to offer right now. However, the merge conflicts are very annoying, especially with long CI build times

[jakirkham]

don't understand how the release cadence affects our ability to fix regressions. In my mind, we agreed to just release where we are at every 2 weeks

I believe two weeks are not sufficient to find and fix the regressions. Due to stability problems in dask/distributed we recently needed to delay multiple releases because we didn't manage to fix them in the given time.

On the flip side releasing less frequently could mean more regressions accumulate in main (as others are only testing the latest release) and we only find out about more of them later (once we finally release).

[jrbourbeau]

On the topic of automating release notes:

• I'm happy to try out some form of automation
• We should leave some point for human intervention / editing (e.g. there are sometimes typos in commit messages)
• For label-based categories, apparently GitHub has some support for this https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes. I've not used this or protobot (thanks for linking to those resources @jameslamb), but the configuration files appear to be similar for the two approaches.

[jrbourbeau]

We've got a release scheduled for tomorrow (xref #209). I'll propose we start improving the format of our changelogs by introducing the following sections for each release:

• Highlights: Large-scale or particularly important changes (e.g. Dropping support for Python 3.6)
• New Features: New functionality added since the last release
• Enhancements: Performance improvements, extended support for existing features, etc.
• Bug fixes: Self explanatory
• Deprecations: Any new or expiring deprecations
• Documentation: Docs-only PRs
• Maintenance: CI changes, internal refactors, etc. Basically this section is for non-user facing changes

Thoughts on these categories? For tomorrow I'll just handle sorting commits into the above categories manually

[jameslamb]

👋 hello from a longtime dask/community lurker.

@jrbourbeau instead of "Highlights" being a bulleted list of PRs, I want to suggest doing something like what XGBoost does for releases. Their release notes typically include a few paragraphs summarizing the content of the release, followed by bulleted lists linking to specific issues / PRs and arranged under headers.

For example: https://github.com/dmlc/xgboost/releases/tag/v1.5.0

That summary is typically written for the audience of "people installing and using XGBoost", and answers the question "why should I care about this release?".

I've really appreciated that because sometimes one feature like "better support for categorical data" is actually a composite of many separate PRs.

[jakirkham]

As a counterpoint, one advantage of bullet points is they are quicker to read for people that have a limited amount of time

[fjetter]

I'd like to revive this conversation again. We've come a long way and have been using the current process (see above, #199 (comment)) for a rather long time now. I think that the changelog improved but is still not where it should be.

A couple of notes about the current state

• I believe the categorization into Enhancement / Bug fix / Maint / etc. is not always accurate
• PR titles are often rather technical and non-descriptive. They are very often not describing what will change for the user
• Very, very minor stuff is ending up in the changelog that interests nobody.

Even I, as an expert, am sometimes struggling to parse our changelog to know where the notable changes are burried. I suspect that most end users do not even bother reading the changelog any longer.

On top of this, I don't think it is very obvious for users whether a change in dask/dask or dask/distributed affects them. I suspect that most users are using distributed as well but are interfacing mostly with dask/dask. While there are technical reasons to have dedicated repos, this does not extend to having dedicated changelogs.

I would like to have a way to

• Highlight noticeable changes
• Provide user valuable instead of technical descriptions for changes
• Ignore / filter entirely irrelevant PRs (fix linting, typo in docs, etc.)
• Categorize even further to give users the ability to more easily understand "What is changing for me?". For instance, a dask.Array user will not care about pyarrow strings but they may care about general improvements, e.g. new instrumentation.
• Consolidate dask/dask and dask/distributed changelogs into one

cc @jrbourbeau @jacobtomlinson @phofl

[phofl]

I agree that we need a better way of filtering out irrelevant stuff (and order things better). The changelog currently contains a lot of things that most folks don't care about.

Quick description how the pandas process works:

The changelog in pandas is updated by the PR author. This means that the author of a PR creates a whatsnew entry in one of the available categories. PRs are only added, if they changed anything that is user-visible, meaning that maintenance PRs or something similar won't show up in the changelog.

This has a couple of advantages:

• Not restricted to the PR title
• The author can choose an appropriate message, chances are that the release manager isn't fully aware of all changes
• The work is parallelized among PR authors, right now the release manager has to take care of the changelog

[jakirkham]

How does the Pandas release process handle conflicts between adjacent changelog entries added by different PRs?

[phofl]

Good question.

We have a script that sorts entries alphabetically per category that's part of pre-commit. It does not remove all conflicts, but we rarely see conflicts since we introduced it.

[fjetter]

I don't think we'll get around manual effort here and considering that the release manager (mostly but not exclusively @jrbourbeau ) isn't always aware of all changes, I'm +1 for a pandas-like approach. Still open for other proposals, of course