DOC clarifications on the release process (#15759)

* some extra comments

* reorganize steps

* more info

* more clarification

* further instructions

* be cautious with tagging

* add pypi note

* git log copy

* upstream

* more explanation

* fix typo

* clarify

* typo

* further clarifications, address comments

* add link to pep101

* explain branch names

* apply comments

* cover more comments

* clarify branching, remove Dash ref

* Update maintainer.rst

* address some of Roman's comments

* address some of Nocolas's suggestions

* trying to address comments

* CLN Adds reference to conda-forge recipe/meta.yml

* DOC Explicitly push the tag by name

* DOC Explicitly git add folders before commiting

* fixed ref and removed redundant sentence (I think)

* Added note about making sure derecations are handled

* fixed title headers, this file is not just about releasing

* removed redundant paragraph

* major changes

* minor comments

Co-authored-by: Hanmin Qin <qinhanmin2005@sina.com>
Co-authored-by: Thomas J Fan <thomasjpfan@gmail.com>
Co-authored-by: Roman Yurchak <rth.yurchak@gmail.com>
Co-authored-by: Nicolas Hug <contact@nicolas-hug.com>

doc/developers/maintainer.rst

@@ -1,14 +1,27 @@
 Maintainer / core-developer information
 ========================================
 
+
+Releasing
+---------
+
+This section is about preparing a major release, incrementing the minor
+version, or a bug fix release incrementing the patch version. Our convention is
+that we release one or more release candidates (0.RRrcN) before releasing the
+final distributions. We follow the `PEP101
+<https://www.python.org/dev/peps/pep-0101/>`_ to indicate release candidates,
+post, and minor releases.
+
 Before a release
-----------------
+................
 
 1. Update authors table::
 
     $ cd build_tools; make authors; cd ..
 
-   and commit.
+   and commit. This is only needed if the authors have changed since the last
+   release. This step is sometimes done independent of the release. This
+   updates the maintainer list and is not the contributor list for the release.
 
 2. Confirm any blockers tagged for the milestone are resolved, and that other
    issues tagged for the milestone can be postponed.
@@ -17,61 +30,96 @@ Before a release
    change log is reasonably well curated. Some tools for these tasks include:
 
    - ``maint_tools/sort_whats_new.py`` can put what's new entries into
-     sections.
+     sections. It's not perfect, and requires manual checking of the changes.
+     If the whats new list is well curated, it may not be necessary.
 
    - The ``maint_tools/whats_missing.sh`` script may be used to identify pull
      requests that were merged but likely missing from What's New.
 
-Preparing a bug-fix-release
-...........................
+4. Make sure the deprecations, FIXME and TODOs tagged for the release have
+   been taken care of.
+
+**Permissions**
+
+The release manager requires a set of permissions on top of the usual
+permissions given to maintainers, which includes:
+
+- *maintainer* role on ``scikit-learn`` projects on ``pypi.org`` and
+  ``test.pypi.org``, separately.
+- become a member of the *scikit-learn* team on conda-forge by editing the 
+  ``recipe/meta.yaml`` file on 
+  ``https://github.com/conda-forge/scikit-learn-feedstock``
+- *maintainer* on ``https://github.com/MacPython/scikit-learn-wheels``
+
 
-Since any commits to a released branch (e.g. 0.999.X) will automatically update
-the web site documentation, it is best to develop a bug-fix release with a pull
-request in which 0.999.X is the base. It also allows you to keep track of any
-tasks towards release with a TO DO list.
+.. _preparing_a_release_pr:
 
-Most development of the bug fix release, and its documentation, should
-happen in master to avoid asynchrony. To select commits from master for use in
-the bug fix (version 0.999.3), you can use::
+Preparing a release PR
+......................
 
+Releasing the first RC of e.g. version `0.99` involves creating the release
+branch `0.99.X` directly on the main repo, where `X` really is the letter X,
+**not a placeholder**. This is considered the *feature freeze*. The
+development for the major and minor releases of 0.99 should
+**also** happen under `0.99.X`. Each release (rc, major, or minor) is a tag
+under that branch.
+
+In terms of including changes, the first RC ideally counts as a *feature
+freeze*. Each coming release candidate and the final release afterwards will
+include minor documentation changes and bug fixes. Any major enhancement or
+feature should be excluded.
+
+The minor releases should include bug fixes and some relevant documentation
+changes only. Any PR resulting in a behavior change which is not a bug fix
+should be excluded.
+
+First, create a branch, **on your own fork** (to release e.g. `0.999.3`)::
+
+    $ # assuming master and upstream/master are the same
     $ git checkout -b release-0.999.3 master
-    $ git rebase -i 0.999.X
 
-Then pick the commits for release and resolve any issues, and create a pull
-request with 0.999.X as base. Add a commit updating ``sklearn.__version__``.
-Additional commits can be cherry-picked into the ``release-0.999.3`` branch
-while preparing the release.
+Then, create a PR **to the** `scikit-learn/0.999.X` **branch** (not to
+master!) with all the desired changes::
+
+	$ git rebase -i upstream/0.999.2
+
+It's nice to have a copy of the ``git rebase -i`` log in the PR to help others
+understand what's included.
 
 Making a release
-----------------
+................
+
+0. Create the release branch on the main repo, if it does not exist. This is
+   done only once, as the major and minor releases happen on the same branch::
 
-1. Update docs:
+     $ git checkout -b 0.99.X
+
+   Again, `X` is literal here, and `99` is replaced by the release number.
+   The branches are called ``0.19.X``, ``0.20.X``, etc.
+
+1. Update docs. Note that this is for the final release, not necessarily for
+   the RC releases. These changes should be made in master and cherry-picked
+   into the release branch, only before the final release.
 
    - Edit the doc/whats_new.rst file to add release title and commit
      statistics. You can retrieve commit statistics with::
 
         $ git shortlog -s 0.99.33.. | cut -f2- | sort --ignore-case | tr '\n' ';' | sed 's/;/, /g;s/, $//'
 
-   - Update the release date in whats_new.rst
+   - Update the release date in ``whats_new.rst``
 
-   - Edit the doc/index.rst to change the 'News' entry of the front page.
-
-   - Note that these changes should be made in master and cherry-picked into
-     the release branch.
+   - Edit the doc/templates/index.html to change the 'News' entry of the front
+     page.
 
 2. On the branch for releasing, update the version number in
-   sklearn/__init__.py, the ``__version__`` variable by removing ``dev*`` only
-   when ready to release.
-   On master, increment the version in the same place (when branching for
-   release).
-
-3. Create the tag and push it::
+   `sklearn/__init__.py`, the ``__version__`` variable by removing ``dev*``
+   only when ready to release. On master, increment the version in the same
+   place (when branching for release). This means while we're in the release
+   candidate period, the latest stable is two versions behind the master
+   branch, instead of one.
 
-    $ git tag -a 0.999
-
-    $ git push git@github.com:scikit-learn/scikit-learn.git --tags
-
-4. Create the source tarball:
+3. At this point all relevant PRs should have been merged into the `0.99.X`
+   branch. Create the source tarball:
 
    - Wipe clean your repo::
 
@@ -81,10 +129,32 @@ Making a release
 
        $ python setup.py sdist
 
+   - You can also test a binary dist build using::
+
+       $ python setup.py bdist_wheel
+
+   - You can test if PyPi is going to accept the package using::
+
+       $ twine check dist/*
+
+   You can run ``twine check`` after step 5 (fetching artifacts) as well.
+
    The result should be in the `dist/` folder. We will upload it later
    with the wheels. Check that you can install it in a new virtualenv and
    that the tests pass.
 
+4. Proceed with caution. Ideally, tags should be created when you're almost
+   certain that the release is ready, since adding a tag to the main repo can
+   trigger certain automated processes. You can test upload the ``sdist`` to
+   ``test.pypi.org``, and test the next step by setting ``BUILD_COMMIT`` to the
+   branch name (``0.99.X`` for instance) in a PR to the wheel building repo.
+   Once all works, you can proceed with tagging. Create the tag and push it (if
+   it's an RC, it can be ``0.xxrc1`` for instance)::
+
+    $ git tag -a 0.99  # in the 0.99.X branch
+
+    $ git push git@github.com:scikit-learn/scikit-learn.git 0.99
+
 5. Update the dependency versions and set ``BUILD_COMMIT`` variable to the
    release tag at:
 
@@ -94,16 +164,20 @@ Making a release
    packages and upload them to PyPI by running the following commands in the
    scikit-learn source folder (checked out at the release tag)::
 
-       $ rm -r dist
+       $ rm -r dist # only if there's anything other than the sdist tar.gz there
        $ pip install -U wheelhouse_uploader twine
        $ python setup.py fetch_artifacts
 
 6. Check the content of the `dist/` folder: it should contain all the wheels
-   along with the source tarball ("scikit-learn-XXX.tar.gz").
+   along with the source tarball ("scikit-learn-RRR.tar.gz").
 
    Make sure that you do not have developer versions or older versions of
    the scikit-learn package in that folder.
 
+   Before uploading to pypi, you can test upload to test.pypi.org::
+
+       $ twine upload --verbose --repository-url https://test.pypi.org/legacy/ dist/*
+
    Upload everything at once to https://pypi.org::
 
        $ twine upload dist/*
@@ -119,21 +193,25 @@ Making a release
        $ git checkout master
        $ rm stable
        $ ln -s 0.999 stable
-       $ sed -i "s/latestStable = '.*/latestStable = '0.999';" versionwarning.js
-       $ git commit -m "Update stable to point to 0.999" stable
+       $ sed -i "s/latestStable = '.*/latestStable = '0.999';/" versionwarning.js
+       $ git add stable/ versionwarning.js
+       $ git commit -m "Update stable to point to 0.999"
        $ git push origin master
 
 The following GitHub checklist might be helpful in a release PR::
 
     * [ ] update news and what's new date in master and release branch
     * [ ] create tag
-    * [ ] update dependencies and release tag at https://github.com/MacPython/scikit-learn-wheels
+    * [ ] update dependencies and release tag at
+      https://github.com/MacPython/scikit-learn-wheels
     * [ ] twine the wheels to PyPI when that's green
     * [ ] https://github.com/scikit-learn/scikit-learn/releases draft
-    * [ ] confirm bot detected at https://github.com/conda-forge/scikit-learn-feedstock and wait for merge
+    * [ ] confirm bot detected at
+      https://github.com/conda-forge/scikit-learn-feedstock and wait for merge
     * [ ] https://github.com/scikit-learn/scikit-learn/releases publish
-    * [ ] announce on mailing list
-    * [ ] (regenerate Dash docs: https://github.com/Kapeli/Dash-User-Contributions/tree/master/docsets/Scikit)
+    * [ ] fix the binder release version in ``.binder/requirement.txt`` (see
+      #15847)
+    * [ ] announce on mailing list and on twitter
 
 The scikit-learn.org web site
 -----------------------------