From f2bb8c3480f23ec5266fcd336f73c96ebc7d6990 Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Fri, 3 Feb 2017 23:53:59 +0100 Subject: [PATCH 1/8] DOC: add HOWTO_RELEASE document --- HOWTO_RELEASE.md | 167 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 HOWTO_RELEASE.md diff --git a/HOWTO_RELEASE.md b/HOWTO_RELEASE.md new file mode 100644 index 00000000000..3cd37711d33 --- /dev/null +++ b/HOWTO_RELEASE.md @@ -0,0 +1,167 @@ +# Release Howto + +This document is intended to give precise instructions on the process of making a release. Its purpose is to avoid broken packages, broken documentation and many other things that can go wrong in the release process. Since this is not everyday work and may be done under the stress of a (self-imposed) deadline, it is clearly beneficial to have a checklist to hold on to. + +**Note:** The instructions in this document are tentative until tested in practice. They are also written from the perspective of Linux and may need adaption for other platforms. + +## 1. Agree on a release schedule + +This involves the "what" and "when" of the release process and fixes a feature set that is supposed to be included in the new version. The steps are: + +- Open an issue on the issue tracker using the title **Release x.y.z**. +- Discuss and agree on a set of open PRs that should be merged before making a release. + +## 2. Make sure tests succeed and docs can be built + +When all required PRs are merged, ensure that the latest `master` branch is sane. Travis CI checks every PR, but certain things like CUDA cannot be tested there and must therefore undergo tests on a local machine, for at least Python 2.7 and one version of Python 3. + +- Make a new test conda environment and install all dependencies: + ``` + $ conda create -n release35 python=3.5 nomkl numpy scipy future matplotlib pytest + $ source activate release35 + $ pip install pyfftw pywavelets + $ cd /path/to/odl_repo + $ git checkout master + $ git pull + $ pip install -e . + ``` +- Run the tests with `pytest`, including doctests, examples documentation and large-scale tests: + ``` + $ pytest --doctest-modules --examples --doctest-doc --largescale + ``` +- Do the same for Python 2.7. +- Make sure the tests also run on the platforms you're currently *not* testing on. + Ask a buddy maintainer if necessary. +- Build the documentation. This requires `sphinx` and the `sphinxext` submodule: + ``` + $ conda install sphinx sphinx-rtd-theme + $ git submodule update --init --recursive + $ cd doc && make clean + $ cd source && python generate_doc.py + $ cd .. + $ make html 2>&1 |\ + grep -E "SEVERE|ERROR|WARNING" |\ + grep -E -v "more than one target found for|__eq__|document isn't included in any toctree" + ``` + The last command builds the documentation and filters from the output all irrelevant warnings, letting through only the "proper" warnings and errors. + If possible, *fix these remaining issues*. +- Glance the built documentation (usually in `doc/_build`) for obvious errors. + +## 3. Make a release branch off `master` and draft the release + +When all tests succeed and the docs are fine, start a release branch. All work until the creation of the git release tag is now done on the release branch. +**Do not touch any code on this branch other than indicated below!** + +- Create a branch off current `master` with the name `release-X.Y.Z`, inserting the correct version number, of course. +- Compile the release notes. + They should contain all *user-visible* changes (internal stuff like modified tests is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. + Check the [Release Notes](https://github.com/odlgroup/odl/blob/master/doc/source/release_notes.rst) file for details on sections, formatting etc. +- Increment the version number. + The current locations of version numbers are the [odl/__init__.py](https://github.com/odlgroup/odl/blob/master/odl/__init__.py) and [conda/meta.yaml](https://github.com/odlgroup/odl/blob/master/conda/meta.yaml). + To make sure you don't miss any other location (or the information here is outdated), perform a search: + ``` + $ cd doc && make clean && cd .. # remove the local HTML doc first + $ grep -Ir "0\.5\.4" . | grep -E -v "\.git|release_notes\.rst|odl\.egg-info" + ``` +- Change the `git_rev` field in `conda/meta.yaml` to `'vX.Y.Z'`, using the upcoming version number. + This is the git tag you will create when making the release on GitHub. +- Commit the changes, using a message like `REL: bump version to X.Y.Z`. +- Make a PR and merge it after review. + +## 4. Make a release on GitHub + +Now that the version is incremented, + +- make a new [Release](https://github.com/odlgroup/odl/releases) on GitHub. +- Paste the short summary from the release notes file (converting from RST to Markdown) but don't insert the details. +- Add a link to the current section in the release notes file. + +## 5. Create packages for PyPI and Conda + +- Making the packages for PyPI is straightforward: + ``` + $ conda install wheel + $ python setup.py sdist + $ python setup.py bdist_wheel + ``` + The packages are by default stored in a `dist` folder. + +- To build the conda packages, you should *not* work in a specific environment but rather exit to the root environment. + There, install the `conda-build` tool for building packages: + ``` + $ source deactivate + $ conda install conda-build + ``` +- Invoke the following command to build a package for your platform and all supported Pyhton versions: + ``` + $ conda build conda/ --python 2.7 + $ conda build conda/ --python 3.4 + $ conda build conda/ --python 3.5 + ... + ``` +- Assuming this succeeds, enter the directory one above where the conda package was stored (as printed in the output), e.g., + ``` + $ cd $HOME/miniconda3/conda-bld/ + ``` + There, for each Python version "translate" the package to all platforms since ODL is actually platform-independent: + ``` + $ conda convert --platform all + ``` + Replace `` by the package file as built by the previous `conda build` command. + +## 6. Upload the packages to test locations and test installing them + +Before actually uploading packages to "official" servers, first use a test location and make sure installation works as expected. + +- Install the `twine` package for uploading packages to PyPI in your working environment: + ``` + $ source activate release35 + $ pip install twine + ``` +- Upload the source package (`*.tar.gz`) and the wheel (`*.whl`) to the test server and try installing it from there into a new conda environment: + ``` + $ cd /path/to/odl_repo + $ twine upload -u odlgroup -r pypitest + $ source deactivate + $ conda create -n pypi_install python=X.Y # choose Python version + $ source activate pypi_install + $ pip install --index-url https://test.pypi.org/legacy odl + $ python -c "import odl; odl.test()" + $ pip uninstall odl + $ pip install --index-url --no-binary https://test.pypi.org/legacy odl + $ python -c "import odl; odl.test()" + ``` + **TODO:** Currently no `odlgroup` user exists on the test server, may need to be created. +- Upload the conda package to *your own* conda channel and test the install procedure in a new conda environment. + The upload requires the `anaconda-client` package: + ``` + $ source deactivate + $ conda install anaconda-client + $ anaconda upload -u <...> + $ conda create -n conda_install python=X.Y # choose Python version + $ source activate conda_install + $ conda install -c odl + $ python -c "import odl; odl.test()" + ``` + +## 7. Upload the packages to the official locations + +Installing the packages works, now it's time to put them out into the wild. + +- Upload the source package and the wheel to the "real" PyPI server using again `twine`: + ``` + $ source deactivate && source activate release35 + $ twine upload -u odlgorup -r pypi + ``` + This requires the access credentials for the `odlgroup` user on PyPI. +- Upload the conda packages to the `odlgroup` channel in the Anaconda cloud: + ``` + $ cd $HOME/miniconda3/conda-bld + $ anaconda upload -u odlgroup `find . -name "odl-X.Y.Z*"` + ``` + For this step, you need the access credentials for the `odlgroup` user on the Anaconda server. + + +## Done! + +Time to clean up. From 8358a7970f5a2a276cf24705a0ef0a9f6cee469a Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Sat, 4 Feb 2017 00:05:53 +0100 Subject: [PATCH 2/8] DOC: add dev version bump to release howto --- HOWTO_RELEASE.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/HOWTO_RELEASE.md b/HOWTO_RELEASE.md index 3cd37711d33..8fe1602e233 100644 --- a/HOWTO_RELEASE.md +++ b/HOWTO_RELEASE.md @@ -30,8 +30,7 @@ When all required PRs are merged, ensure that the latest `master` branch is sane $ pytest --doctest-modules --examples --doctest-doc --largescale ``` - Do the same for Python 2.7. -- Make sure the tests also run on the platforms you're currently *not* testing on. - Ask a buddy maintainer if necessary. +- Make sure the tests also run on the platforms you're currently *not* testing on. Ask a buddy maintainer if necessary. - Build the documentation. This requires `sphinx` and the `sphinxext` submodule: ``` $ conda install sphinx sphinx-rtd-theme @@ -43,8 +42,7 @@ When all required PRs are merged, ensure that the latest `master` branch is sane grep -E "SEVERE|ERROR|WARNING" |\ grep -E -v "more than one target found for|__eq__|document isn't included in any toctree" ``` - The last command builds the documentation and filters from the output all irrelevant warnings, letting through only the "proper" warnings and errors. - If possible, *fix these remaining issues*. + The last command builds the documentation and filters from the output all irrelevant warnings, letting through only the "proper" warnings and errors. If possible, *fix these remaining issues*. - Glance the built documentation (usually in `doc/_build`) for obvious errors. ## 3. Make a release branch off `master` and draft the release @@ -53,18 +51,13 @@ When all tests succeed and the docs are fine, start a release branch. All work u **Do not touch any code on this branch other than indicated below!** - Create a branch off current `master` with the name `release-X.Y.Z`, inserting the correct version number, of course. -- Compile the release notes. - They should contain all *user-visible* changes (internal stuff like modified tests is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. - Check the [Release Notes](https://github.com/odlgroup/odl/blob/master/doc/source/release_notes.rst) file for details on sections, formatting etc. -- Increment the version number. - The current locations of version numbers are the [odl/__init__.py](https://github.com/odlgroup/odl/blob/master/odl/__init__.py) and [conda/meta.yaml](https://github.com/odlgroup/odl/blob/master/conda/meta.yaml). - To make sure you don't miss any other location (or the information here is outdated), perform a search: +- Compile the release notes. They should contain all *user-visible* changes (internal stuff like modified tests is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. Check the [Release Notes](https://github.com/odlgroup/odl/blob/master/doc/source/release_notes.rst) file for details on sections, formatting etc. +- Increment the version number. The current locations of version numbers are the [odl/__init__.py](https://github.com/odlgroup/odl/blob/master/odl/__init__.py) and [conda/meta.yaml](https://github.com/odlgroup/odl/blob/master/conda/meta.yaml). To make sure you don't miss any other location (or the information here is outdated), perform a search: ``` $ cd doc && make clean && cd .. # remove the local HTML doc first $ grep -Ir "0\.5\.4" . | grep -E -v "\.git|release_notes\.rst|odl\.egg-info" ``` -- Change the `git_rev` field in `conda/meta.yaml` to `'vX.Y.Z'`, using the upcoming version number. - This is the git tag you will create when making the release on GitHub. +- Change the `git_rev` field in `conda/meta.yaml` to `'vX.Y.Z'`, using the upcoming version number. This is the git tag you will create when making the release on GitHub. - Commit the changes, using a message like `REL: bump version to X.Y.Z`. - Make a PR and merge it after review. @@ -86,8 +79,7 @@ Now that the version is incremented, ``` The packages are by default stored in a `dist` folder. -- To build the conda packages, you should *not* work in a specific environment but rather exit to the root environment. - There, install the `conda-build` tool for building packages: +- To build the conda packages, you should *not* work in a specific environment but rather exit to the root environment. There, install the `conda-build` tool for building packages: ``` $ source deactivate $ conda install conda-build @@ -132,8 +124,7 @@ Before actually uploading packages to "official" servers, first use a test locat $ python -c "import odl; odl.test()" ``` **TODO:** Currently no `odlgroup` user exists on the test server, may need to be created. -- Upload the conda package to *your own* conda channel and test the install procedure in a new conda environment. - The upload requires the `anaconda-client` package: +- Upload the conda package to *your own* conda channel and test the install procedure in a new conda environment. The upload requires the `anaconda-client` package: ``` $ source deactivate $ conda install anaconda-client @@ -161,6 +152,12 @@ Installing the packages works, now it's time to put them out into the wild. ``` For this step, you need the access credentials for the `odlgroup` user on the Anaconda server. +## 8. Bump current `master` to a development version + +To ensure a larger version number for installations from the git `master` branch, the version number must be increased immediately. + +- Add a `.dev0` at the end of the version string in `odl/__init__.py`. +- Make a PR with just this change. It should be the first one that goes in after the release. ## Done! From f92d08621f55aa1b8d5c7729b79b30e4b80fc289 Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Sat, 4 Feb 2017 00:12:32 +0100 Subject: [PATCH 3/8] DOC: improve dev version bumping instructions --- HOWTO_RELEASE.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/HOWTO_RELEASE.md b/HOWTO_RELEASE.md index 8fe1602e233..d89b0351341 100644 --- a/HOWTO_RELEASE.md +++ b/HOWTO_RELEASE.md @@ -156,8 +156,10 @@ Installing the packages works, now it's time to put them out into the wild. To ensure a larger version number for installations from the git `master` branch, the version number must be increased immediately. -- Add a `.dev0` at the end of the version string in `odl/__init__.py`. -- Make a PR with just this change. It should be the first one that goes in after the release. +- Change the version string `'X.Y.Z'` in `odl/__init__.py` to `'X.Y.Z+1.dev0'` (e.g. from `'0.5.3'` to `'0.5.4.dev0'`). +- Change the `git_rev` field in `conda/meta.yaml` to `'master'`. +- Commit the changes, using a message like `REL: bump version to X.Y.Z.dev0`. +- Make a PR with just this change and merge it after review. It should be the first one that goes in after the release. ## Done! From 85e1cbf4e48a5e8838e94cc8a466beaaebd17ec4 Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Sat, 4 Feb 2017 02:03:24 +0100 Subject: [PATCH 4/8] DOC: simplify and improve release howto --- HOWTO_RELEASE.md | 53 +++++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 25 deletions(-) diff --git a/HOWTO_RELEASE.md b/HOWTO_RELEASE.md index d89b0351341..ba2f420bcb3 100644 --- a/HOWTO_RELEASE.md +++ b/HOWTO_RELEASE.md @@ -71,9 +71,17 @@ Now that the version is incremented, ## 5. Create packages for PyPI and Conda -- Making the packages for PyPI is straightforward: +It is important to **update the local git repository** before creating packages, to make sure that they are built from the correct git revision. + +- Pull from GitHub and switch to `master` (or to the new version tag, which should be equivalent): + ``` + $ git checkout master + $ git pull + ``` + +- Making the packages for PyPI is straightforward. However, **make sure you delete old `build` directories** since they can pollute new builds: ``` - $ conda install wheel + $ rm build/ -rf $ python setup.py sdist $ python setup.py bdist_wheel ``` @@ -101,37 +109,25 @@ Now that the version is incremented, ``` Replace `` by the package file as built by the previous `conda build` command. -## 6. Upload the packages to test locations and test installing them +## 6. Test installing the local packages and check them -Before actually uploading packages to "official" servers, first use a test location and make sure installation works as expected. +Before actually uploading packages to "official" servers, first install the local packages and run the unit tests. -- Install the `twine` package for uploading packages to PyPI in your working environment: - ``` - $ source activate release35 - $ pip install twine - ``` -- Upload the source package (`*.tar.gz`) and the wheel (`*.whl`) to the test server and try installing it from there into a new conda environment: +- Install directly from the source package (`*.tar.gz`) or the wheel (`*.whl`) into a new conda environment: ``` - $ cd /path/to/odl_repo - $ twine upload -u odlgroup -r pypitest $ source deactivate $ conda create -n pypi_install python=X.Y # choose Python version $ source activate pypi_install - $ pip install --index-url https://test.pypi.org/legacy odl - $ python -c "import odl; odl.test()" - $ pip uninstall odl - $ pip install --index-url --no-binary https://test.pypi.org/legacy odl + $ cd /path/to/odl_repo + $ pip install dist/ $ python -c "import odl; odl.test()" ``` - **TODO:** Currently no `odlgroup` user exists on the test server, may need to be created. -- Upload the conda package to *your own* conda channel and test the install procedure in a new conda environment. The upload requires the `anaconda-client` package: +- Install and test the local conda packages in a new conda environment: ``` $ source deactivate - $ conda install anaconda-client - $ anaconda upload -u <...> $ conda create -n conda_install python=X.Y # choose Python version $ source activate conda_install - $ conda install -c odl + $ conda install --use-local nomkl odl $ python -c "import odl; odl.test()" ``` @@ -139,14 +135,21 @@ Before actually uploading packages to "official" servers, first use a test locat Installing the packages works, now it's time to put them out into the wild. -- Upload the source package and the wheel to the "real" PyPI server using again `twine`: +- Install the `twine` package for uploading packages to PyPI in your working environment: ``` - $ source deactivate && source activate release35 - $ twine upload -u odlgorup -r pypi + $ source deactivate + $ source activate release35 + $ pip install twine + ``` +- Upload the source package and the wheel to the PyPI server using `twine`: + ``` + $ cd /path/to/odl_repo + $ twine upload -u odlgorup dist/ ``` This requires the access credentials for the `odlgroup` user on PyPI. -- Upload the conda packages to the `odlgroup` channel in the Anaconda cloud: +- Upload the conda packages to the `odlgroup` channel in the Anaconda cloud. The upload requires the `anaconda-client` package: ``` + $ conda install anaconda-client $ cd $HOME/miniconda3/conda-bld $ anaconda upload -u odlgroup `find . -name "odl-X.Y.Z*"` ``` From e9ed4f94d045b8b745aa6da5cca4e60d440f32cd Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Sun, 12 Feb 2017 20:57:38 +0100 Subject: [PATCH 5/8] MAINT: move release HOWTO to sphinx docs --- HOWTO_RELEASE.md => doc/source/dev/release.rst | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename HOWTO_RELEASE.md => doc/source/dev/release.rst (100%) diff --git a/HOWTO_RELEASE.md b/doc/source/dev/release.rst similarity index 100% rename from HOWTO_RELEASE.md rename to doc/source/dev/release.rst From 7c5dc5c0b3076aa0f75469ba087022e4c00e3f5c Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Sun, 12 Feb 2017 21:02:13 +0100 Subject: [PATCH 6/8] DOC: rewrite release HOWTO as RST --- doc/source/dev/dev.rst | 18 +- doc/source/dev/document.rst | 2 +- doc/source/dev/extend.rst | 6 +- doc/source/dev/release.rst | 324 +++++++++++++++++++++--------------- 4 files changed, 202 insertions(+), 148 deletions(-) diff --git a/doc/source/dev/dev.rst b/doc/source/dev/dev.rst index f3e9ddfe383..1e1cbe31017 100644 --- a/doc/source/dev/dev.rst +++ b/doc/source/dev/dev.rst @@ -6,18 +6,15 @@ Contributing to ODL Introduction ------------ -Great that you want to help making ODL better! There are basically two ways how you can contribute: -as a user or as a developer. +Great that you want to help making ODL better! +There are basically two ways how you can contribute: as a user or as a developer. -The best way to make contributions as a user is to play around with the software, try to use it for -your purposes and get back to us if you encounter problems or miss features. When this happens, -take a look at our `issue tracker `_, and if there is no -existing issue dealing with your problem, create a new one. Don't be shy -- use the issue tracker to -ask questions, too. +The best way to make contributions as a user is to play around with the software, try to use it for your purposes and get back to us if you encounter problems or miss features. +When this happens, take a look at our `issue tracker `_, and if there is no existing issue dealing with your problem, create a new one. +Don't be shy -- use the issue tracker to ask questions, too. -If you are a developer and want to contribute code, you may want to read through the -subsequent instructions. If you are experienced with Git, you may want to skip directly to -:ref:`the development workflow section `. +If you are a developer and want to contribute code, you may want to read through the subsequent instructions. +If you are experienced with Git, you may want to skip directly to :ref:`the development workflow section `. In order to properly follow the ODL style, we recommend that you read the `How to document`_:ref:`dev_document` and :ref:`dev_testing` sections. @@ -30,4 +27,5 @@ Contents extend document testing + release gitwash/index diff --git a/doc/source/dev/document.rst b/doc/source/dev/document.rst index bb94f3c0669..8cd8377aded 100644 --- a/doc/source/dev/document.rst +++ b/doc/source/dev/document.rst @@ -1,4 +1,4 @@ -.. _dev_document +.. _dev_document: ############### How to document diff --git a/doc/source/dev/extend.rst b/doc/source/dev/extend.rst index adb3a08e07c..26c858cd737 100644 --- a/doc/source/dev/extend.rst +++ b/doc/source/dev/extend.rst @@ -1,8 +1,12 @@ +.. _dev_extend: + ############# Extending ODL ############# -ODL is written to be easy to extend with new functionality and classes, and new content is welcome. With that said, not everything fits inside the main library and some ideas are better done as *extension packages*. This may give your package more freedom and allows a faster development cycle and help keep ODL from overflowing with content. +ODL is written to be easy to extend with new functionality and classes, and new content is welcome. +With that said, not everything fits inside the main library, and some ideas are better realized as *extension packages*, i.e., packages that use the core ODL library and extend it with experimental features. +This lowers the requirement on code maturity, completeness of documentation, unit tests etc. on your side and allows the core library to stay slim and develop faster. There are several ways to extend ODL, some of which are listed below. diff --git a/doc/source/dev/release.rst b/doc/source/dev/release.rst index ba2f420bcb3..5484bc68623 100644 --- a/doc/source/dev/release.rst +++ b/doc/source/dev/release.rst @@ -1,169 +1,221 @@ -# Release Howto +.. _dev_release: -This document is intended to give precise instructions on the process of making a release. Its purpose is to avoid broken packages, broken documentation and many other things that can go wrong in the release process. Since this is not everyday work and may be done under the stress of a (self-imposed) deadline, it is clearly beneficial to have a checklist to hold on to. +####################### +The ODL release process +####################### -**Note:** The instructions in this document are tentative until tested in practice. They are also written from the perspective of Linux and may need adaption for other platforms. +This document is intended to give precise instructions on the process of making a release. +Its purpose is to avoid broken packages, broken documentation and many other things that can go wrong as a result of mistakes during the release process. +Since this is not everyday work and may be done under the stress of a (self-imposed) deadline, it is clearly beneficial to have a checklist to hold on to. -## 1. Agree on a release schedule +.. note:: + The instructions in this document are tentative until tested in practice. + They are also written from the perspective of Linux and may need adaption for other platforms. -This involves the "what" and "when" of the release process and fixes a feature set that is supposed to be included in the new version. The steps are: +1. Agree on a release schedule +------------------------------ +This involves the "what" and "when" of the release process and fixes a feature set that is supposed to be included in the new version. +The steps are: -- Open an issue on the issue tracker using the title **Release x.y.z**. +- Open an issue on the issue tracker using the title **Release X.Y.Z** (insert numbers, of course). - Discuss and agree on a set of open PRs that should be merged before making a release. -## 2. Make sure tests succeed and docs can be built - -When all required PRs are merged, ensure that the latest `master` branch is sane. Travis CI checks every PR, but certain things like CUDA cannot be tested there and must therefore undergo tests on a local machine, for at least Python 2.7 and one version of Python 3. +2. Make sure tests succeed and docs are built properly +------------------------------------------------------ +When all required PRs are merged, ensure that the latest ``master`` branch is sane. Travis CI checks every PR, but certain things like CUDA cannot be tested there and must therefore undergo tests on a local machine, for at least Python 2.7 and one version of Python 3. - Make a new test conda environment and install all dependencies: - ``` - $ conda create -n release35 python=3.5 nomkl numpy scipy future matplotlib pytest - $ source activate release35 - $ pip install pyfftw pywavelets - $ cd /path/to/odl_repo - $ git checkout master - $ git pull - $ pip install -e . - ``` -- Run the tests with `pytest`, including doctests, examples documentation and large-scale tests: - ``` - $ pytest --doctest-modules --examples --doctest-doc --largescale - ``` + + .. code-block:: bash + + conda create -n release36 python=3.6 nomkl numpy scipy future matplotlib pytest + source activate release36 + pip install pyfftw pywavelets + cd /path/to/odl_repo + git checkout master + git pull + pip install -e . + +- Run the tests with ``pytest``, including doctests, examples documentation and large-scale tests: + + .. code-block:: bash + + pytest --doctest-modules --examples --doctest-doc --largescale + - Do the same for Python 2.7. -- Make sure the tests also run on the platforms you're currently *not* testing on. Ask a buddy maintainer if necessary. -- Build the documentation. This requires `sphinx` and the `sphinxext` submodule: - ``` - $ conda install sphinx sphinx-rtd-theme - $ git submodule update --init --recursive - $ cd doc && make clean - $ cd source && python generate_doc.py - $ cd .. - $ make html 2>&1 |\ - grep -E "SEVERE|ERROR|WARNING" |\ - grep -E -v "more than one target found for|__eq__|document isn't included in any toctree" - ``` - The last command builds the documentation and filters from the output all irrelevant warnings, letting through only the "proper" warnings and errors. If possible, *fix these remaining issues*. -- Glance the built documentation (usually in `doc/_build`) for obvious errors. - -## 3. Make a release branch off `master` and draft the release - -When all tests succeed and the docs are fine, start a release branch. All work until the creation of the git release tag is now done on the release branch. +- Make sure the tests also run on the platforms you're currently *not* testing on. + Ask a buddy maintainer if necessary. +- Build the documentation. + This requires ``sphinx`` and the ``sphinxext`` submodule: + + .. code-block:: bash + + conda install sphinx sphinx_rtd_theme + git submodule update --init --recursive + cd doc && make clean + cd source && python generate_doc.py + cd .. + make html 2>&1 |\ + grep -E "SEVERE|ERROR|WARNING" |\ + grep -E -v "more than one target found for|__eq__|document isn't included in any toctree" + + The last command builds the documentation and filters from the output all irrelevant warnings, letting through only the "proper" warnings and errors. + If possible, *fix these remaining issues*. +- Glance the built documentation (usually in ``doc/_build``) for obvious errors. + +3. Make a release branch off ``master`` and draft the release +------------------------------------------------------------- +When all tests succeed and the docs are fine, start a release branch. +All work until the creation of the git release tag is now done on the release branch. **Do not touch any code on this branch other than indicated below!** -- Create a branch off current `master` with the name `release-X.Y.Z`, inserting the correct version number, of course. -- Compile the release notes. They should contain all *user-visible* changes (internal stuff like modified tests is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. Check the [Release Notes](https://github.com/odlgroup/odl/blob/master/doc/source/release_notes.rst) file for details on sections, formatting etc. -- Increment the version number. The current locations of version numbers are the [odl/__init__.py](https://github.com/odlgroup/odl/blob/master/odl/__init__.py) and [conda/meta.yaml](https://github.com/odlgroup/odl/blob/master/conda/meta.yaml). To make sure you don't miss any other location (or the information here is outdated), perform a search: - ``` - $ cd doc && make clean && cd .. # remove the local HTML doc first - $ grep -Ir "0\.5\.4" . | grep -E -v "\.git|release_notes\.rst|odl\.egg-info" - ``` -- Change the `git_rev` field in `conda/meta.yaml` to `'vX.Y.Z'`, using the upcoming version number. This is the git tag you will create when making the release on GitHub. -- Commit the changes, using a message like `REL: bump version to X.Y.Z`. -- Make a PR and merge it after review. +- Create a branch off current ``master`` with the name ``release-X.Y.Z``, inserting the correct version number, of course. +- Compile the release notes. + They should contain all *user-visible* changes (internal stuff like test modifications is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. + Check the `Release Notes `_ file for details on sections, formatting etc. +- Increment the version number. + The current locations of version numbers are in `odl/__init__.py `_ and `conda/meta.yaml `_. + To make sure you don't miss any other location (or the information here is outdated), perform a search: + + .. code-block:: bash -## 4. Make a release on GitHub + cd doc && make clean && cd .. # remove the local HTML doc first + grep -Ir "0\.5\.4" . | grep -E -v "\.git|release_notes\.rst|odl\.egg-info" +- Change the ``git_rev`` field in ``conda/meta.yaml`` to ``'vX.Y.Z'``, using the upcoming version number. + This is the git tag you will create when making the release on GitHub. +- Commit the changes, using a message like ``REL: bump version to X.Y.Z``. +- Make a PR and merge it after review. + +4. Make a release on GitHub +--------------------------- Now that the version is incremented, -- make a new [Release](https://github.com/odlgroup/odl/releases) on GitHub. +- make a new `Release `_ on GitHub. - Paste the short summary from the release notes file (converting from RST to Markdown) but don't insert the details. - Add a link to the current section in the release notes file. -## 5. Create packages for PyPI and Conda - +5. Create packages for PyPI and Conda +------------------------------------- It is important to **update the local git repository** before creating packages, to make sure that they are built from the correct git revision. -- Pull from GitHub and switch to `master` (or to the new version tag, which should be equivalent): - ``` - $ git checkout master - $ git pull - ``` - -- Making the packages for PyPI is straightforward. However, **make sure you delete old `build` directories** since they can pollute new builds: - ``` - $ rm build/ -rf - $ python setup.py sdist - $ python setup.py bdist_wheel - ``` - The packages are by default stored in a `dist` folder. - -- To build the conda packages, you should *not* work in a specific environment but rather exit to the root environment. There, install the `conda-build` tool for building packages: - ``` - $ source deactivate - $ conda install conda-build - ``` +- Pull from GitHub and switch to the new version tag: + + .. code-block:: bash + + git checkout master + git pull + git checkout vX.Y.Z + +- Making the packages for PyPI is straightforward. + However, **make sure you delete old** ``build`` **directories** since they can pollute new builds: + + .. code-block:: bash + + rm build/ -rf + python setup.py sdist + python setup.py bdist_wheel + + The packages are by default stored in a ``dist`` folder. + +- To build the conda packages, you should *not* work in a specific environment but rather exit to the root environment. + There, install the ``conda-build`` tool for building packages: + + .. code-block:: bash + + source deactivate + conda install conda-build + - Invoke the following command to build a package for your platform and all supported Pyhton versions: - ``` - $ conda build conda/ --python 2.7 - $ conda build conda/ --python 3.4 - $ conda build conda/ --python 3.5 - ... - ``` + + .. code-block:: bash + + conda build conda/ --python 2.7 + conda build conda/ --python 3.4 + conda build conda/ --python 3.5 + conda build conda/ --python 3.6 + ... + - Assuming this succeeds, enter the directory one above where the conda package was stored (as printed in the output), e.g., - ``` - $ cd $HOME/miniconda3/conda-bld/ - ``` + + .. code-block:: bash + + cd $HOME/miniconda3/conda-bld/ + There, for each Python version "translate" the package to all platforms since ODL is actually platform-independent: - ``` - $ conda convert --platform all - ``` - Replace `` by the package file as built by the previous `conda build` command. -## 6. Test installing the local packages and check them + .. code-block:: bash + + conda convert --platform all + Replace ```` by the package file as built by the previous ``conda build`` command. + +6. Test installing the local packages and check them +---------------------------------------------------- Before actually uploading packages to "official" servers, first install the local packages and run the unit tests. -- Install directly from the source package (`*.tar.gz`) or the wheel (`*.whl`) into a new conda environment: - ``` - $ source deactivate - $ conda create -n pypi_install python=X.Y # choose Python version - $ source activate pypi_install - $ cd /path/to/odl_repo - $ pip install dist/ - $ python -c "import odl; odl.test()" - ``` +- Install directly from the source package (``*.tar.gz``) or the wheel (``*.whl``) into a new conda environment: + + .. code-block:: bash + + source deactivate + conda create -n pypi_install python=X.Y # choose Python version + source activate pypi_install + cd /path/to/odl_repo + pip install dist/ + python -c "import odl; odl.test()" + - Install and test the local conda packages in a new conda environment: - ``` - $ source deactivate - $ conda create -n conda_install python=X.Y # choose Python version - $ source activate conda_install - $ conda install --use-local nomkl odl - $ python -c "import odl; odl.test()" - ``` -## 7. Upload the packages to the official locations + .. code-block:: bash + + source deactivate + conda create -n conda_install python=X.Y # choose Python version + source activate conda_install + conda install --use-local nomkl odl + python -c "import odl; odl.test()" +7. Upload the packages to the official locations +------------------------------------------------ Installing the packages works, now it's time to put them out into the wild. -- Install the `twine` package for uploading packages to PyPI in your working environment: - ``` - $ source deactivate - $ source activate release35 - $ pip install twine - ``` -- Upload the source package and the wheel to the PyPI server using `twine`: - ``` - $ cd /path/to/odl_repo - $ twine upload -u odlgorup dist/ - ``` - This requires the access credentials for the `odlgroup` user on PyPI. -- Upload the conda packages to the `odlgroup` channel in the Anaconda cloud. The upload requires the `anaconda-client` package: - ``` - $ conda install anaconda-client - $ cd $HOME/miniconda3/conda-bld - $ anaconda upload -u odlgroup `find . -name "odl-X.Y.Z*"` - ``` - For this step, you need the access credentials for the `odlgroup` user on the Anaconda server. - -## 8. Bump current `master` to a development version - -To ensure a larger version number for installations from the git `master` branch, the version number must be increased immediately. - -- Change the version string `'X.Y.Z'` in `odl/__init__.py` to `'X.Y.Z+1.dev0'` (e.g. from `'0.5.3'` to `'0.5.4.dev0'`). -- Change the `git_rev` field in `conda/meta.yaml` to `'master'`. -- Commit the changes, using a message like `REL: bump version to X.Y.Z.dev0`. -- Make a PR with just this change and merge it after review. It should be the first one that goes in after the release. - -## Done! - -Time to clean up. +- Install the ``twine`` package for uploading packages to PyPI in your working environment: + + .. code-block:: bash + + source deactivate + source activate release36 + pip install twine + +- Upload the source package and the wheel to the PyPI server using ``twine``: + + .. code-block:: bash + + cd /path/to/odl_repo + twine upload -u odlgorup dist/ + + This requires the access credentials for the ``odlgroup`` user on PyPI. +- Upload the conda packages to the ``odlgroup`` channel in the Anaconda cloud. + The upload requires the ``anaconda-client`` package: + + .. code-block:: bash + + conda install anaconda-client + cd $HOME/miniconda3/conda-bld + anaconda upload -u odlgroup `find . -name "odl-X.Y.Z*"` + + For this step, you need the access credentials for the ``odlgroup`` user on the Anaconda server. + +8. Bump current ``master`` to a development version +--------------------------------------------------- +To ensure a larger version number for installations from the git ``master`` branch, the version number must be increased immediately. + +- Change the version string ``'X.Y.Z'`` in ``odl/__init__.py`` to ``'X.Y.Z+1.dev0'`` (e.g. from ``'0.5.3'`` to ``'0.5.4.dev0'``). +- Change the ``git_rev`` field in ``conda/meta.yaml`` to ``'master'``. +- Commit the changes, using a message like ``REL: bump version to X.Y.Z.dev0``. +- Make a PR with just this change and merge it after review. + It should be the first one that goes in after the release. + +Done! +----- +Time to clean up, i.e., remove temporary conda environments, run ``conda build purge``, remove files in ``dist`` and ``build`` generated for the PyPI packages, etc. From d54355b94ee45ea063728c1f10aa190edcbb864c Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Sun, 12 Feb 2017 23:08:49 +0100 Subject: [PATCH 7/8] WIP: change order of release branch handling --- doc/source/dev/release.rst | 108 +++++++++++++++++++++++++------------ 1 file changed, 73 insertions(+), 35 deletions(-) diff --git a/doc/source/dev/release.rst b/doc/source/dev/release.rst index 5484bc68623..d638892d95a 100644 --- a/doc/source/dev/release.rst +++ b/doc/source/dev/release.rst @@ -12,6 +12,8 @@ Since this is not everyday work and may be done under the stress of a (self-impo The instructions in this document are tentative until tested in practice. They are also written from the perspective of Linux and may need adaption for other platforms. + +.. _dev_rel_release_schedule: 1. Agree on a release schedule ------------------------------ This involves the "what" and "when" of the release process and fixes a feature set that is supposed to be included in the new version. @@ -20,6 +22,8 @@ The steps are: - Open an issue on the issue tracker using the title **Release X.Y.Z** (insert numbers, of course). - Discuss and agree on a set of open PRs that should be merged before making a release. + +.. _dev_rel_master_ok: 2. Make sure tests succeed and docs are built properly ------------------------------------------------------ When all required PRs are merged, ensure that the latest ``master`` branch is sane. Travis CI checks every PR, but certain things like CUDA cannot be tested there and must therefore undergo tests on a local machine, for at least Python 2.7 and one version of Python 3. @@ -63,18 +67,33 @@ When all required PRs are merged, ensure that the latest ``master`` branch is sa If possible, *fix these remaining issues*. - Glance the built documentation (usually in ``doc/_build``) for obvious errors. -3. Make a release branch off ``master`` and draft the release -------------------------------------------------------------- + +.. _dev_rel_release_branch: +3. Make a release branch off ``master`` +--------------------------------------- When all tests succeed and the docs are fine, start a release branch. -All work until the creation of the git release tag is now done on the release branch. -**Do not touch any code on this branch other than indicated below!** +**Do not touch any actual code on this branch other than indicated below!** - Create a branch off current ``master`` with the name ``release-X.Y.Z``, inserting the correct version number, of course. -- Compile the release notes. - They should contain all *user-visible* changes (internal stuff like test modifications is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. - Check the `Release Notes `_ file for details on sections, formatting etc. -- Increment the version number. - The current locations of version numbers are in `odl/__init__.py `_ and `conda/meta.yaml `_. + + .. code-block:: bash + + git checkout master + git pull + git checkout -b release-X.Y.Z + git push -u my_fork release-X.Y.Z + + Like any regular branch that should result in a PR, the release branch is pushed to a fork. + + +.. _dev_rel_bump_master: +4. Bump the ``master`` branch to the next development version +------------------------------------------------------------- +To ensure a larger version number for installations from the git master branch, the version number must be increased before merging the release branch. + +- On the ``master`` branch, change the version string in ``odl/__init__.py`` to the next revision larger than the upcoming release version, plus ``'dev0'``. + For example, if the release version string is ``'0.5.3'``, use ``'0.5.4.dev0'``. + To make sure you don't miss any other location (or the information here is outdated), perform a search: .. code-block:: bash @@ -82,30 +101,43 @@ All work until the creation of the git release tag is now done on the release br cd doc && make clean && cd .. # remove the local HTML doc first grep -Ir "0\.5\.4" . | grep -E -v "\.git|release_notes\.rst|odl\.egg-info" +- In the file ``conda/meta.yaml``, change the version string after ``version: `` to the same as above, but without the ``dev0`` tag. + In the example above, this would mean to change it from ``"0.5.3"`` to ``"0.5.4"``. + + If necessary, change ``git_rev`` value to ``master``, although that should already be the case. + +- Commit the changes, using a message like ``REL: bump version to X.Y.Z.dev0``. +- Make a PR with just this change and merge it after review. + It must be merged before the release branch. + + +.. _dev_rel_publish: +5. Compile and publish the release +---------------------------------- +Back on the release branch with a ``git checkout release-X.Y.Z``, it is now time to prepare the release documents, increment the version number and make a release on GitHub. + +- Compile the release notes. + They should contain all *user-visible* changes (internal stuff like test modifications is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. + Check the `Release Notes `_ file for details on sections, formatting etc. +- Increment the version number in ``odl/__init__.py`` and ``conda/meta.yaml``. + As in :ref:`Section 4`, perform a search to make sure you didn't miss a version info location. - Change the ``git_rev`` field in ``conda/meta.yaml`` to ``'vX.Y.Z'``, using the upcoming version number. This is the git tag you will create when making the release on GitHub. - Commit the changes, using a message like ``REL: bump version to X.Y.Z``. -- Make a PR and merge it after review. +- Make a PR and fix review comments. + When doing so, try to keep the ``REL: bump version to X.Y.Z`` commit last, for example by using ``git commit --amend`` for fixes, or by squashing the commits on the release branch. -4. Make a release on GitHub ---------------------------- -Now that the version is incremented, + **Don't merge immediately when ready!** -- make a new `Release `_ on GitHub. +- Make a new `Release `_ on GitHub **from the release branch, not master**. - Paste the short summary from the release notes file (converting from RST to Markdown) but don't insert the details. - Add a link to the current section in the release notes file. -5. Create packages for PyPI and Conda -------------------------------------- -It is important to **update the local git repository** before creating packages, to make sure that they are built from the correct git revision. -- Pull from GitHub and switch to the new version tag: - - .. code-block:: bash - - git checkout master - git pull - git checkout vX.Y.Z +.. _dev_rel_create_pkgs: +6. Create packages for PyPI and Conda +------------------------------------- +The packages should be built on the release branch to make sure that the version information is correct. - Making the packages for PyPI is straightforward. However, **make sure you delete old** ``build`` **directories** since they can pollute new builds: @@ -126,7 +158,7 @@ It is important to **update the local git repository** before creating packages, source deactivate conda install conda-build -- Invoke the following command to build a package for your platform and all supported Pyhton versions: +- Invoke the following command to build a package for your platform and all supported Python versions: .. code-block:: bash @@ -150,7 +182,9 @@ It is important to **update the local git repository** before creating packages, Replace ```` by the package file as built by the previous ``conda build`` command. -6. Test installing the local packages and check them + +.. _dev_rel_test_pkgs: +7. Test installing the local packages and check them ---------------------------------------------------- Before actually uploading packages to "official" servers, first install the local packages and run the unit tests. @@ -175,7 +209,9 @@ Before actually uploading packages to "official" servers, first install the loca conda install --use-local nomkl odl python -c "import odl; odl.test()" -7. Upload the packages to the official locations + +.. _dev_rel_upload_pkgs: +8. Upload the packages to the official locations ------------------------------------------------ Installing the packages works, now it's time to put them out into the wild. @@ -206,15 +242,17 @@ Installing the packages works, now it's time to put them out into the wild. For this step, you need the access credentials for the ``odlgroup`` user on the Anaconda server. -8. Bump current ``master`` to a development version ---------------------------------------------------- -To ensure a larger version number for installations from the git ``master`` branch, the version number must be increased immediately. -- Change the version string ``'X.Y.Z'`` in ``odl/__init__.py`` to ``'X.Y.Z+1.dev0'`` (e.g. from ``'0.5.3'`` to ``'0.5.4.dev0'``). -- Change the ``git_rev`` field in ``conda/meta.yaml`` to ``'master'``. -- Commit the changes, using a message like ``REL: bump version to X.Y.Z.dev0``. -- Make a PR with just this change and merge it after review. - It should be the first one that goes in after the release. +.. _dev_rel_merge_release_pr: +9. Merge the release branch +--------------------------- +Now the release branch can finally be merged. +The sole purpose of this step is to update the release notes on ``master`` and potentially get the last minute changes. + +- The release branch will have conflicts with ``master`` since both have modified the version information. + Resolve them in favor of the changes made on ``master``. + In particular, make sure that the changes in :ref:`Section 4` stay intact. +- Merge the PR for the release. Done! ----- From ff67142aecbdf08d8b05f5783883b7354cdcf632 Mon Sep 17 00:00:00 2001 From: Holger Kohr Date: Wed, 15 Feb 2017 01:06:22 +0100 Subject: [PATCH 8/8] DOC: clarify some things in the release docs --- doc/source/dev/release.rst | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/doc/source/dev/release.rst b/doc/source/dev/release.rst index d638892d95a..9f67ec7a45a 100644 --- a/doc/source/dev/release.rst +++ b/doc/source/dev/release.rst @@ -20,7 +20,7 @@ This involves the "what" and "when" of the release process and fixes a feature s The steps are: - Open an issue on the issue tracker using the title **Release X.Y.Z** (insert numbers, of course). -- Discuss and agree on a set of open PRs that should be merged before making a release. +- Discuss and agree on a set of open PRs that should be merged and issues that should be resolved before making a release. .. _dev_rel_master_ok: @@ -89,7 +89,7 @@ When all tests succeed and the docs are fine, start a release branch. .. _dev_rel_bump_master: 4. Bump the ``master`` branch to the next development version ------------------------------------------------------------- -To ensure a larger version number for installations from the git master branch, the version number must be increased before merging the release branch. +To ensure a higher version number for installations from the git master branch, the version number must be increased before merging the release branch. - On the ``master`` branch, change the version string in ``odl/__init__.py`` to the next revision larger than the upcoming release version, plus ``'dev0'``. For example, if the release version string is ``'0.5.3'``, use ``'0.5.4.dev0'``. @@ -116,8 +116,14 @@ To ensure a larger version number for installations from the git master branch, ---------------------------------- Back on the release branch with a ``git checkout release-X.Y.Z``, it is now time to prepare the release documents, increment the version number and make a release on GitHub. +.. note:: + The release notes should actually be a running document where everybody who files a PR also makes an entry into the release notes file. + If not, tough on you -- it is your duty now to make up for all that missed work. + Maybe you'll remind your co-workers to do this in their next PR. + - Compile the release notes. - They should contain all *user-visible* changes (internal stuff like test modifications is not required) and should be summarized in one or two sentences on top, perhaps mentioning the most notable changes. + They should contain all *user-visible* changes, including performance improvements and other niceties -- internal stuff like test modifications don't belong here. + The changes should be summarized in one or two sentences on top, perhaps mentioning the most notable ones in this release. Check the `Release Notes `_ file for details on sections, formatting etc. - Increment the version number in ``odl/__init__.py`` and ``conda/meta.yaml``. As in :ref:`Section 4`, perform a search to make sure you didn't miss a version info location. @@ -168,13 +174,14 @@ The packages should be built on the release branch to make sure that the version conda build conda/ --python 3.6 ... -- Assuming this succeeds, enter the directory one above where the conda package was stored (as printed in the output), e.g., +- Assuming this succeeds, enter the directory one above where the conda package was stored (as printed in the output). + For example, if the package was stored as ``$HOME/miniconda3/conda-bld/linux-64/odl-X.Y.Z-py36_0.bz2``, issue the command .. code-block:: bash cd $HOME/miniconda3/conda-bld/ - There, for each Python version "translate" the package to all platforms since ODL is actually platform-independent: + In this directory, for each Python version "translate" the package to all platforms since ODL is actually platform-independent: .. code-block:: bash @@ -230,7 +237,8 @@ Installing the packages works, now it's time to put them out into the wild. cd /path/to/odl_repo twine upload -u odlgorup dist/ - This requires the access credentials for the ``odlgroup`` user on PyPI. + This requires the access credentials for the ``odlgroup`` user on PyPI -- the maintainers have them. + - Upload the conda packages to the ``odlgroup`` channel in the Anaconda cloud. The upload requires the ``anaconda-client`` package: @@ -241,7 +249,7 @@ Installing the packages works, now it's time to put them out into the wild. anaconda upload -u odlgroup `find . -name "odl-X.Y.Z*"` For this step, you need the access credentials for the ``odlgroup`` user on the Anaconda server. - + Talk to the maintainers to get them. .. _dev_rel_merge_release_pr: 9. Merge the release branch