Merge branch '751-release-docs'

doc/index.rst

@@ -104,6 +104,7 @@ For CKAN Developers
    i18n
    test
    release-cycle
+   release-process
 
 Other material
 ==============

doc/release-cycle.rst

@@ -1,41 +1,69 @@
-=============
-Release Cycle
-=============
-
-In order to ensure that our releases are stable and backwards compatible, we
-follow a strict release process. There are two kinds of releases:
-
-Point releases (e.g 1.8)
-------------------------
-
-These will be branched from master at a certain point (we aim to do a point
-release roughly every two or three months). There will be a period of three
-weeks until the actual release. During the first two weeks changes would be
-allowed  to stabilize the code, update i18n etc. During the last week only
-critical bug fixes will be allowed. Point releases are merged back into master
-after the actual release.
-
-Point point releases (e.g 1.8.1)
---------------------------------
-
-These are branched from every point release and *must not* break compatibility:
-
-- No DB migrations or schema changes
-- No function interface changes
-- No plugin interface changes
-- No new dependencies
-- No big refactorings or new features on critical parts of the code
-- Point point releases are not merged back into master, and all changes must be
-  cherry-picked from master.
-- Point point releases are released as needed depending on the severity of the
-  bugs fixed. They will be distributed via the same apt repository as their
-  parent release.
-
-Allowed optimizations and small features in point point releases are somewhat
-open to interpretation. In any case, the release manager (or another single
-developer designated by him) has the final say on what is merged into release
-branches.
-
-Both core and external supported extensions need to be tested before the
-release (during the two week period after branching and before the final
-freeze). These supported extensions need to defined as soon as possible. 
+CKAN Release Cycle
+==================
+
+CKAN follows a standardized release process to ensure that releases
+are stable and backwards compatible, and to allow external developers to
+oversee the development process and plan future upgrades.
+
+Release types
+-------------
+
+Versions for each release are numbered as *X.Y* or *X.Y.Z*. There are three
+types of releases:
+
+* **Major Releases**
+    These increment the main version count (eg *2.0*) and
+    represent major changes in the CKAN code base, with significant refactorings
+    and breaking changes, for instance on the API or the templates. These releases
+    are very infrequent, 2.0 will be the first one in CKAN's history.
+
+* **Point Releases**
+    These increment the second version number (eg *2.1*) and
+    represent stable lines among CKAN releases. Although not as disruptive as on
+    major releases, backwards incompatible changes may be introduced on point
+    releases. The :doc:`CHANGELOG` will describe which are these incompatible changes.
+    We aim to do a point release roughly every three months.
+
+* **Point-point Releases**
+    These increment the third version number (eg *2.1.3*)
+    and don't break compatibility. That means for example that an application
+    running CKAN 1.8 or 1.8.1 can be safely upgraded to 1.8.2 or 1.8.3. These
+    releases are branched from their point release or from the last point-point
+    release if any. They only include bug fixes and non-breaking optimizations or
+    small features. They must not include:
+
+  - DB migrations or schema changes
+  - Function interface changes
+  - Plugin interface changes
+  - New dependencies
+  - Big refactorings or new features on critical parts of the code
+
+
+CKAN Release Process Overview
+-----------------------------
+
+When the development team thinks master is at a good point to start the
+release process a new branch will be created with the name *release-v{version
+number}*. This is the beta branch for this release, and it will be deployed to
+`beta.ckan.org <http://beta.ckan.org>`_. During the next two-three weeks
+changes will be allowed to stabilize the code, update i18n and documentation,
+etc. During the last week, only critical bug fixes are allowed.
+
+At some point during the beta process a translation freeze will be put in
+place. That means that no changes to the translatable strings are allowed (new
+strings or changes on existing ones). This will give time to translators to
+update the translations on Transifex_.
+
+Release branches are not merged back into master. All changes must be
+cherry-picked from master (or merged from special branches based on the release
+branch).
+
+To ensure that the release guidelines are enforced one of the CKAN core
+developers will act as Release Manager. They have the final say on what is
+merged into the release branches.
+
+The actual process followed by CKAN developers can be found in
+:doc:`release-process`.
+
+
+.. _Transifex: https://www.transifex.com/projects/p/ckan

doc/release-process.rst

@@ -0,0 +1,223 @@
+Doing a CKAN Release
+====================
+
+These are the steps followed by CKAN developers to do a release. To get an
+overview of CKAN releases, check :doc:`release-cycle`.
+
+
+Doing a Beta release
+--------------------
+
+Beta releases are branched off a certain point in master and will eventually
+become stable releases.
+
+1. Create a new release branch::
+
+        git checkout -b release-v1.8
+
+   Update ``ckan/__init__.py`` to change the version number to the new version
+   with a *b* after it, e.g. *1.8b*.
+   Commit the change and push the new branch to GitHub::
+
+        git commit -am "Update version number"
+        git push origin release-v1.8
+
+   You will probably need to update the same file on master to increase the
+   version number, in this case ending with an *a* (for alpha).
+
+2. During the beta process, all changes to the release branch must be
+   cherry-picked from master (or merged from special branches based on the
+   release branch if the original branch was not compatible).
+
+3. As in the master branch, if some commits involving CSS changes are
+   cherry-picked from master, the less compiling command needs to be run on
+   the release branch. This will update the ``main.css`` file::
+
+        ./bin/less --production
+        git commit -am "Rebuild CSS"
+        git push
+
+   There will be a final front-end build before the actual release.
+
+4. The beta staging site (http://beta.ckan.org, currently on s084) should be
+   updated regularly to allow user testing.
+
+5. Once the translation freeze is in place (ie no changes to the translatable
+   strings are allowed), strings need to be extracted and uploaded to
+   Transifex_:
+
+   a. Install the Babel and Transifex libraries if necessary::
+
+        pip install --upgrade Babel
+        pip install transifex-client
+
+   b. Create a ``~/.transifexrc`` file if necessary with your login details
+      (``token`` should be left blank)::
+
+        [https://www.transifex.com]
+        hostname = https://www.transifex.com
+        username = <username>
+        password = <password>
+        token =
+
+   c. Extract new strings from the CKAN source code into the ``ckan.pot``
+      file. The pot file is a text file that contains the original,
+      untranslated strings extracted from the CKAN source code.::
+
+        python setup.py extract_messages
+
+   d. Pull new and updated translations from Transifex into the ``ckan.po``
+      files::
+
+        tx pull --all --force
+
+      The po files are text files, one for each language CKAN is translated to,
+      that contain the translated strings next to the originals. Translators edit
+      the po files (on Transifex) to update the translations. We never edit the
+      po files locally.
+
+      ``--force`` tells Transifex to update all ``ckan.po`` files, regardless of the
+      modification time.
+
+   e. Run our script that checks for mistakes in the ckan.po files::
+
+        pip install polib
+        paster check-po-files ckan/i18n/*/LC_MESSAGES/ckan.po
+
+      If the script finds any mistakes correct them on Transifex and then run the
+      tx pull command again, don't edit the files directly. Repeat until the
+      script finds no mistakes.
+
+   f. Edit ``.tx/config``, on line 4 to set the Transifex 'resource' to the new
+      major release name (if different), using dashes instead of dots.
+      For instance v1.2, v1.2.1 and v1.2.2 all share: ``[ckan.1-2]``.
+
+   g. Update the ``ckan.po`` files with the new strings from the ``ckan.pot`` file::
+
+        python setup.py update_catalog --no-fuzzy-matching
+
+      Any new or updated strings from the CKAN source code will get into the po
+      files, and any strings in the po files that no longer exist in the source
+      code will be deleted (along with their translations).
+
+      We use the ``--no-fuzzy-matching`` option because fuzzy matching often
+      causes problems with Babel and Transifex.
+
+   h. Create a new resource in the CKAN project on Transifex by pushing the new
+      pot and po files::
+
+        tx push --source --translations --force
+
+      Because it reads the new version number in the ``.tx/config`` file, tx will
+      create a new resource on Transifex rather than updating an existing
+      resource (updating an existing resource, especially with the ``--force``
+      option, can result in translations being deleted from Transifex).
+
+   i. Update the ``ckan.mo`` files by compiling the po files::
+
+        python setup.py compile_catalog
+
+      The mo files are the files that CKAN actually reads when displaying
+      strings to the user.
+
+   j. Commit all the above changes to git and push them to GitHub::
+
+        git commit -am " Update strings files before CKAN X.Y call for translations"
+        git push
+
+   k. Announce that strings for the new release are ready for translators. Send
+      an email to the mailing lists, tweet or post it on the blog. Make sure to
+      post a link to the correct Transifex resource (like
+      `this one <https://www.transifex.com/projects/p/ckan/resource/2-0/>`_)
+      and tell users that they can register on Transifex to contribute.
+
+   l. A week before the translations will be closed send a reminder email.
+
+   m. Once the translations are closed, pull the updated strings from Transifex,
+      check them, compile and push as described in the previous steps::
+
+        tx pull --all --force
+        paster check-po-files ckan/i18n/*/LC_MESSAGES/ckan.po
+        python setup.py compile_catalog
+        git commit -am " Update translations from Transifex"
+        git push
+
+
+Doing a Proper release
+----------------------
+
+Once the release branch has been thoroughly tested and is stable we can do
+a release.
+
+1. Run the most thorough tests::
+
+        nosetests ckan/tests --ckan --ckan-migration --with-pylons=test-core.ini
+
+2. Do a final build of the front-end and commit the changes::
+
+        paster front-end-build
+        git commit -am "Rebuild front-end"
+
+3. Update the CHANGELOG.txt with the new version changes:
+
+   * Add the release date next to the version number
+   * Add the following notices at the top of the release, reflecting whether
+     updates in requirements, database or Solr schema are required or not::
+
+        Note: This version requires a requirements upgrade on source installations
+        Note: This version requires a database upgrade
+        Note: This version does not require a Solr schema upgrade
+
+   * Check the issue numbers on the commit messages for information about
+     the changes. These are some helpful git commands::
+
+        git branch -a --merged > merged-current.txt
+        git branch -a --merged ckan-1.8.1 > merged-previous.txt
+        diff merged-previous.txt merged-current.txt
+
+        git log --no-merges release-v1.8.1..release-v2.0
+        git shortlog --no-merges release-v1.8.1..release-v2.0
+
+4. Check that the docs compile correctly::
+
+        rm build/sphinx -rf
+        python setup.py build_sphinx
+
+5. Remove the beta letter in the version number in ``ckan/__init__.py``
+   (eg 1.1b -> 1.1) and commit the change::
+
+        git commit -am "Update version number for release X.Y"
+
+6. Tag the repository with the version number, and make sure to push it to
+   GitHub afterwards::
+
+        git tag -a -m '[release]: Release tag' ckan-X.Y
+        git push --tags
+
+7. Upload the release to PyPI::
+
+        python setup.py sdist upload
+
+   You will need a PyPI account with admin permissions on the ckan package,
+   and your credentials should be defined on a ``~/.pypirc`` file, as described
+   `here <http://docs.python.org/distutils/packageindex.html#pypirc>`_
+   If you make a mistake, you can always remove the release file on PyPI and
+   re-upload it.
+
+8. Enable the new version of the docs on Read the Docs (you will need an admin
+   account):
+
+   a. Go to the `versions page <http://readthedocs.org/dashboard/ckan/versions/>`
+      and enable the relevant release (make sure to use the tag, ie ckan-X.Y,
+      not the branch, ie release-vX.Y).
+
+   b. If it is the latest stable release, set it to be the Default Version and
+      check it is displayed on http://docs.ckan.org.
+
+9. Write a `CKAN Blog post <http://ckan.org/wp-admin>`_ and send an email to
+   the mailing list announcing the release, including the relevant bit of
+   changelog.
+
+
+.. _Transifex: https://www.transifex.com/projects/p/ckan
+.. _ReadTheDocs: http://readthedocs.org/dashboard/ckan/versions/