Skip to content

OpenWIS Testing Regime

lmika-bom edited this page Nov 27, 2014 · 2 revisions

This page documents the current test and release regime used for the development of OpenWIS.

Testing Regime

This has been adopted from the current practice used to test OpenWIS 3.13. It is to be read as a guide, rather than a rigid procedure.

  1. A separate branch for the release is created from the tip of develop. The branch will have the name openwis-X where X is the version that is to be released (e.g. openwis-3.13).
  2. A new Jenkins build will be created for building releases from this branch. An initial build will be created.
  3. After the build has finished, each partner involved in the testing and bug-fixing of the release will install the artefacts from the build onto their own internal testing environment.
  4. The partners involved in testing will test features and bugs in the following order:
    • Retesting of any bugs with fixes merged in the previous build.
    • Testing of new features added since the previous build, either their own or those of another partner.
    • Testing of "baseline functionality" of OpenWIS itself, to confirm stableness of the software.
  5. Any bugs discovered will be added as issues in GitHub. These issues will be given the label Bug and will be created unassigned.
  6. Partners volunteering to fix a bug will choose one by assigning the bug to themselves. Partners are free to choose any bugs marked for fixing for this release.
  7. Once a fix for a bug has been implemented, the fixer will create a pull request. Another partner (the reviewer) will review the changes within that pull request.
  8. Once the reviewer indicates that the pull request is acceptable, the pull request will be merged and the issue closed.
    • Pull requests are to be merged into the release branch (openwis-X).
    • A separate pull request should be made for merging the fix into develop.
  9. Once all bugs to be fixed have been closed, another build is created and retesting of the bugs will be conducted from step 3.
  10. When there are no more bugs to be fixed remaining for this release, the release can officially be announced.

Releasing OpenWIS

There are four key decisions that need to be made:

  • A release criteria (what do we want?)
  • A release date (when do we want it?)
  • A testing regime (where are we now?)
  • A bug-fixing regime (how do we get to where we want to be?)

The Release Criteria

This is the condition that the codebase needs to be in in order for it to be deemed acceptable for release. This can be either be as simple or as complex as deemed necessary, but probably should be documented and agreed upon by all TMC members.

Example criteria’s may include:

  • The number of critical, major, minor or trivial bugs deemed acceptable for a release (e.g. no major bugs, X minor bugs, Y trivial bugs).
  • The scope of bugs that are chosen to be fixed for this release (e.g. only bugs raised from regressions and new features will be fixed for this release: existing bugs will be out of scope).
  • The properties of the bug. Large bugs which may destabilise the feature might be resolved by turning off the feature instead of delaying the release indefinitely.

The Release Date

A target release date should be specified. By this date, the code must match the release criteria and the release cut and publicised. If the code does not match the release criteria, either the date can be pushed back, or the criteria adjusted to allow the release to go ahead (depending on the requirements of various stakeholders).