Add a roadmap for our policy regarding dependencies (#594)

DEPENDENCIES_ROADMAP.md

@@ -0,0 +1,182 @@
+# Package dependencies and reproducibility in OGGM: a roadmap
+
+
+OGGM relies on a large number of external python packages (dependencies).
+Many of them have complex dependencies themselves, often compiled binaries
+(for example `rasterio`, which relies on a C package: `GDAL`).
+
+The complexity of this dependency tree as well as the permanent updates of 
+both OGGM and its dependencies has lead to several unfortunate situations
+in the past: this involved a lot of maintenance work for the OGGM 
+developers that had little or nothing to do with the model itself.
+
+Furthermore, while the vast majority of the dependency updates are 
+without consequences, some might change the model results. As an example,
+updates in the interpolation routines of GDAL/rasterio can change the 
+glacier topography in a non-traceable way for OGGM. This is an obstacle
+to [reproducible science](https://en.wikipedia.org/wiki/Reproducibility), 
+and we should try to avoid these situations.
+
+OGGM, as a high level "top of the dependency pyramid" software, does not have
+the same requirements in terms of being always up-to-date as, say, general
+purpose libraries like pandas or matplotlib. With this document, the OGGM 
+developers attempt to define a clear policy for handling package dependencies
+in the OGGM project. This policy can be seen as a set of "rules" or 
+"guidelines" that we will try to follow as much as possible.
+
+
+## Example situations
+
+Here are some example of situations that happened in the past or might happen
+in the future:
+
+**Situation 1.**
+   A new user installs OGGM with `conda install oggm-deps` and the installation 
+   fails because one of one of the dependencies. This 
+   is a problem in conda (or conda-forge) and has nothing to do with OGGM.
+
+**Situation 2.**
+   A new user installs OGGM with `conda install oggm-deps`: the installation 
+   succeeds, but the OGGM tests fail to pass. One of our dependencies 
+   renamed a functionality or deprecated it, and we didn't update OGGM in time.
+   This requires action in OGGM itself, but we don't always have time to 
+   fix it quickly enough, leading to an overall bad user experience.
+
+**Situation 3.**
+   A developer writes a new feature and sends a pull-request: the tests pass
+   on her machine but not on Travis. The failing tests are unrelated to the PR:
+   it is one of our dependency update that broke something in OGGM. 
+   This a bad experience for new developers, and it is the job of an OGGM core 
+   developer to solve the problem. 
+
+**Situation 4.**
+   A developer writes a quantitative test of the type:
+   "the model simulation should yield a volume of xx.xxx km3 ice": the test
+   passes locally but fails on Travis. Indeed, the glacier topography is 
+   slightly different on Travis because of difference in the installed GDAL
+   version, yielding non-negligible differences in model results.
+
+**Situation 5.**
+   Worst case scenario: someone tries to replicate the results of a simulation
+   published in a paper, and her results are off by 10% to the published 
+   ones. A reason could be that the system and/or 
+   package dependencies are different for the new simulation environment. But
+   which results are the correct ones? 
+
+These situations are frequent and apply to any project with complex 
+dependencies (not only OGGM). **These problems currently do not have a simple,
+"out-of-the box" solution**. This document attempts to prioritize some of the 
+issues and provide guidelines for how to handle software dependencies 
+within the OGGM framework.
+
+
+## Goals
+
+Here is a set of goals that we should always try to follow:
+
+1. Stability is more important than dependency updates.
+2. A *standard dependency list* of the names and fixed version number of the 
+   major OGGM dependencies should be defined and documented. 
+   The standard dependency list has 
+   its own version number which is decoupled from the OGGM one.
+3. Updates of the standard dependency list should be rare, well justified and
+   documented. Examples for updates include: a new feature we 
+   want to use, a performance increase, or regular updates to keep track with 
+   the scientific python stack (e.g. twice a year).
+4. It should be possible to use a python environment fixed to the standard
+   dependency list on all these platforms: the Bremen cluster, on Travis, 
+   on a local linux machine and on a university cluster with singularity.
+5. The latest OGGM code should always run error-free on the latest standard 
+   dependency list. Older OGGM versions should have a standard dependency list 
+   version number attached to them, and scientific publications as well.
+6. As far as possible, OGGM should run on the latest version of all packages
+   as well. This rule is less important than Rule 5 and should not require
+   urgent handling by the OGGM developers.
+7. It should be possible for a new user to create a working environment based 
+   on the standard dependency list from scratch, either with a meta-package
+   (i.e. `conda install oggm-deps`) or a file 
+   (`conda install oggm-deps.yml` or `pip install requirements.txt`).
+8. If 7 fails, it should be possible for a user to create a working environment 
+   based on the latest dependencies from scratch, either with a meta-package
+   (i.e. `conda install oggm-deps-latest`) or a file 
+   (`conda install oggm-deps-latest.yml` or 
+   `pip install requirements-latest.txt`).
+9. We recommend users to define a fixed environment for OGGM. We do not take 
+   responsibility if users update packages on their own afterwards.
+10. OGGM should provide tools to quickly and easily test a user installation.
+11. OGGM should always work on Linux. Because of these dependency problems,
+    we make no guarantee that OGGM will work on other operating systems 
+    (Mac OSX, Windows).
+
+
+## Standard dependency list and updates (goals 1, 2, 3)
+
+The latest and all previous standard dependency lists can be found on the
+github repository: https://github.com/OGGM/OGGM-dependency-list
+
+Discussions about whether the standard dependency list should be updated or not
+will take place on this repository or the main OGGM repository.
+
+
+## Docker and Singularity containers (goal 4)
+
+The easiest way to guarantee consistency accross platforms and over longer
+periods of time are [containers](https://www.docker.com/resources/what-container).
+A container image is a lightweight, standalone, executable package of software 
+that includes everything needed to run an application: code, runtime, system 
+tools, system libraries and settings.
+
+The most widely used container platform is [Docker](https://www.docker.com/).
+For security and preformance reasons, HPC centers prefer to use 
+[Singularity](https://www.sylabs.io/). Fortunately, Singularity containers 
+can be built from Docker images.
+
+OGGM maintains an Ubuntu container image that users can download and use 
+for free, and convert it to a singularity image for use on HPC.
+
+The images can be found at: https://hub.docker.com/r/oggm/oggm/
+
+The build scripts can be found at https://github.com/OGGM/OGGM-Docker
+
+**OGGM releases will point to a specific version of the Docker image
+for reproducible results over time**.
+
+
+## Travis CI (goals 5, 6 and 11)
+
+[Travis CI](https://travis-ci.org/OGGM/oggm) is the tool we use for continuous
+testing of the OGGM software. **The tests should run on the stable
+Docker image built with the standard dependency list**. Optionally,
+we will monitor the tests on the latest image as well, but the tests are 
+not guaranteed to pass ("allowed failures").
+
+**The main OGGM reporistory will not test on other platforms than Linux**. 
+We might run the tests for other platforms as well, but this is without 
+guarantee and should happen on a separate repository (e.g. on 
+https://github.com/OGGM/OGGM-dependency-list).
+
+
+## pip and conda (goals 7 and 8)
+
+Docker and Singularity containers are the most secure and consistent way to 
+run OGGM. However, they require some knowledge about containers and the 
+command line, and they still do not belong to the standard set of tools 
+of most scientists.
+
+Therefore, we should help and support users in installing OGGM dependencies
+"the standard way", i.e. using pip or conda. We can do this by maintaining 
+and testing the standard and latest dependency lists on various platforms
+as often as possible. When problems arise, we can attempt to fix them but
+make no guarantee for the problems generated upstream, i.e. problems which are 
+unrelated to OGGM itself.
+
+
+## Check install (goal 10)
+
+The user will have two complementary ways to test the correct installation
+of OGGM:
+- `pytest --pyargs oggm --run-slow --mpl` will run the test suite. This test
+  suite does not contain quantitative tests, i.e. it does not guarantee
+  consistency of model results accross platforms
+- `oggm.check_install()` will be a top level function performing quantitative 
+  tests to see if user results are consistent with the benchmark container.