incremented version to 2.5

README.rst

@@ -1,35 +1,86 @@
-The ILAMB Benchmarking System
-=============================
+ILAMB 2.5 Release
+=================
 
-The International Land Model Benchmarking (ILAMB) project is a
-model-data intercomparison and integration project designed to improve
-the performance of land models and, in parallel, improve the design of
-new measurement campaigns to reduce uncertainties associated with key
-land surface processes. Building upon past model evaluation studies,
-the goals of ILAMB are to:
+We are pleased to announce a new version of the ILAMB package. In
+addition to a new version, the ILAMB repository is now hosted at:
 
-* develop internationally accepted benchmarks for land model
-  performance, promote the use of these benchmarks by the
-  international community for model intercomparison,
-* strengthen linkages between experimental, remote sensing, and
-  climate modeling communities in the design of new model tests and
-  new measurement programs, and
-* support the design and development of a new, open source,
-  benchmarking software system for use by the international community.
+  `https://github.com/rubisco-sfa/ILAMB <https://github.com/rubisco-sfa/ILAMB>`_
+
+This release includes some interface improvements as well as core
+technology enhancements increasing speed and reliability. See the
+following list for major changes:
+
+* The landing `page <http://www.ilamb.org/CMIP6/historical/>`_
+  generated from `ilamb-run` has received an overhaul. We have merged
+  the two tabs with the image and data into one dynamic table which
+  can be clicked. Clicking on a row header will either expand the row
+  or take you to the dataset page. Clicking on a particular model's
+  square will take you to the dataset page with that particular model
+  highlighted. In addition to this, you can now select which scalar
+  you wish to plot in the table (i.e. Overall Score, Bias Score) over
+  any region included in the study.
+* The appearance of the `Data Information` tabs on the dataset pages
+  has been greatly enhanced. References can be included in the netCDF
+  files in Bibtex format and will be rendered inside of
+  ILAMB. Hyperlinks also will be detected and rendered as clickable
+  links in the output pages. Thanks to Mingquan Mu for this addition.
+* Added a soil carbon temperature sensitivity metric from Charlie
+  Koven, added this to our curated configure file `cmip.cfg`.
+* The CO2 emulation code will now account for ocean and fossil fuel
+  fluxes when emulating the land model's `nbp`. Thanks to Ke Xu for this
+  contribution.
+* We have added new datasets `LORA
+  <http://dx.doi.org/10.25914/5b612e993d8ea>`_ for runoff and `DOLCE
+  <http://dx.doi.org/10.4225/41/58980b55b0495>`_ for latent
+  heat. While these datasets include uncertainty estimates, we are
+  currently not making use of them in our analysis.
+* We have replaced `basemap <https://github.com/matplotlib/basemap>`_
+  in favor of `cartopy <https://github.com/SciTools/cartopy>`_ as the
+  tool for plotting on maps. Not only is this needed as basemap is
+  being deprecated, but plotting is now approximately 10x faster. For
+  the most part, this change will be invisible to the user.
+* Added options and structure to `ilamb-run` to improve runtimes. If
+  running a large set of models on a cluster, we recommend first
+  running with the `--skip_plots` option and using a low number of
+  processes per node. This is because memory utilization tends to
+  dominate the analysis phase and you do not want to run out. Then you
+  can submit a second job without `--skip_plots` and using a large
+  number of processes per node.
+* Intermediate files generated during `ilamb-run` will now include a
+  `complete` flag, initialized to `False` and only flagged true if the
+  file closed at the end of the analysis phase without error. This
+  helps us reinitialize `ilamb-run` when a parallel run crashes and
+  leaves file present and not corrupted, but neither complete.
+* If the `psutil` python package is installed, `ilamb-run` will now
+  log the peak memory being used during the analysis phase in the
+  logfiles along with the node name and process rank. This is to help
+  in memory debugging for when high resolution models are being run.
+* In addition to this, `ilamb-run` now caches the model initialization
+  process which should speed up re-initialization for when multiple
+  jobs must be submitted.
+* Added initial support for uncertainty bounds in the
+  ILAMB.Variable. If uncertainty is included in the observations, such
+  as in the Hoffman `nbp` dataset, then ILAMB will automatically
+  operate of it and show it as a shading in plots without changes to
+  your scripts.
+* `ilamb-fetch` now will correctly try to decode server SSL
+  certificates before downloading files. However, the authority that
+  `www.ilamb.org` uses to create its certificates is not in the list
+  that python supports. Your browser maintains a different list of
+  authorities, which is why you can navigate to sites like `this
+  <http://www.ilamb.org/CMIP6/historical/>`_. You will likely need to
+  run with the `--no-check-certificate` option which implies that you
+  trust that we are who we say we are.
 
-It is the last of these goals to which this repository is
-concerned. We have developed a python-based generic benchmarking
-system, initially focused on assessing land model performance.
-
 Useful Information
 ------------------
 
 * `Documentation <https://www.ilamb.org/doc/>`_ - installation and
   basic usage tutorials
 * Sample Output
 
-  * `CLM <http://www.ilamb.org/CLM/>`_ - land comparison against 3 CLM versions
-  * `CMIP5 <http://www.ilamb.org/CMIP5/esmHistorical/>`_ - land comparison against a collection of CMIP5 models
+  * `CMIP6 <http://www.ilamb.org/CMIP6/historical/>`_ - land comparison against a collection of CMIP6 models
+  * `CMIP5 vs CMIP6 <http://www.ilamb.org/CMIP6/historical/>`_ - land comparison against a collection of CMIP5 and CMIP6 models
   * `IOMB <http://www.ilamb.org/IOMB/>`_ - ocean comparison against a few ocean models
 
 * `Paper <https://doi.org/10.1029/2018MS001354>`_ published in JAMES
@@ -38,74 +89,28 @@ Useful Information
   research or development efforts, we kindly ask you to cite this
   work.
 
-ILAMB 2.4 Release
------------------
+Description
+-----------
 
-This release marks an important technical shift in ILAMB
-development--ILAMB v2.4 and onward will be python3 only. If you are
-new to python, it might seem strange that python3 has been released
-for 10 years and yet python2 is still ubiquitous. There is now an
-official `announcement <https://pythonclock.org/>`_ that python2 will
-reach its end of life at the end of 2019. Furthermore there is a
-growing `list <https://python3statement.org/>`_ of popular python
-packages (most of our dependencies) that are phasing out support for
-python2 during this year. So in keeping with this community trend, the
-last version of ILAMB which will be compatible with python2.7x is
-2.3.1, version 2.4 and beyond will be python3 only.
-
-Part of my reason for sticking with python2 for so long was that ILAMB
-was designed to run on large machines whose software stacks are often
-not frequently updated. I wanted to ensure that ILAMB would run on old
-software. However, this is less an issue as computing centers are
-moving away from providing users with python packages they load via
-center-supported environment modules and towards users creating
-personalized environments using `conda
-<https://conda.io/docs/>`_. Look for the ``ilamb.yml`` file in the
-repository which conda can use to create an environment that will
-support an ILAMB installation. If these words do not mean anything to
-you, look for a more detailed explanation in the `tutorials
-<https://www.ilamb.org/doc/install.html>`_ which have been rewritten to
-reflect this shift.
-
-We have also changed the location of the supported configure files in
-the repository. They used to be located in the ``demo`` directory, but
-now may be found in ``src/ILAMB/data``. The motivation for this change
-is to have these files be installed along with the
-package. Previously, if a user were to get ILAMB via PyPI, these
-configure files would not install along with the package. This made
-the next step of finding a configure file confusing. Furthermore,
-these configure files change as we continue to work on the package and
-so it is important that they are installed along with the python
-source code. Now, for example, if you install ILAMB via ``pip install
-ILAMB --user``, you can expect to find the configure files in a
-location like
-``${HOME}/.local/lib/python3.7/site-packages/ILAMB/data/``.
-
-We have published a `paper <https://doi.org/10.1029/2018MS001354>`_ in
-JAMES which details the methodology which this package implements. If
-you find ILAMB helpful in your research, we would appreciate a
-citation to this work as it helps us communicate the impact that these
-investments have on the community.
+The International Land Model Benchmarking (ILAMB) project is a
+model-data intercomparison and integration project designed to improve
+the performance of land models and, in parallel, improve the design of
+new measurement campaigns to reduce uncertainties associated with key
+land surface processes. Building upon past model evaluation studies,
+the goals of ILAMB are to:
 
-The collection of land surface confrontations now includes the
-emulation of CO\ :sub:`2`\ fluxes. The default setup is to compare
-``nbp`` fluxes to those recorded at a subset of NOAA sites, but this
-is configurable from inside the configure file. Browse the CMIP5
-`output
-<https://www.ilamb.org/CMIP5/esmHistorical/EcosystemandCarbonCycle/CarbonDioxide/NOAA.Emulated/NOAA.Emulated.html>`_
-for an overview of what this addition provides.
+* develop internationally accepted benchmarks for land model
+  performance, promote the use of these benchmarks by the
+  international community for model intercomparison,
+* strengthen linkages between experimental, remote sensing, and
+  climate modeling communities in the design of new model tests and
+  new measurement programs, and
+* support the design and development of a new, open source,
+  benchmarking software system for use by the international community.
 
-Finally, we are making some shifts in how we support ILAMB. Until now,
-I have directed user questions to my personal email. This is still ok,
-however consider `joining
-<https://www.ilamb.org/mailman/listinfo/ilamb-users>`_ the ILAMB
-mailing list and sending your questions there. Not only does this open
-up your question to being answered more quickly by the community, but
-the answers are searchable which may help the next user. In addition
-to this, we have a Slack `channel
-<https://ilamb-community.slack.com/>`_ if you prefer to ask your
-questions there. This has more of a chat interface but the
-conversations are all still public and searchable by the members.
+It is the last of these goals to which this repository is
+concerned. We have developed a python-based generic benchmarking
+system, initially focused on assessing land model performance.
 
 Funding
 -------

setup.py

@@ -4,7 +4,7 @@
 import subprocess
 import os
 
-VERSION    = '2.4'
+VERSION    = '2.5'
 
 def git_version():
     """

src/ILAMB/__init__.py

@@ -1,6 +1,6 @@
 __author__       = 'Nathan Collier'
-__date__         = 'Jan 2019'
-__version__      = '2.4'
+__date__         = 'Sep 2019'
+__version__      = '2.5'
 
 from distutils.version import LooseVersion
 import platform