Merge branch 'master' into refactor_diagnostics

.pre-commit-config.yaml

@@ -8,7 +8,7 @@ exclude: |
   ^esmvalcore/preprocessor/ne_masks/
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v2.4.0
+    rev: v3.4.0
     hooks:
       - id: check-added-large-files
       - id: check-ast
@@ -19,26 +19,26 @@ repos:
       - id: trailing-whitespace
         args: [--markdown-linebreak-ext=md]
   - repo: https://github.com/adrienverge/yamllint
-    rev: ''
+    rev: 'v1.26.0'
     hooks:
       - id: yamllint
   - repo: https://github.com/codespell-project/codespell
-    rev: ''
+    rev: 'v2.0.0'
     hooks:
       - id: codespell
   - repo: https://github.com/PyCQA/isort
-    rev: ''
+    rev: '5.7.0'
     hooks:
       - id: isort
   - repo: https://github.com/pre-commit/mirrors-yapf
-    rev: ''
+    rev: 'v0.30.0'
     hooks:
       - id: yapf
   - repo: https://github.com/myint/docformatter
-    rev: ''
+    rev: 'v1.4'
     hooks:
       - id: docformatter
   - repo: https://gitlab.com/pycqa/flake8
-    rev: ''
+    rev: '3.8.4'
     hooks:
       - id: flake8

CITATION.cff

@@ -158,11 +158,11 @@ authors:
     "orcid": "https://orcid.org/0000-0003-0590-7843"
 
 cff-version: "1.0.3"
-date-released: 2020-10-12
+date-released: 2021-2-8
 doi: "10.5281/zenodo.3387139"
 license: "Apache-2.0"
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ESMValGroup/ESMValCore/"
 title: ESMValCore
-version: "v2.1.0"
+version: "v2.2.0"
 ...

CODE_OF_CONDUCT.md

@@ -55,7 +55,8 @@ further defined and clarified by project maintainers.
 ## Enforcement
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at veronika.eyring@dlr.de. All
+reported by contacting the project team at birgit.hassler@dlr.de or
+alistair.sellar@metoffice.gov.uk. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.

NOTICE

@@ -14,12 +14,14 @@ Copyright 2008- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR) and partne
 ================================================
 (1) ESMValCore VERSION 2.0 CORE DEVELOPMENT TEAM
 ================================================
-Copyright 2008- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Germany - ESMValTool Principal Investigator (PI)
+Copyright 2008- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Germany - ESMValTool Co-PI
+Copyright 2020- Met Office, UK - ESMValTool Co-PI
 Copyright 2017- Alfred-Wegener-Institute (AWI), Germany
 Copyright 2017- Barcelona Supercomputing Center (BSC), Spain
 Copyright 2016- Ludwig Maximilian University, Germany
-Copyright 2017- Netherlands e-Science Center (NLeSC), Netherlands
+Copyright 2017- Netherlands e-Science Center (NLeSC), The Netherlands
 Copyright 2019- Plymouth Marine Laboratory, UK
+Copyright 2011- Swedish Meteorological and Hydrological Institute (SMHI), Sweden
 Copyright 2017- University of Reading, UK
 
 =======================================
@@ -33,10 +35,10 @@ Unless required by applicable law or agreed to in writing, software distributed
 
 ==========================================
 
-Users who apply the Software resulting in presentations or papers are kindly asked to cite the following “Software Documentation Paper” alongside with the Software doi (doi:10.17874/ac8548f0315) and version number:
-Eyring et al., ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth System Models in CMIP, Geosci. Model Dev., 2016.
+Users who apply the Software resulting in presentations or papers are kindly asked to cite the following “Software Documentation Paper” alongside with the Software doi (doi:10.5281/zenodo.3952695) and version number:
+Righi et al., Earth System Model Evaluation Tool (ESMValTool) v2.0 - technical overview, Geosci. Model Dev., 2020.
 
-Besides the above citation, users are kindly asked to register any journal articles (or other scientific documents) that use the Software at ESMValTool webpage (see http://www.esmvaltool.org/).
+Besides the above citation, users are kindly asked to register any journal articles (or other scientific documents) that use the Software at ESMValTool webpage (see https://www.esmvaltool.org/).
 Citing the Software Documentation Paper and registering your paper(s) will serve to document the scientific impact of the Software, which is of vital importance for securing future funding. You should consider this an obligation if you have taken advantage of the Software, which represents the end product of considerable effort by the development team.
 
 ==========================================
@@ -45,9 +47,8 @@ In addition to using the Software, we encourage the community to join the Softwa
 
 ==========================================
 
-To join the ESMValTool Development Team, please contact Prof. Veronika Eyring (veronika.eyring@dlr.de) and Dr. Axel Lauer (axel.lauer@dlr.de).
+To join the ESMValTool Development Team, please contact Dr. Birgit Hassler (birgit.hassler@dlr.de) and Dr. Axel Lauer (axel.lauer@dlr.de).
 
 ==========================================
 
 
-

README.md

@@ -39,7 +39,7 @@ The ESMValCore package provides various functions for:
 -   Reading CMIP/CMOR tables and using those to check model and observational data.
 
 -   ESMValTool preprocessor functions based on
-    [iris](https://scitools.org.uk/iris/docs/latest/) for e.g. regridding,
+    [iris](https://scitools-iris.readthedocs.io) for e.g. regridding,
     vertical interpolation, statistics, correcting (meta)data errors, extracting
     a time range, etcetera.
 

doc/api/esmvalcore.api.config.rst

@@ -3,13 +3,13 @@
 Configuration
 =============
 
-This section describes the config submodule of the API.
+This section describes the :py:class:`~esmvalcore.experimental.config` submodule of the API (:py:mod:`esmvalcore.experimental`).
 
 Config
 ******
 
-Configuration of ESMValCore/Tool is done via the ``Config`` object.
-The global configuration can be imported from the ``esmvalcore.experimental`` module as ``CFG``:
+Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.experimental.config.Config` object.
+The global configuration can be imported from the :py:mod:`esmvalcore.experimental` module as :py:data:`~esmvalcore.experimental.CFG`:
 
 .. code-block:: python
 
@@ -36,7 +36,7 @@ The global configuration can be imported from the ``esmvalcore.experimental`` mo
 
 The parameters for the user configuration file are listed `here <https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#user-configuration-file>`__.
 
-``CFG`` is essentially a python dictionary with a few extra functions, similar to ``matplotlib.rcParams``.
+:py:data:`~esmvalcore.experimental.CFG` is essentially a python dictionary with a few extra functions, similar to :py:mod:`matplotlib.rcParams`.
 This means that values can be updated like this:
 
 .. code-block:: python
@@ -45,27 +45,27 @@ This means that values can be updated like this:
     >>> CFG['output_dir']
     PosixPath('/home/user/esmvaltool_output')
 
-Notice that ``CFG`` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
+Notice that :py:data:`~esmvalcore.experimental.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
 All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
 
 .. code-block:: python
 
-    >>> CFG['otoptu_dri'] = '~/esmvaltool_output'
-    InvalidConfigParameter: `otoptu_dri` is not a valid config parameter.
+    >>> CFG['output_directory'] = '~/esmvaltool_output'
+    InvalidConfigParameter: `output_directory` is not a valid config parameter.
 
 Or, if the value entered cannot be converted to the expected type:
 
 .. code-block:: python
 
-    >>> CFG['max_years'] = '🐜'
-    InvalidConfigParameter: Key `max_years`: Could not convert '🐜' to int
+    >>> CFG['max_parallel_tasks'] = '🐜'
+    InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
 
-``Config`` is also flexible, so it tries to correct the type of your input if possible:
+:py:class:`~esmvalcore.experimental.config.Config` is also flexible, so it tries to correct the type of your input if possible:
 
 .. code-block:: python
 
-    >>> CFG['max_years'] = '123'  # str
-    >>> type(CFG['max_years'])
+    >>> CFG['max_parallel_tasks'] = '8'  # str
+    >>> type(CFG['max_parallel_tasks'])
     int
 
 By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
@@ -87,16 +87,16 @@ Session
 *******
 
 Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the ``Session`` object.
-A ``Session`` can be initiated from the global ``Config``.
+This behaviour can be controlled via the :py:data:`~esmvalcore.experimental.config.Session` object.
+A :py:data:`~esmvalcore.experimental.config.Session` can be initiated from the global :py:class:`~esmvalcore.experimental.config.Config`.
 
 .. code-block:: python
 
     >>> session = CFG.start_session(name='my_session')
 
-A ``Session`` is very similar to the config.
-It is also a dictionary, and copies all the keys from the ``Config``.
-At this moment, ``session`` is essentially a copy of ``CFG``:
+A :py:data:`~esmvalcore.experimental.config.Session` is very similar to the config.
+It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.experimental.config.Config`.
+At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.experimental.CFG`:
 
 .. code-block:: python
 
@@ -106,7 +106,7 @@ At this moment, ``session`` is essentially a copy of ``CFG``:
     >>> print(session == CFG)  # False
     False
 
-A ``Session`` also knows about the directories where the data will stored.
+A :py:data:`~esmvalcore.experimental.config.Session` also knows about the directories where the data will stored.
 The session name is used to prefix the directories.
 
 .. code-block:: python
@@ -122,7 +122,7 @@ The session name is used to prefix the directories.
     >>> session.plot_dir
     /home/user/my_output_dir/my_session_20201203_155821/plots
 
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from the ``Config``.
+Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.experimental.config.Config`.
 
 
 API reference

doc/api/esmvalcore.api.recipe.rst

@@ -3,12 +3,12 @@
 Recipes
 =======
 
-This section describes the :py:mod:`esmvalcore.experimental.recipe` submodule of the API.
+This section describes the :py:mod:`~esmvalcore.experimental.recipe` submodule of the API (:py:mod:`esmvalcore.experimental`).
 
 Recipe metadata
 ***************
 
-:py:class:`esmvalcore.experimental.recipe.Recipe` info is a class that holds metadata from a recipe.
+:py:class:`~esmvalcore.experimental.recipe.Recipe` is a class that holds metadata from a recipe.
 
 .. code-block:: python
 
@@ -41,14 +41,14 @@ Printing the recipe will give a nice overview of the recipe:
 Running a recipe
 ****************
 
-To run the recipe, call the :py:meth:`esmvalcore.experimental.Recipe.run` method.
+To run the recipe, call the :py:meth:`~esmvalcore.experimental.recipe.Recipe.run` method.
 
 .. code-block:: python
 
     >>> output = recipe.run()
     <log messages>
 
-By default, a new :py:class:`esmvalcore.experimental.config.Session` is automatically created, so that data are never overwritten.
+By default, a new :py:class:`~esmvalcore.experimental.config.Session` is automatically created, so that data are never overwritten.
 Data are stored in the ``esmvaltool_output`` directory specified in the config.
 Sessions can also be explicitly specified.
 
@@ -59,7 +59,52 @@ Sessions can also be explicitly specified.
     >>> output = recipe.run(session)
     <log messages>
 
-:py:meth:`esmvalcore.experimental.Recipe.run` returns an object that contains the locations of the data and figures (not implemented yet).
+:py:meth:`~esmvalcore.experimental.recipe.Recipe.run` returns an dictionary of objects that can be used to inspect
+the output of the recipe. The output is an instance of :py:class:`~esmvalcore.experimental.recipe_output.ImageFile` or
+:py:class:`~esmvalcore.experimental.recipe_output.DataFile` depending on its type.
+
+For working with recipe output, see: :ref:`api_recipe_output`.
+
+Running a single diagnostic or preprocessor task
+************************************************
+
+The python example recipe contains 5 tasks:
+
+Preprocessors:
+
+- ``timeseries/tas_amsterdam``
+- ``timeseries/script1``
+- ``map/tas``
+
+Diagnostics:
+
+- ``timeseries/tas_global``
+- ``map/script1``
+
+To run a single diagnostic or preprocessor, the name of the task can be passed
+as an argument to :py:meth:`~esmvalcore.experimental.recipe.Recipe.run`. If a diagnostic
+is passed, all ancestors will automatically be run too.
+
+.. code-block:: python
+
+    >>> output = recipe.run('map/script1')
+    >>> output
+    map/script1:
+      DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc')
+      DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc')
+      ImageFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.png')
+      ImageFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.png')
+
+It is also possible to run a single preprocessor task:
+
+.. code-block:: python
+
+    >>> output = recipe.run('map/tas')
+    >>> output
+    map/tas:
+      DataFile('CMIP5_CanESM2_Amon_historical_r1i1p1_tas_2000-2000.nc')
+      DataFile('CMIP6_BCC-ESM1_Amon_historical_r1i1p1f1_tas_2000-2000.nc')
+
 
 API reference
 *************

doc/api/esmvalcore.api.recipe_metadata.rst

@@ -0,0 +1,11 @@
+.. _api_recipe_metadata:
+
+Recipe Metadata
+===============
+
+This section describes the :py:mod:`~esmvalcore.experimental.recipe_metadata` submodule of the API (:py:mod:`esmvalcore.experimental`).
+
+API reference
+*************
+
+.. automodule:: esmvalcore.experimental.recipe_metadata