Merge branch 'master' into feature/aggregate_time

IAMconsortium · May 11, 2020 · 082078a · 082078a
2 parents 30e0023 + 8e1393e
commit 082078a
Show file tree

Hide file tree

Showing 17 changed files with 340 additions and 106 deletions.
diff --git a/.gitmodules b/.gitmodules
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
@@ -1,12 +1,17 @@
 # Next Release
 
+## Individual Updates
+- [#373](https://github.com/IAMconsortium/pyam/pull/373) Extends the error message when initializing with duplicate rows.
+- [#370](https://github.com/IAMconsortium/pyam/pull/370) Allowed filter to work with np.int64 years and np.datetime64 dates.  
+- [#369](https://github.com/IAMconsortium/pyam/pull/369) `convert_unit()` supports GWP conversion of same GHG species without context, lower-case aliases for species symbols.
+- [#361](https://github.com/IAMconsortium/pyam/pull/361) iam-units refactored from a Git submodule to a Python dependency of pyam.
 - [#322](https://github.com/IAMconsortium/pyam/pull/322) Add feature to aggregate timeseries at sub-annual time resolution #322
 
 # Release v0.5.0
 
 ## Highlights
 
-- Improved feature for unit conversion 
+- Improved feature for unit conversion
   using the [pint package](https://pint.readthedocs.io) and
   the [IAMconsortium/units](https://github.com/IAMconsortium/units) repository,
   providing out-of-the-box conversion of unit definitions commonly used
@@ -15,7 +20,7 @@
   for more information
 - Increased support for operations on timeseries data with continuous-time
   resolution
-- New tutorial for working with various input data formats; 
+- New tutorial for working with various input data formats;
   [take a look](https://pyam-iamc.readthedocs.io/en/stable/tutorials/data_table_formats.html)
 - Rewrite and extension of the documentation pages for the API;
   [read the new docs](https://pyam-iamc.readthedocs.io/en/stable/api.html)!
@@ -28,7 +33,7 @@ the API of `IamDataFrame.convert_unit()` from a dictionary to explicit kwargs
 
 PR [#334](https://github.com/IAMconsortium/pyam/pull/334) changed the arguments
 of `IamDataFrame.interpolate()` and `pyam.fill_series()` to `time`. It can still
-be an integer (i.e., a year). 
+be an integer (i.e., a year).
 
 With PR [#337](https://github.com/IAMconsortium/pyam/pull/337), initializing
 an IamDataFrame with `n/a` entries in columns other than `value` raises an error.
@@ -72,7 +77,7 @@ behaviour of `aggregate_region()` regarding the treatment of components at the
 region-level. To keep the previous behaviour, add `components=True`.
 
 PR [#315](https://github.com/IAMconsortium/pyam/pull/314) changed the return
-type of `aggregate[_region]()` to an `IamDataFrame` instance. 
+type of `aggregate[_region]()` to an `IamDataFrame` instance.
 To keep the previous behaviour, add `timeseries()`.
 The object returned by `[check_]aggregate[_region]()` now includes both the
 actual and the expected value as a `pd.DataFrame` instance.
@@ -89,7 +94,7 @@ rather than a dictionary and also includes optional treatment of components
 - [#305](https://github.com/IAMconsortium/pyam/pull/305) Add `method` and `weight` options to the (region) aggregation functions
 - [#302](https://github.com/IAMconsortium/pyam/pull/302) Rework the tutorials
 - [#301](https://github.com/IAMconsortium/pyam/pull/301) Bugfix when using `to_excel()` with a `pd.ExcelWriter`
-- [#297](https://github.com/IAMconsortium/pyam/pull/297) Add `empty` attribute, better error for `timeseries()` on empty dataframe 
+- [#297](https://github.com/IAMconsortium/pyam/pull/297) Add `empty` attribute, better error for `timeseries()` on empty dataframe
 - [#295](https://github.com/IAMconsortium/pyam/pull/295) Include `meta` table when writing to or reading from `xlsx` files
 - [#292](https://github.com/IAMconsortium/pyam/pull/292) Add warning message if `data` is empty at initialization (after formatting)
 - [#288](https://github.com/IAMconsortium/pyam/pull/288) Put `pyam` logger in its own namespace (see [here](https://docs.python-guide.org/writing/logging/#logging-in-a-library>))

diff --git a/appveyor.yml b/appveyor.yml
@@ -9,10 +9,6 @@ init:
   - "ECHO %PYTHON_VERSION% %MINICONDA%"
 
 install:
-  # initialize/update submodule(s) - this seems to be not necessary on travis
-  - git submodule update --init
-  - git submodule status
-
   # these correspond to folder naming of miniconda installs on appveyor.  See
   # https://www.appveyor.com/docs/installed-software#python
   - if "%PYTHON_VERSION%" == "3.6" set "BASE_PYTHON_VERSION=36"

diff --git a/doc/source/api/io.rst b/doc/source/api/io.rst
@@ -8,10 +8,8 @@ DataFrames and xlsx/csv files
 
 A :class:`pandas.DataFrame` or a path to an :code:`xlsx` or :code:`csv`
 with data in the required structure (i.e., index/columns) can be imported
-directly by initializing an :class:`IamDataFrame` - 
-see this tutorial_ for more information.
-
-.. _tutorial: ../tutorials/data_table_formats.html
+directly by initializing an :class:`IamDataFrame` - see
+`this tutorial <../tutorials/data_table_formats.html>`_ for more information.
 
 Exporting to these formats is implemented via the following functions:
 
@@ -30,3 +28,14 @@ The |pyam| package supports reading and writing to the
 .. autofunction:: read_datapackage
 
 .. automethod:: IamDataFrame.to_datapackage
+
+Connecting to an IIASA scenario explorer instance
+-------------------------------------------------
+
+IIASA's ixmp scenario explorer infrastructure includes a RestAPI
+to query data directly from the database server.
+The |pyam| package uses this interface to read resources
+and initialize an :class:`IamDataFrame`.
+See `this tutorial ../tutorials/iiasa_dbs.html>`_ for more information.
+
+.. autofunction:: read_iiasa
diff --git a/doc/source/conf.py b/doc/source/conf.py
@@ -37,10 +37,8 @@
     'sphinx.ext.intersphinx',
     'sphinx.ext.todo',
     'sphinx.ext.coverage',
-    'sphinx.ext.mathjax',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
-    'sphinx.ext.mathbase',
     'sphinxcontrib.bibtex',
     'sphinxcontrib.programoutput',
     'cloud_sptheme.ext.table_styling',

diff --git a/doc/source/index.rst b/doc/source/index.rst
@@ -34,8 +34,8 @@ Release v\ |version|.
 .. |coveralls| image:: https://coveralls.io/repos/github/IAMconsortium/pyam/badge.svg?branch=master
    :target: https://coveralls.io/github/IAMconsortium/pyam?branch=master
 
-.. |joss| image:: http://joss.theoj.org/papers/356bc013105642ec4e94a3b951836cfe/status.svg
-   :target: http://joss.theoj.org/papers/356bc013105642ec4e94a3b951836cfe
+.. |joss| image:: https://joss.theoj.org/papers/10.21105/joss.01095/status.svg
+   :target: https://joss.theoj.org/papers/10.21105/joss.01095
 
 .. |doi| image:: https://zenodo.org/badge/113359260.svg
    :target: https://zenodo.org/badge/latestdoi/113359260

diff --git a/doc/source/tutorials.rst b/doc/source/tutorials.rst
@@ -20,5 +20,6 @@ The source code of the tutorials is available in the folder
    tutorials/subannual_time_resolution.ipynb
    tutorials/ipcc_colors.ipynb
    tutorials/iiasa_dbs.ipynb
+   tutorials/legends.ipynb
    tutorials/aggregating_variables_and_plotting_with_negative_values.ipynb
    tutorials/pyam_logo.ipynb
diff --git a/doc/source/tutorials/legends.ipynb b/doc/source/tutorials/legends.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Customizing Legends\n",
+    "# Customizing legends\n",
     "\n",
     "This is a short tutorial showing how different arguments to the `legend` keyword in the **pyam** plotting library affects where the legend is located.\n",
     "\n",
@@ -17,8 +17,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import pyam\n",
-    "\n",
+    "import pyam"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "df = (\n",
     "    pyam.IamDataFrame(data='tutorial_data.csv')\n",
     "    .filter(variable='Emissions|CO2', region='World')\n",

diff --git a/pyam/core.py b/pyam/core.py
@@ -767,41 +767,78 @@ def rename(self, mapping=None, inplace=False, append=False,
 
     def convert_unit(self, current, to=None, factor=None, registry=None,
                      context=None, inplace=False):
-        """Converts a unit using a given factor or the pint package
+        r"""Convert all data having *current* units to new units.
 
-        The `pint package <https://pint.readthedocs.io>`_ natively handles
-        conversion of standard (SI) units (e.g., exajoule to terawatt-hours,
-        :code:`EJ -> TWh`). It can also parse combined units (e.g.,
-        exajoule per year, :code:`EJ/yr`).
+        If *factor* is given, existing values are multiplied by it, and the
+        *to* units are assigned to the 'unit' column.
 
-        The :class:`pint.UnitRegistry` used by default loads additional
-        unit definitions relevant for integrated assessment models and energy
-        systems analysis from the `IAMconsortium/units
-        <https://github.com/IAMconsortium/units>`_ repository.
-        You can access that unit registry
-        via :func:`pint.get_application_registry`.
+        Otherwise, the :mod:`pint` package is used to convert from *current* ->
+        *to* units without an explicit conversion factor. Pint natively handles
+        conversion between any standard (SI) units that have compatible
+        dimensionality, such as exajoule to terawatt-hours, :code:`EJ -> TWh`,
+        or tonne per year to gram per second, :code:`t / yr -> g / sec`.
+
+        The default *registry* includes additional unit definitions relevant
+        for integrated assessment models and energy systems analysis, via the
+        `iam-units <https://github.com/IAMconsortium/units>`_ package.
+        This registry can also be accessed directly, using::
+
+            from iam_units import registry
+
+        When using this registry, *current* and *to* may contain the symbols of
+        greenhouse gas (GHG) species, such as 'CO2e', 'C', 'CH4', 'N2O',
+        'HFC236fa', etc., as well as lower-case aliases like 'co2' supported by
+        :mod:`pyam`. In this case, *context* must contain 'gwp\_' followed by
+        the name of a specific global warming potential (GWP) metric supported
+        by :mod:`iam_units`, e.g. 'gwp_AR5GWP100'.
+
+        Rows with units other than *current* are not altered.
 
         Parameters
         ----------
         current : str (or mapping, deprecated)
-            name of current unit (to be converted from)
+            Current units to be converted.
         to : str
-            name of new unit (to be converted to)
+            New unit (to be converted to) or symbol for target GHG species. If
+            only the GHG species is provided, the units (e.g. :code:`Mt /
+            year`) will be the same as `current`, and an expression combining
+            units and species (e.g. 'Mt CO2e / yr') will be placed in the
+            'unit' column.
         factor : value, optional
-            conversion factor if given, otherwise defaults to the application
-            UnitRegistry
+            Explicit factor for conversion without `pint`.
         registry : pint.UnitRegistry, optional
-            use a specific :class:´pint.UnitRegistry`; if `None`, use default
-            application registry with definitions imported from the
-            `IAMconsortium/units <https://github.com/IAMconsortium/units>`_
-            repository
-        context : str, optional
-            passed to the UnitRegistry
-        inplace : bool, default False
-            if True, do operation inplace and return None
+            Specific unit registry to use for conversion. Default: the
+            `iam-units <https://github.com/IAMconsortium/units>`_ registry.
+        context : str or pint.Context, optional
+            (Name of) a :ref:`pint context <pint:context>` to use in
+            conversion. Required when converting between GHG species using GWP
+            metrics, unless the species indicated by *current* and *to* are the
+            same.
+        inplace : bool, optional
+            Whether to return a new IamDataFrame.
+
+        Returns
+        -------
+        IamDataFrame
+            If *inplace* is :obj:`False`.
+        None
+            If *inplace* is :obj:`True`.
+
+        Raises
+        ------
+        pint.UndefinedUnitError
+            if attempting a GWP conversion but *context* is not given.
+        pint.DimensionalityError
+            without *factor*, when *current* and *to* are not compatible units.
         """
         # TODO: deprecate using `dict` in next release (>=0.6.0)
         # TODO: make `to` required
+
+        # Handle user input
+        # Check that (only) either factor or registry/context is provided
+        if factor and any([registry, context]):
+            raise ValueError('use either `factor` or `pint.UnitRegistry`')
+
         if isinstance(current, dict) and to is None and factor is None:
             deprecation_warning('Use explicit keyword arguments instead!',
                                 type='Using a dictionary to convert units')
@@ -1184,10 +1221,10 @@ def filter(self, keep=True, inplace=False, **kwargs):
                string or list of strings, where `*` can be used as a wildcard
              - 'level': the maximum "depth" of IAM variables (number of '|')
                (excluding the strings given in the 'variable' argument)
-             - 'year': takes an integer, a list of integers or a range
-               note that the last year of a range is not included,
+             - 'year': takes an integer (int/np.int64), a list of integers or
+                a range. Note that the last year of a range is not included,
                so `range(2010, 2015)` is interpreted as `[2010, ..., 2014]`
-             - arguments for filtering by `datetime.datetime`
+             - arguments for filtering by `datetime.datetime` or np.datetime64
                ('month', 'hour', 'time')
              - 'regexp=True' disables pseudo-regexp syntax in `pattern_match()`
         """

diff --git a/pyam/iiasa.py b/pyam/iiasa.py
@@ -330,15 +330,16 @@ def _match(data, patterns):
         # pass empty list to API if all regions selected
         if len(regions) == len(self.regions()):
             regions = []
-
+        logger.debug(f"Prepared filter for {len(regions)} region(s), "
+                     f"{len(variables)} variables and {len(runs)} runs")
         data = {
             "filters": {
                 "regions": list(regions),
                 "variables": list(variables),
                 "runs": list(runs),
                 "years": [],
                 "units": [],
-                "times": []
+                "timeslices": []
             }
         }
         return data