Merge pull request #103 from khaeru/enh/2023-W36

Miscellaneous enhancements for 2023-W36

.github/workflows/pytest.yaml

@@ -71,3 +71,13 @@ jobs:
 
     - name: Upload test coverage to Codecov.io
       uses: codecov/codecov-action@v3
+
+  pre-commit:
+    name: Code quality
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v4
+    - uses: pre-commit/action@v3.0.0

.gitignore

@@ -4,10 +4,13 @@ __pycache__
 build
 dist
 
-# pytest, pytest-benchmark, pytest-cov
+# Editors and development tools
 .benchmarks
 .coverage*
+.mypy_cache
 .pytest_cache
+.ruff_cache
+.vscode
 coverage.xml
 htmlcov
 prof/
@@ -17,13 +20,7 @@ prof/
 # Generated by .tests.core.test_computer.test_cache()
 myfunc2-*.pkl
 
-# mypy
-.mypy_cache
-
 # Sphinx and related generated files
 doc/*.svg
 doc/.ipynb_checkpoints
 doc/_build
-
-# Editor/IDE-specific files
-/.vscode

.pre-commit-config.yaml

@@ -0,0 +1,26 @@
+repos:
+- repo: https://github.com/pre-commit/mirrors-mypy
+  rev: v1.4.1
+  hooks:
+  - id: mypy
+    additional_dependencies:
+    - importlib_resources
+    - nbclient
+    - pint
+    - pytest
+    - sdmx1
+    - Sphinx
+    - types-PyYAML
+    - types-pytz
+    - types-python-dateutil
+    - types-setuptools
+    - xarray
+    args: []
+- repo: https://github.com/psf/black
+  rev: 23.7.0
+  hooks:
+  - id: black
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.0.287
+  hooks:
+  - id: ruff

doc/api.rst

@@ -18,7 +18,7 @@ Top-level classes and functions
 
 .. autoclass:: genno.Computer
    :members:
-   :exclude-members: add, apply, eval, graph
+   :exclude-members: add, add_queue, apply, eval, graph
 
    A Computer is used to prepare (:meth:`add` and related methods) and then execute (:meth:`get` and related methods) **computations** stored in a :attr:`graph`.
    Advanced users may manipulate the graph directly; but most computations can be prepared can be handled by using Computer methods.
@@ -106,9 +106,6 @@ Top-level classes and functions
          e.g. "select", retrievable with :meth:`get_comp`.
          :meth:`add_single` is called with :py:`(key=args[0], data, *args[1], **kwargs)`, that is, applying the named operator to the other parameters.
 
-      :class:`str` naming another Computer method
-         e.g. :meth:`add_file` → the named method is called with the `args` and `kwargs`.
-
       :class:`.Key` or other :class:`str`:
          Passed to :meth:`add_single`.
 
@@ -130,6 +127,22 @@ Top-level classes and functions
         >>> rep.get('my report')
         foo
 
+   .. automethod:: add_queue
+
+      This method allows to add many computations at once by, in effect, calling :meth:`add` repeatedly with sets of positional and (optionally) keyword arguments taken from the `queue`.
+      The argument may be:
+
+      - A prepared/static data structure, like a :class:`list`, where each item is either a 2-:class:`tuple` of :py:`(args, kwargs)` or only a tuple of :py:`args` that can be passed to :meth:`add`.
+      - A generator that yields items of the same type(s).
+
+      Given this initial sequence of items, :meth:`add_queue` will…
+
+      - Pass each item in turn to :meth:`add`;
+      - If an item fails to be added—for instance, with :class:`MissingKeyError` on one of its inputs—and `max_tries` > 1: re-append that item to the queue so that it can be attempted again;
+      - If an item fails to be added at least `max_tries` times: take an action according to `fail`.
+
+      This behaviour makes :meth:`add_queue` tolerant of entries in `queue` that are out-of-order: individual items may fail in calls to :meth:`add` on initial passes through the queue, but eventually succeed once their inputs are available.
+
    .. automethod:: apply
 
       The `generator` may have a type annotation for Computer on its first positional argument.

doc/compat-sdmx.rst

@@ -0,0 +1,24 @@
+SDMX (:mod:`.compat.sdmx`)
+**************************
+
+:doc:`Package documentation <sdmx1:index>`
+
+Note that this package is available in PyPI as ``sdmx1``.
+To install the correct package, use:
+
+.. code-block:: sh
+
+   pip install genno[sdmx]
+
+To ensure the function is available:
+
+.. code-block:: python
+
+   c = Computer()
+   c.require_compat("genno.compat.sdmx")
+   c.add(..., "codelist_to_groups", ...)
+
+.. currentmodule:: genno.compat.sdmx
+
+.. automodule:: genno.compat.sdmx
+   :members:

doc/conf.py

@@ -97,6 +97,7 @@ def can_document_member(cls, member, membername, isattr, parent) -> bool:
     "plotnine": ("https://plotnine.readthedocs.io/en/stable/", None),
     "pyam": ("https://pyam-iamc.readthedocs.io/en/stable/", None),
     "python": ("https://docs.python.org/3/", None),
+    "sdmx1": ("https://sdmx1.readthedocs.io/en/stable", None),
     "xarray": ("https://docs.xarray.dev/en/stable/", None),
 }
 

doc/index.rst

@@ -24,6 +24,7 @@ Compatibility
 
 - :doc:`Plotnine <compat-plotnine>` (:mod:`.plotnine`), via :mod:`.compat.plotnine`.
 - :doc:`Pyam <compat-pyam>` (:mod:`.pyam`), via :mod:`.compat.pyam`.
+- :doc:`SDMX <compat-sdmx>` (:mod:`.sdmx`), via :mod:`.compat.sdmx`.
 
 .. toctree::
    :maxdepth: 1
@@ -32,6 +33,7 @@ Compatibility
 
    compat-plotnine
    compat-pyam
+   compat-sdmx
 
 Packages that extend :mod:`genno` include:
 

doc/whatsnew.rst

@@ -1,8 +1,18 @@
 What's new
 **********
 
-.. Next release
-.. ============
+Next release
+============
+
+- Allow use of regular expressions in :func:`~.computations.aggregate` (:issue:`35`, :pull:`103`).
+- Improve documentation of :meth:`.add_queue` (:issue:`18`, :pull:`103`).
+- New compatibility module for :doc:`compat-sdmx` (:pull:`103`).
+- Ensure consistent :py:`drop=True` behaviour of :func:`.select` and :meth:`.AttrSeries.sel` with :meth:`.SparseDataArray.sel` (:pull:`103`).
+  See the docstring for details.
+- Creating a 1-D :class:`.AttrSeries` with an "anonymous" (un-named) dimension or index results in a dimension name :py:`"dim_0"`, consistent with :class:`xarray.DataArray` (:pull:`103`).
+- Don't allow exceptions to raise from :meth:`.Graph.__contains__`; return :obj:`False` (:pull:`103`).
+- Reduce verbosity of logging from :func:`.collect_units` (:pull:`103`).
+- Provide typed signatures for :meth:`.Quantity.shape` and :attr:`~.Quantity.size` for the benefit of downstream applications (:pull:`103`).
 
 v1.18.1 (2023-08-31)
 ====================

genno/compat/sdmx.py

@@ -0,0 +1,38 @@
+from typing import TYPE_CHECKING, Iterable, List, Mapping, Optional, Union
+
+if TYPE_CHECKING:
+    from sdmx.model.common import Code, Codelist
+
+
+def codelist_to_groups(
+    codes: Union["Codelist", Iterable["Code"]], dim: Optional[str] = None
+) -> Mapping[str, Mapping[str, List[str]]]:
+    """Convert `codes` into a mapping from parent items to their children.
+
+    The returned value is suitable for use with :func:`~.computations.aggregate`.
+
+    Parameters
+    ----------
+    codes
+        Either a :class:`sdmx.Codelist <sdmx.model.common.Codelist>` object or any
+        iterable of :class:`sdmx.Code <sdmx.model.common.Code>`.
+    dim : str, optional
+        Dimension to aggregate. If `codes` is a code list and `dim` is not given, the
+        ID of the code list is used; otherwise `dim` must be supplied.
+    """
+    from sdmx.model.common import Codelist
+
+    if isinstance(codes, Codelist):
+        items: Iterable["Code"] = codes.items.values()
+        dim = dim or codes.id
+    else:
+        items = codes
+
+    if dim is None:
+        raise ValueError("Must provide a dimension ID for aggregation")
+
+    groups = dict()
+    for code in filter(lambda c: len(c.child), items):
+        groups[code.id] = list(map(str, code.child))
+
+    return {dim: groups}

genno/compat/xarray.py

@@ -18,11 +18,17 @@
 import pandas as pd
 import xarray
 from xarray.core.types import InterpOptions
+from xarray.core.utils import is_scalar
 
 from genno.core.types import Dims
 
 T = TypeVar("T", covariant=True)
 
+__all__ = [
+    "DataArrayLike",
+    "is_scalar",
+]
+
 
 class DataArrayLike(Generic[T]):
     """Class with :class:`.xarray.DataArray` -like API.
@@ -82,6 +88,14 @@ def coords(
     def dims(self) -> Tuple[Hashable, ...]:
         return NotImplemented
 
+    @property
+    def shape(self) -> Tuple[int, ...]:
+        return NotImplemented
+
+    @property
+    def size(self) -> int:
+        return NotImplemented
+
     def assign_coords(
         self,
         coords: Optional[Mapping[Any, Any]] = None,