Refactor to use ixmp4 (#766)

Co-authored-by: Philip Hackstock <20710924+phackstock@users.noreply.github.com>

.github/workflows/nightly.yml

@@ -1,4 +1,4 @@
-# This workflow installs the package on Python 3.8, runs the tests and builds the docs
+# This workflow installs the package, runs the tests and builds the docs
 # For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
 
 name: nightly

.github/workflows/pytest-legacy.yml

@@ -20,12 +20,12 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v4
       with:
-        python-version: '3.8'
+        python-version: '3.10'
 
     - name: Install specific out-dated version of dependencies
       # Update the package requirements when changing minimum dependency versions
       # Please also add a section "Dependency changes" to the release notes
-      run: pip install pandas==1.2.0 numpy==1.19.0 matplotlib==3.5.0 iam-units==2020.4.21 xlrd==2.0 pint==0.13
+      run: pip install pandas==2.0.0 numpy==1.23.0 matplotlib==3.6.0 iam-units==2020.4.21 xlrd==2.0.1 pint==0.13
 
     - name: Install other dependencies and package
       run: pip install .[tests,optional_plotting,optional_io_formats,tutorials]

.github/workflows/pytest.yml

@@ -18,10 +18,7 @@ jobs:
         - ubuntu-latest
         - windows-latest
         python-version:
-        - '3.11'
         - '3.10'
-        - '3.9'
-        - '3.8'
 
       fail-fast: false
 

README.md

@@ -8,7 +8,7 @@ pyam: analysis & visualization <br /> of integrated-assessment and macro-energy
 
 <!-- replace python version by dynamic reference to pypi once Python versions are configured there -->
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![python](https://img.shields.io/badge/python-≥3.8,<3.12-blue?logo=python&logoColor=white)](https://github.com/IAMconsortium/pyam)
+[![python](https://img.shields.io/badge/python-≥3.10,<3.12-blue?logo=python&logoColor=white)](https://github.com/IAMconsortium/pyam)
 [![pytest](https://github.com/IAMconsortium/pyam/actions/workflows/pytest.yml/badge.svg)](https://github.com/IAMconsortium/pyam/actions/workflows/pytest.yml)
 [![ReadTheDocs](https://readthedocs.org/projects/pyam-iamc/badge/?version=latest)](https://pyam-iamc.readthedocs.io/en/latest/?badge=latest)
 [![codecov](https://codecov.io/gh/IAMconsortium/pyam/branch/main/graph/badge.svg)](https://codecov.io/gh/IAMconsortium/pyam)

RELEASE_NOTES.md

@@ -1,9 +1,21 @@
 # Next Release
 
 The next release must bump the major version number.
+Reactivate tests for Python 3.11 once ixmp4 0.3 is released.
+
+## Dependency changes
+
+Support for Python 3.7-3.9 was removed due to an incompatible dependency.  
+
+PR [#766](https://github.com/IAMconsortium/pyam/pull/766) added the **ixmp4** package
+for better integration with the IIASA scenario database infrastructure.
 
 ## API changes
 
+Credentials to access the IIASA scenario database infrastructure should now be managed
+using the **ixmp4** package
+(see [here](https://pyam-iamc.readthedocs.io/en/stable/api/iiasa.html)).
+
 The column *exclude* of the `meta` indicators was moved to a new attribute `exclude`.
 All validation methods are refactored such that the argument `exclude_on_fail` changes
 this new attribute (see PR [#759](https://github.com/IAMconsortium/pyam/pull/759)).
@@ -19,8 +31,10 @@ instead of `pyam.to_list()`.
 
 ## Individual updates
 
+
 - [#772](https://github.com/IAMconsortium/pyam/pull/772) Show all missing rows for `require_data()`
 - [#771](https://github.com/IAMconsortium/pyam/pull/771) Refactor to start a separate validation module
+- [#766](https://github.com/IAMconsortium/pyam/pull/766) Use **ixmp4** for credentials to access a Scenario Explorer database
 - [#764](https://github.com/IAMconsortium/pyam/pull/764) Clean-up exposing internal methods and attributes
 - [#763](https://github.com/IAMconsortium/pyam/pull/763) Implement a fix against carrying over unused levels when initializing from an indexed pandas object
 - [#759](https://github.com/IAMconsortium/pyam/pull/759) Excise "exclude" column from meta and add a own attribute

docs/api/iiasa.rst

@@ -1,15 +1,51 @@
 .. currentmodule:: pyam.iiasa
 
-The **Connection** class
-========================
+Databases hosted by IIASA
+=========================
 
-IIASA's ixmp Scenario Explorer infrastructure implements a RestAPI
-to directly query the database server connected to an explorer instance.
-See https://software.ene.iiasa.ac.at/ixmp-server for more information.
+The |pyam| package allows to directly query the scenario databases hosted by the
+IIASA Energy, Climate and Environment program (ECE), commonly known as
+the *Scenario Explorer* infrastructure. It is developed and maintained
+by the ECE `Scenario Services and Scientific Software team`_.
+
+.. _`Scenario Services and Scientific Software team` : https://software.ece.iiasa.ac.at
+
+You do not have to provide username/password credentials to connect to any public
+database instance using |pyam|. However, to connect to project-internal databases,
+you have to create an account at the IIASA-ECE *Manager Service*
+(https://manager.ece.iiasa.ac.at). Please contact the respective project coordinator
+for permission to access a project-internal database.
+
+To store the credentials on your machine so that |pyam| can use it to query a database,
+we depend on the Python package |ixmp4|. You only have to do this once
+(unless you change your password).
+
+The credentials will be valid for connecting to *Scenario Apps* based on |ixmp4|
+as well as for (legacy) *Scenario Explorer* database backends (see below).
+
+In a console, run the following:
+
+.. code-block:: console
+
+    ixmp4 login <username>
+
+You will be prompted to enter your password.
+
+.. warning::
+
+    Your username and password will be saved locally in plain-text for future use!
+
+*Scenario Apps* instances
+-------------------------
+
+Coming soon...
+
+*Scenario Explorer* instances
+-----------------------------
+
+The *Scenario Explorer* infrastructure developed by the Scenario Services and Scientific
+Software team was developed and used for projects from 2018 until 2023.
 
-The |pyam| package uses this interface to read timeseries data as well as
-categorization and quantitative indicators.
-The data is returned as an :class:`IamDataFrame`.
 See `this tutorial <../tutorials/iiasa_dbs.html>`_ for more information.
 
 .. autoclass:: Connection
@@ -20,5 +56,3 @@ See `this tutorial <../tutorials/iiasa_dbs.html>`_ for more information.
 
 .. autofunction:: lazy_read_iiasa
    :noindex:
-
-.. autofunction:: set_config

docs/conf.py

@@ -357,13 +357,18 @@
 
 .. |pyam| replace:: :class:`pyam`
 
+.. |ixmp4| raw:: html
+
+   <a href="https://docs.ece.iiasa.ac.at/ixmp4/">
+   <code class="xref py py-class docutils literal notranslate"><span class="pre">
+   ixmp4</span></code></a> 
+
 .. |br| raw:: html
 
-    <br>
+   <br>
 
 .. |datapackage.Package.docs| raw:: html
 
-   <a href="https://github.com/frictionlessdata/datapackage-py#package">read
-   the docs</a>
+   <a href="https://github.com/frictionlessdata/datapackage-py#package">read the docs</a>
 
 """

docs/index.rst

@@ -24,7 +24,7 @@ Release v\ |version|.
 .. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
    :target: https://github.com/psf/black
 
-.. |python| image:: https://img.shields.io/badge/python-≥3.8,<3.12-blue?logo=python&logoColor=white
+.. |python| image:: https://img.shields.io/badge/python-≥3.10,<3.12-blue?logo=python&logoColor=white
    :target: https://github.com/IAMconsortium/pyam
 
 .. |pytest| image:: https://github.com/IAMconsortium/pyam/actions/workflows/pytest.yml/badge.svg

docs/tutorials/iiasa_dbs.ipynb

@@ -47,12 +47,23 @@
    "metadata": {},
    "source": [
     "If you have credentials to connect to a non-public or restricted Scenario Explorer instance,\n",
-    "you can store this information by running the following command in a separate Python console:\n",
+    "you can store this information by running the following command in a console:\n",
     "\n",
     "```\n",
-    "import pyam\n",
-    "pyam.iiasa.set_config(<username>, <password>)\n",
+    "ixmp4 login <username>\n",
+    "\n",
     "```\n",
+    "\n",
+    "You will be prompted to enter your password.\n",
+    "\n",
+    "<div class=\"alert alert-warning\">\n",
+    "\n",
+    "Your username and password will be saved locally in plain-text for future use!\n",
+    "\n",
+    "</div>\n",
+    "\n",
+    "\n",
+    "\n",
     "When initializing a new **Connection** instance, **pyam** will automatically search for the configuration in a known location."
    ]
   },

pyam/iiasa.py

@@ -23,6 +23,10 @@
     is_list_like,
 )
 from pyam.logging import deprecation_warning
+import ixmp4
+from ixmp4.conf import settings
+from ixmp4.conf.auth import ManagerAuth
+
 
 logger = logging.getLogger(__name__)
 # set requests-logger to WARNING only
@@ -36,28 +40,22 @@
 """.replace(
     "\n", ""
 )
+IXMP4_LOGIN = "Please run `ixmp4 login <username>` in a console"
 
 # path to local configuration settings
 DEFAULT_IIASA_CREDS = Path("~").expanduser() / ".local" / "pyam" / "iiasa.yaml"
 
-JWT_DECODE_ARGS = {"verify_signature": False, "verify_exp": True}
-
 
 def set_config(user, password, file=None):
-    """Save username and password for the IIASA API connection to a file"""
-    file = Path(file) if file is not None else DEFAULT_IIASA_CREDS
-    if not file.parent.exists():
-        file.parent.mkdir(parents=True)
-
-    with open(file, mode="w") as f:
-        logger.info(f"Setting IIASA-connection configuration file: {file}")
-        yaml.dump(dict(username=user, password=password), f, sort_keys=False)
+    raise DeprecationWarning(f"This method is deprecated. {IXMP4_LOGIN}.")
 
 
 def _read_config(file):
     """Read username and password for IIASA API connection from file"""
     with open(file, "r") as stream:
-        return yaml.safe_load(stream)
+        creds = yaml.safe_load(stream)
+
+    return ManagerAuth(**creds, url=settings.manager_url)
 
 
 def _check_response(r, msg="Error connecting to IIASA database", error=RuntimeError):
@@ -73,74 +71,58 @@ def __init__(self, creds: str = None, auth_url: str = _AUTH_URL):
         ----------
         creds : pathlib.Path or str, optional
             Path to a file with authentication credentials
-        auth_url : str, optionl
+        auth_url : str, optional
             Url of the authentication service
         """
         self.client = httpx.Client(base_url=auth_url, timeout=10.0, http2=True)
-        self.access_token, self.refresh_token = None, None
 
         if creds is None:
             if DEFAULT_IIASA_CREDS.exists():
-                self.creds = _read_config(DEFAULT_IIASA_CREDS)
+                deprecation_warning(
+                    f"{IXMP4_LOGIN} and manually delete the file '{DEFAULT_IIASA_CREDS}'.",
+                    "Using a pyam-credentials file",
+                )
+                self.auth = _read_config(DEFAULT_IIASA_CREDS)
             else:
-                self.creds = None
+                self.auth = ixmp4.conf.settings.default_auth
         elif isinstance(creds, Path) or is_str(creds):
-            self.creds = _read_config(creds)
+            self.auth = _read_config(creds)
         else:
             raise DeprecationWarning(
                 "Passing credentials as clear-text is not allowed. "
-                "Please use `pyam.iiasa.set_config(<user>, <password>)` instead!"
+                f"{IXMP4_LOGIN} instead."
             )
 
-        # if no creds, get anonymous token
-        # TODO: explicit token for anonymous login will not be necessary for ixmp-server
-        if self.creds is None:
-            r = self.client.get("/legacy/anonym/")
-            if r.status_code >= 400:
-                raise ValueError("Unknown API error: " + r.text)
-            self.user = None
-            self.access_token = r.json()
+        # explicit token for anonymous login is not necessary for ixmp4 platforms
+        # but is required for legacy Scenario Explorer databases
+        if self.auth.user.username == "@anonymous":
+            self._get_anonymous_token()
 
-        # else get user-token
         else:
-            self.user = self.creds["username"]
-            self.obtain_jwt()
-
-    def __call__(self):
-        try:
-            # raises jwt.ExpiredSignatureError if token is expired
-            jwt.decode(self.access_token, options=JWT_DECODE_ARGS)
-
-        except jwt.ExpiredSignatureError:
-            self.refresh_jwt()
+            self.user = self.auth.user.username
+            self.access_token = self.auth.access_token
 
-        return {"Authorization": "Bearer " + self.access_token}
-
-    def obtain_jwt(self):
-        r = self.client.post("/v1/token/obtain/", json=self.creds)
-        if r.status_code == 401:
-            raise ValueError(
-                "Credentials not valid to connect to https://manager.ece.iiasa.ac.at."
-            )
-        elif r.status_code >= 400:
+    def _get_anonymous_token(self):
+        r = self.client.get("/legacy/anonym/")
+        if r.status_code >= 400:
             raise ValueError("Unknown API error: " + r.text)
+        self.user, self.access_token = None, r.json()
 
-        _json = r.json()
-        self.access_token = _json["access"]
-        self.refresh_token = _json["refresh"]
-
-    def refresh_jwt(self):
+    def __call__(self):
         try:
             # raises jwt.ExpiredSignatureError if token is expired
-            jwt.decode(self.refresh_token, options=JWT_DECODE_ARGS)
-            r = self.client.post(
-                "/v1/token/refresh/", json={"refresh": self.refresh_token}
+            jwt.decode(
+                self.access_token,
+                options={"verify_signature": False, "verify_exp": True},
             )
-            if r.status_code >= 400:
-                raise ValueError("Unknown API error: " + r.text)
-            self.access_token = r.json()["access"]
         except jwt.ExpiredSignatureError:
-            self.obtain_jwt()
+            if self.auth.user.username == "@anonymous":
+                self._get_anonymous_token()
+            else:
+                self.auth.refresh_or_reobtain_jwt()
+                self.access_token = self.auth.access_token
+
+        return {"Authorization": "Bearer " + self.access_token}
 
 
 class Connection(object):

pyam/logging.py

@@ -20,6 +20,7 @@ def adjust_log_level(logger="pyam", level="ERROR"):
 
 def deprecation_warning(msg, item="This method", stacklevel=3):
     """Write deprecation warning to log"""
+    warnings.simplefilter("always", DeprecationWarning)
     message = f"{item} is deprecated and will be removed in future versions. {msg}"
     warnings.warn(message, DeprecationWarning, stacklevel=stacklevel)
 

setup.cfg

@@ -19,25 +19,25 @@ classifiers =
 [options]
 packages = pyam
 include_package_data = True
-python_requires = >=3.8, <3.12
+python_requires = >=3.10, <3.12
 
 # NOTE TO DEVS
 # If you change a minimum version below, please explicitly implement the change
 # in our minimum-reqs test in the file ./.github/workflows/pytest-dependency.yml
 # Please also add a section "Dependency changes" to the release notes
 install_requires =
     iam-units >= 2020.4.21
-    numpy >= 1.19.0, < 1.24
+    ixmp4 >= 0.2.0
+    numpy >= 1.23.0, < 1.24
     requests
     pyjwt
     httpx[http2]
     openpyxl
-    # remove exception in test_io.py when changing requirement to pandas ≥ 1.5
-    pandas >= 1.2.0
+    pandas >= 2.0.0
     scipy
     pint >= 0.13
     PyYAML
-    matplotlib >= 3.5.0, < 3.7.1
+    matplotlib >= 3.6.0, < 3.7.1
     seaborn
     six
     setuptools >= 41