init sankey

[mabudz]

Create sankey diagram and add tutorial.

The approach here creates the sankey diagram by using the sankey plot function of the pyam package. This function requires a mapping dictionary. The mapping dict is created by using the newly created automatic report message::sankey and a utility function sankey_mapper().

This pull request is related to the draft pull request in message_data , which can be removed, since the sankey diagram has been implemented in the pyam package.

How to review

General approach should be reviewed. Also, I am not sure if the util function sankey_mapper() is at the right place.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update release notes.

[CLAassistant]

All committers have signed the CLA.

[glatterf42]

Thanks for this PR :)
Getting the tests to pass is a bit tricky since the API for pyam changed when they went from version 1.9.0 to 2.0.0 (as you might expect) and our code needs to satisfy both versions since we test on Python <= 3.9, which is not supported for pyam 2.0.0. So let me take a closer look, maybe we'd want to use the function that pyam's sankey function wraps around directly, after all.

[glatterf42]

Thanks @daymontas1 for stepping up here, much appreciated :)

You should be able to test the PR as is and see how you like it's usefulness, but there are a few things that I would like to see happen before we merge it:

• Please rebase on main to keep it up to date with current developments.
• Please migrate the main functionality from util/__init__.py to somewhere more appropriate. E.g. report/sankey.py if it's a reporting tool or util/sankey.py if there are more general use cases.
• Please check if we can use plotly directly instead of pyam's wrapper functionality. This should allow us to utilize more design options and avoids some dependency constraints (see above). See some examples and their docs.
• Please confirm that we want to integrate the sankey functionality in the reporting as it currently stands. I'd probably ask at least @khaeru for his opinion.
• Please write a test if possible to confirm that sankey_mapper maps the correct values to the correct keys. We might also need to adapt one or two reporting tests that check the number of reporting steps if this stays in the default reporting workflow.

Please feel free to ask about any of these steps if you need help :)

[gidden]

Hi all, to simplify here, I would simply limit this functionality to pyam > 2 - we can do a version check for this on the fly and raise an error.

[gidden]

I would get rid of this, and simply do the concatenation outside of reporter (e.g., in the jupyter notebook here)

[glatterf42]

I kept this for now as I don't know what the Reporter actually does and it seems useful to just get the dataframe format we need from the Reporter. But please elaborate :)

[daymontas1]

I have created a new file, sankey.py, within the report directory, which extracts Sankey diagram functionality from init.py. Fridolin (@glatterf42), could you please confirm if this is what we want? Additionally, I have updated the westeros_sankey.ipynb file to ensure compatibility with the new sankey.py file in the report directory. I have annotated these changes in the westeros_sankey.ipynb below. Also, in this case, the following line inside the init.py file must be either removed or commented out:
("message::sankey", "concat", "out::pyam", "in::pyam")

[glatterf42]

Thanks, @daymontas1, these changes sound good to me. For the line that has become redundant, please remove it rather than commenting it out, this keeps the code clean.
Could you please push your changes to the branch so that we can take a look and see how the tests are doing?
If you don't have write access to @mabudz's fork, please let us know so that we can figure out a solution :)

[glatterf42]

Rebased onto current main and expanded the PR a bit :)
Thanks for jumping in here, @gidden :)
@daymontas1, please pull these latest changes before making further edits. If you already have some edits locally, you can git stash them, git pull these changes and git stash pop your work on top of this.

The missing type hints (from the failing code quality check) probably indicate that we need to add pyam.* to our mypy config exclude list.

[glatterf42]

The other main error I'm seeing just indicates that by adding this reporter step, we need to adapt the expected value of reporter steps in another test, which is no problem :)

[daymontas1]

#''Sankey''
import logging
from functools import partial
from typing import Tuple, Mapping, List

from genno.operator import broadcast_map
from ixmp.report import Key
from .pyam import collapse_message_cols

# Assuming TASKS1 was where Sankey tasks were defined:
TASKS1 = (
    ("message::sankey", "concat", "out::pyam", "in::pyam"),
)

def get_sankey_tasks() -> List[Tuple[Tuple, Mapping]]:
    """Return a list of tasks for Sankey diagram reporting."""
    to_add: List[Tuple[Tuple, Mapping]] = []
    strict = dict(strict=True)

    # This might include specific Sankey diagram configuration or additional tasks.
    for t in TASKS1:
        to_add.append((t, strict))

    return to_add

class SankeyReporter:
    """A specialized reporter for generating Sankey diagrams."""

    @staticmethod
    def add_tasks(reporter, fail_action: str = "raise") -> None:
        """Add Sankey-related tasks to a given reporter."""
        reporter.add_queue(get_sankey_tasks(), fail=fail_action)

```suggestion
    ("message::sankey", "concat", "out::pyam", "in::pyam"),
```

[glatterf42]

Thanks for this suggestion, @daymontas1 :)
To clarify here: are these the contents of the new sankey.py file you mentioned above? They probably don't need to include the suggestion part, then.
If I understand correctly, what you want to suggest is deleting this line that is no longer needed here:

Suggested change

("message::sankey", "concat", "out::pyam", "in::pyam"),