[CLI] flexmeasures show beliefs entity path in case of duplicated s…

…ensors (#1026)

* add sensor aliases

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* Add `get_path` method to the classes Account, Asset and Sensor to get their entity path.

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* Add docstring and simplify code

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* Add changelog entry.

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* Fix/harmonize options to show IDs or path (#1031)

* docs: clarify flag option

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: improve title

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: harmonize how two CLI options work together (show path vs. show ID)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: harmonize how paths are shown in case of having only a part of the sensor names be duplicated

Signed-off-by: F.N. Claessen <felix@seita.nl>

* refactor: repurpose util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* refactor: simplify logic

Signed-off-by: F.N. Claessen <felix@seita.nl>

* docs: fix typos

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>

* improve docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

---------

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>

documentation/changelog.rst

@@ -9,6 +9,9 @@ v0.22.0 | June XX, 2024
 New features
 -------------   
 
+* Add `asset/<id>/status` page to view asset statuses [see `PR #41 <https://github.com/FlexMeasures/flexmeasures/pull/941/>`_ and `PR #1035 <https://github.com/FlexMeasures/flexmeasures/pull/1035/>`_]
+* Support `start_date` and `end_date` query parameters for the asset page [see `PR #1030 <https://github.com/FlexMeasures/flexmeasures/pull/1030/>`_]
+
 Bugfixes
 -----------
 
@@ -19,24 +22,16 @@ Infrastructure / Support
  main
 * Add option `droplevels` to the `PandasReporter` to drop all the levels except the `event_start` and `event_value` [see `PR #1043 <https://github.com/FlexMeasures/flexmeasures/pull/1043/>`_]
 * `PandasReporter` accepts the parameter `use_latest_version_only` to filter input data [see `PR #1045 <https://github.com/FlexMeasures/flexmeasures/pull/1045/>`_]
-
-
-
-v0.21.1 | May XX, 2024
-============================
-
-Bugfixes
------------
+* `flexmeasures show beliefs` uses the entity path (`<Account>/../<Sensor>`) in case of duplicated sensors [see `PR #1026 <https://github.com/FlexMeasures/flexmeasures/pull/1026/>`_]
 
 
 
 v0.21.0 | May 16, 2024
-=======
+=========================
 * Allow installing dependencies in docker-compose worker [see `PR #1057 <https://github.com/FlexMeasures/flexmeasures/pull/1057/>`_]
 
 
 v0.21.0 | April 16, 2024
->>>>>>> 23f9bd308f37df38237c8becaf9e12cde090a576
 ============================
 
 .. note:: Read more on these features on `the FlexMeasures blog <https://flexmeasures.io/021-service-better-status-and-audit/>`_.
@@ -48,9 +43,13 @@ New features
 
 * Add `asset/<id>/status` page to view asset statuses [see `PR #41 <https://github.com/FlexMeasures/flexmeasures/pull/941/>`_ and `PR #1035 <https://github.com/FlexMeasures/flexmeasures/pull/1035/>`_]
 * Add `account/<id>/auditlog` and `user/<id>/auditlog` to view user and account related actions [see `PR #1042 <https://github.com/FlexMeasures/flexmeasures/pull/1042>`_]
+<<<<<<< HEAD
 * Support `start_date` and `end_date` query parameters for the asset page [see `PR #1030 <https://github.com/FlexMeasures/flexmeasures/pull/1030/>`_]
 * In plots, add the asset name to the title of the tooltip to improve the identification of the lines [see `PR #1054 <https://github.com/FlexMeasures/flexmeasures/pull/1054/>`_]
 * On asset page, show sensor IDs in sensor table [see `PR #1053 <https://github.com/FlexMeasures/flexmeasures/pull/1053/>`_]
+=======
+* In `flexmeasures show beliefs`, the entity path (`<Account name>/<Asset 1>/.../<Asset>`) is used to differentiate between duplicated sensors names [see `PR #1026 <https://github.com/FlexMeasures/flexmeasures/pull/1026/>`_]
+>>>>>>> origin/cli/improve-duplicate-sensor-names
 
 Bugfixes
 -----------

flexmeasures/cli/data_show.py

@@ -33,7 +33,12 @@
 from flexmeasures.utils.time_utils import determine_minimum_resampling_resolution
 from flexmeasures.cli.utils import MsgStyle, validate_unique
 from flexmeasures.utils.coding_utils import delete_key_recursive
-from flexmeasures.cli.utils import DeprecatedOptionsCommand, DeprecatedOption
+from flexmeasures.utils.flexmeasures_inflection import join_words_into_a_list
+from flexmeasures.cli.utils import (
+    DeprecatedOptionsCommand,
+    DeprecatedOption,
+    get_sensor_aliases,
+)
 
 
 @click.group("show")
@@ -551,6 +556,15 @@ def chart(
     help="Include sensor IDs in the plot's legend labels and the file's column headers. "
     "NB non-unique sensor names will always show an ID.",
 )
+@click.option(
+    "--reduced-paths/--full-paths",
+    "reduce_paths",
+    default=True,
+    type=bool,
+    help="Whether to include the full path to the asset that the sensor belongs to"
+    "which shows any parent assets and their account, "
+    "or a reduced version of the path, which shows as much detail as is needed to distinguish the sensors.",
+)
 def plot_beliefs(
     sensors: list[Sensor],
     start: datetime,
@@ -562,6 +576,7 @@ def plot_beliefs(
     filepath: str | None,
     source_types: list[str] = None,
     include_ids: bool = False,
+    reduce_paths: bool = True,
 ):
     """
     Show a simple plot of belief data directly in the terminal, and optionally, save the data to a CSV file.
@@ -608,21 +623,24 @@ def plot_beliefs(
             **MsgStyle.WARN,
         )
 
-    # Decide whether to include sensor IDs (always include them in case of non-unique sensor names)
+    # Decide whether to include sensor IDs
     if include_ids:
         df.columns = [f"{s.name} (ID {s.id})" for s in sensors]
     else:
+        # In case of non-unique sensor names, show more of the sensor's ancestry
         duplicates = find_duplicates(sensors, "name")
         if duplicates:
-            df.columns = [
-                f"{s.name} (ID {s.id})" if s.name in duplicates else s for s in sensors
-            ]
-            click.secho(
-                f"The following sensor names are duplicated: {duplicates}. To distinguish them, their plot labels will include their IDs. To include IDs for all sensors, use the --include-ids flag.",
-                **MsgStyle.WARN,
+            message = "The following sensor name"
+            message += "s are " if len(duplicates) > 1 else " is "
+            message += (
+                f"duplicated: {join_words_into_a_list(duplicates)}. "
+                f"To distinguish the sensors, their plot labels will include more parent assets and their account, as needed. "
+                f"To show the full path for each sensor, use the --full-path flag. "
+                f"Or to uniquely label them by their ID instead, use the --include-ids flag."
             )
-        else:
-            df.columns = [s.name for s in sensors]
+            click.secho(message, **MsgStyle.WARN)
+        sensor_aliases = get_sensor_aliases(sensors, reduce_paths=reduce_paths)
+        df.columns = [sensor_aliases.get(s.id, s.name) for s in sensors]
 
     # Convert to the requested or default timezone
     if timezone is not None:
@@ -633,7 +651,7 @@ def plot_beliefs(
     if len(sensors) == 1:
         title = f"Beliefs for Sensor '{sensors[0].name}' (ID {sensors[0].id}).\n"
     else:
-        title = f"Beliefs for Sensor(s) [{', '.join([s.name for s in sensors])}], (ID(s): [{', '.join([str(s.id) for s in sensors])}]).\n"
+        title = f"Beliefs for Sensors {join_words_into_a_list([s.name + ' (ID ' + str(s.id) + ')' for s in sensors])}.\n"
     title += f"Data spans {naturaldelta(duration)} and starts at {start}."
     if belief_time_before:
         title += f"\nOnly beliefs made before: {belief_time_before}."

flexmeasures/cli/tests/conftest.py

@@ -214,38 +214,49 @@ def storage_schedule_sensors(
 
 @pytest.fixture(scope="module")
 def add_asset_with_children(db, setup_accounts):
-    account_id = setup_accounts["Supplier"].id
+    assets_dict = {}
     parent_type = GenericAssetType(
         name="parent",
     )
     child_type = GenericAssetType(name="child")
 
-    db.session.add_all([parent_type, child_type])
+    for account_name in ["Supplier", "Dummy"]:
+        account_id = setup_accounts[account_name].id
 
-    parent = GenericAsset(
-        name="parent",
-        generic_asset_type=parent_type,
-        account_id=account_id,
-    )
-    db.session.add(parent)
-    db.session.flush()  # assign parent asset id
-
-    assets = [
-        GenericAsset(
-            name=f"child_{i}",
-            generic_asset_type=child_type,
-            parent_asset_id=parent.id,
+        db.session.add_all([parent_type, child_type])
+
+        parent = GenericAsset(
+            name="parent",
+            generic_asset_type=parent_type,
             account_id=account_id,
         )
-        for i in range(1, 3)
-    ]
+        db.session.add(parent)
+        db.session.flush()  # assign parent asset id
+
+        assets = [
+            GenericAsset(
+                name=f"child_{i}",
+                generic_asset_type=child_type,
+                parent_asset_id=parent.id,
+                account_id=account_id,
+            )
+            for i in range(1, 3)
+        ]
+
+        db.session.add_all(assets)
+        db.session.flush()  # assign children asset ids
+
+        assets.append(parent)
 
-    db.session.add_all(assets)
-    db.session.flush()  # assign children asset ids
+        # add a sensor with the same name to the parent and children
+        for asset in assets:
+            sensor = Sensor(name="power", generic_asset=asset)
+            db.session.add(sensor)
 
-    assets.append(parent)
+        db.session.flush()
+        assets_dict[account_name] = {a.name: a for a in assets}
 
-    return {a.name: a for a in assets}
+    return assets_dict
 
 
 @pytest.fixture(scope="module")

flexmeasures/cli/tests/test_data_edit.py

@@ -125,7 +125,7 @@ def test_transfer_ownership(app, db, add_asset_with_children, add_alternative_ac
 
     from flexmeasures.cli.data_edit import transfer_ownership
 
-    parent = add_asset_with_children["parent"]
+    parent = add_asset_with_children["Supplier"]["parent"]
     old_account = parent.owner
     new_account = add_alternative_account
 

flexmeasures/cli/tests/test_utils.py

@@ -69,3 +69,45 @@ def test_get_timerange_from_flag(monkeypatch, now, flag, expected_start, expecte
 
     assert start == expected_start
     assert end == expected_end
+
+
+def test_get_unique_sensor_names(app, db, add_asset_with_children):
+    from flexmeasures.cli.utils import get_sensor_aliases
+    from flexmeasures.cli.data_show import find_duplicates
+
+    sensors = []
+    for assets in add_asset_with_children.values():
+        for asset in assets.values():
+            sensors.extend(asset.sensors)
+
+    duplicates = find_duplicates(sensors, "name")
+    aliases = get_sensor_aliases(sensors, duplicates)
+    expected_aliases = [
+        "power (Test Supplier Account/parent/child_1)",
+        "power (Test Supplier Account/parent/child_2)",
+        "power (Test Supplier Account/parent)",
+        "power (Test Dummy Account/parent/child_1)",
+        "power (Test Dummy Account/parent/child_2)",
+        "power (Test Dummy Account/parent)",
+    ]
+
+    assert list(aliases.values()) == expected_aliases
+
+    duplicates = find_duplicates(sensors, "name")
+    aliases = get_sensor_aliases(sensors[:2], duplicates)
+    expected_aliases = [
+        "power (child_1)",
+        "power (child_2)",
+    ]
+
+    assert list(aliases.values()) == expected_aliases
+
+    duplicates = find_duplicates(sensors, "name")
+    aliases = get_sensor_aliases(sensors[:3], duplicates)
+    expected_aliases = [
+        "power (parent/child_1)",
+        "power (parent/child_2)",
+        "power (parent)",
+    ]
+
+    assert list(aliases.values()) == expected_aliases

flexmeasures/cli/utils.py

@@ -12,6 +12,7 @@
 from click_default_group import DefaultGroup
 
 from flexmeasures.utils.time_utils import get_most_recent_hour, get_timezone
+from flexmeasures import Sensor
 
 
 class MsgStyle(object):
@@ -216,3 +217,84 @@ def abort(message: str):
 
 def done(message: str):
     click.secho(message, **MsgStyle.SUCCESS)
+
+
+def path_to_str(path: list, separator: str = ">") -> str:
+    """
+    Converts a list representing a path to a string format, using a specified separator.
+    """
+
+    return separator.join(path)
+
+
+def are_all_equal(paths: list[list[str]]) -> bool:
+    """
+    Checks if all given entity paths represent the same path.
+    """
+    return len(set(path_to_str(p) for p in paths)) == 1
+
+
+def reduce_entity_paths(asset_paths: list[list[str]]) -> list[list[str]]:
+    """
+    Simplifies a list of entity paths by trimming their common ancestor.
+
+    Examples:
+    >>> reduce_entity_paths([["Account1", "Asset1"], ["Account2", "Asset2"]])
+    [["Account1", "Asset1"], ["Account2", "Asset2"]]
+
+    >>> reduce_entity_paths([["Asset1"], ["Asset2"]])
+    [["Asset1"], ["Asset2"]]
+
+    >>> reduce_entity_paths([["Account1", "Asset1"], ["Account1", "Asset2"]])
+    [["Asset1"], ["Asset2"]]
+
+    >>> reduce_entity_paths([["Asset1", "Asset2"], ["Asset1"]])
+    [["Asset1"], ["Asset1", "Asset2"]]
+
+    >>> reduce_entity_paths([["Account1", "Asset", "Asset1"], ["Account1", "Asset", "Asset2"]])
+    [["Asset1"], ["Asset2"]]
+    """
+    reduced_entities = 0
+
+    # At least we need to leave one entity in each list
+    max_reduced_entities = min([len(p) - 1 for p in asset_paths])
+
+    # Find the common path
+    while (
+        are_all_equal([p[:reduced_entities] for p in asset_paths])
+        and reduced_entities <= max_reduced_entities
+    ):
+        reduced_entities += 1
+
+    return [p[reduced_entities - 1 :] for p in asset_paths]
+
+
+def get_sensor_aliases(
+    sensors: list[Sensor],
+    reduce_paths: bool = True,
+    separator: str = "/",
+) -> dict:
+    """
+    Generates aliases for all sensors by appending a unique path to each sensor's name.
+
+    Parameters:
+    :param sensors:         A list of Sensor objects.
+    :param reduce_paths:    Flag indicating whether to reduce each sensor's entity path. Defaults to True.
+    :param separator:       Character or string used to separate entities within each sensor's path. Defaults to "/".
+
+    :return: A dictionary mapping sensor IDs to their generated aliases.
+    """
+
+    entity_paths = [
+        s.generic_asset.get_path(separator=separator).split(separator) for s in sensors
+    ]
+    if reduce_paths:
+        entity_paths = reduce_entity_paths(entity_paths)
+    entity_paths = [path_to_str(p, separator=separator) for p in entity_paths]
+
+    aliases = {
+        sensor.id: f"{sensor.name} ({path})"
+        for path, sensor in zip(entity_paths, sensors)
+    }
+
+    return aliases

flexmeasures/data/models/generic_assets.py

@@ -137,6 +137,14 @@ def asset_type(self) -> GenericAssetType:
         ),
     )
 
+    def get_path(self, separator: str = ">") -> str:
+        if self.parent_asset is not None:
+            return f"{self.parent_asset.get_path(separator=separator)}{separator}{self.name}"
+        elif self.owner is None:
+            return f"PUBLIC{separator}{self.name}"
+        else:
+            return f"{self.owner.get_path(separator=separator)}{separator}{self.name}"
+
     @property
     def offspring(self) -> list[GenericAsset]:
         """Returns a flattened list of all offspring, which is looked up recursively."""

flexmeasures/data/models/time_series.py

@@ -67,6 +67,11 @@ class Sensor(db.Model, tb.SensorDBMixin, AuthModelMixin):
         backref=db.backref("sensors", lazy="dynamic"),
     )
 
+    def get_path(self, separator: str = ">"):
+        return (
+            f"{self.generic_asset.get_path(separator=separator)}{separator}{self.name}"
+        )
+
     def __init__(
         self,
         name: str,

flexmeasures/data/models/user.py

@@ -102,6 +102,9 @@ def __acl__(self):
             "update": f"account:{self.id}",
         }
 
+    def get_path(self, separator: str = ">"):
+        return self.name
+
     def has_role(self, role: str | AccountRole) -> bool:
         """Returns `True` if the account has the specified role.