Merge branch 'main' into feature/plannign/soc-constraints-as-sensors

documentation/api/change_log.rst

@@ -5,6 +5,12 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace.
 
+
+v3.0-16 | 2024-02-26
+""""""""""""""""""""
+
+- Fix support for providing a sensor definition to the ``power-capacity`` flex-model field for `/sensors/<id>/schedules/trigger` (POST).
+
 v3.0-15 | 2024-01-11
 """"""""""""""""""""
 

documentation/cli/change_log.rst

@@ -9,6 +9,11 @@ since v.0.20.0 | March XX, 2024
 
 * Add command ``flexmeasures edit transfer-ownership`` to transfer the ownership of an asset and its children.
 
+since v0.19.1 | February 26, 2024
+=======================================
+
+* Fix support for providing a sensor definition to the ``--storage-power-capacity`` option of the ``flexmeasures add schedule for-storage`` command.
+
 since v0.19.0 | February 18, 2024
 =======================================
 
@@ -32,13 +37,11 @@ since v0.19.0 | February 18, 2024
     * ``--source-id`` -> ``--source``
     * ``--user-id`` -> ``--user`
 
-
-since v0.18.1 | January 15, 2023
+since v0.18.1 | January 15, 2024
 =======================================
 
 * Fix the validation of the option ``--parent-asset`` of command ``flexmeasures add asset``.
 
-
 since v0.17.0 | November 8, 2023
 =======================================
 

documentation/concepts/security_auth.rst

@@ -41,13 +41,14 @@ For instance, a user is authorized to update his or her personal data, like the
 
 .. note:: Each user belongs to exactly one account.
 
-In a nutshell, the way FlexMeasures implements authorization works as follows: The data models codify under which conditions a user can have certain permissions to work with their data. Permissions allow distinct ways of access like reading, writing or deleting. The API endpoints are where we know what needs to happen to what data, so there we make sure that the user has the necessary permissions.
+In a nutshell, the way FlexMeasures implements authorization works as follows: The data models codify under which conditions a user can have certain permissions to work with their data (in code, look for the ``__acl__`` function, where the access control list is defined). Permissions allow distinct ways of access like reading, writing or deleting. The API endpoints are where we know what needs to happen to what data, so there we make sure that the user has the necessary permissions.
 
 We already discussed certain conditions under which a user has access to data ― being a certain user or belonging to a specific account. Furthermore, authorization conditions can also be implemented via *roles*: 
 
 * ``Account roles`` are often used for authorization. We support several roles which are mentioned in the USEF framework but more roles are possible (e.g. defined by custom-made services, see below). For example, a user might be authorized to write sensor data if they belong to an account with the "MDC" account role ("MDC" being short for meter data company).
 * ``User roles`` give a user personal authorizations. For instance, we have a few `admin`\ s who can perform all actions, and `admin-reader`\ s who can read everything. Other roles have only an effect within the user's account, e.g. there could be an "HR" role which allows to edit user data like surnames within the account.
+* A special case are consultant accounts ― accounts which can read data on other accounts (usually their clients, handy for servicing them). For this, accounts have an attribute called ``consultancy_account_id``. Users in the consultant account with role `consultant` can read data in their client accounts. We plan to introduce some editing/creation capabilities in the future. You can also add a consultant account when creating a client account, for instance on the CLI: ``flexmeasures add account --name "Account2" --consultancy 1``.
 * Roles cannot be edited via the UI at the moment. They are decided when a user or account is created in the CLI (for adding roles later, we use the database for now). Editing roles in UI and CLI is future work.
 
 
-.. note:: Custom energy flexibility services developed on top of FlexMeasures also need to implement authorization. More on this in :ref:`auth-dev`. Here is an example for a custom authorization concept: services can use account roles to achieve their custom authorization. E.g. if several services run on one FlexMeasures server, each service could define a "MyService-subscriber" account role, to make sure that only users of such accounts can use the endpoints.
+.. note:: Custom energy flexibility services developed on top of FlexMeasures also need to implement authorization. More on this in :ref:`auth-dev`. Here is an example for a custom authorization concept: services can use account roles to achieve their custom authorization. E.g. if several services run on one FlexMeasures server, each service could define a "MyService-subscriber" account role, to make sure that only users of such accounts can use the endpoints.

flexmeasures/cli/data_add.py

@@ -1218,10 +1218,11 @@ def create_schedule(ctx):
 @click.option(
     "--storage-power-capacity",
     "storage_power_capacity",
-    type=QuantityField("MW"),
+    type=QuantityOrSensor("MW"),
     required=False,
     default=None,
-    help="Storage consumption/production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)."
+    help="Storage consumption/production power capacity. Provide this as a quantity in power units (e.g. 1 MW or 1000 kW)"
+    "or reference a sensor using 'sensor:<id>' (e.g. sensor:34)."
     "It defines both-ways maximum power capacity.",
 )
 @click.option(

flexmeasures/cli/tests/test_data_add.py

@@ -461,7 +461,7 @@ def test_add_account(
 @pytest.mark.skip_github
 @pytest.mark.parametrize("storage_power_capacity", ["sensor", "quantity", None])
 @pytest.mark.parametrize("storage_efficiency", ["sensor", "quantity", None])
-def test_add_storage_scheduler(
+def test_add_storage_schedule(
     app,
     add_market_prices_fresh_db,
     storage_schedule_sensors,

flexmeasures/conftest.py

@@ -1140,6 +1140,14 @@ def capacity_sensors(db, add_battery_assets, setup_sources):
         attributes={"consumption_is_positive": True},
     )
 
+    power_capacity_sensor = Sensor(
+        name="power capacity",
+        generic_asset=battery,
+        unit="kW",
+        event_resolution="PT15M",
+        attributes={"consumption_is_positive": True},
+    )
+
     db.session.add_all([production_capacity_sensor, consumption_capacity_sensor])
     db.session.flush()
 
@@ -1181,6 +1189,23 @@ def capacity_sensors(db, add_battery_assets, setup_sources):
     db.session.add_all(beliefs)
     db.session.commit()
 
+    values = [225] * 4 * 4 + [199] * 4 * 4
+
+    beliefs = [
+        TimedBelief(
+            event_start=dt,
+            belief_horizon=parse_duration("PT0M"),
+            event_value=val,
+            sensor=power_capacity_sensor,
+            source=setup_sources["Seita"],
+        )
+        for dt, val in zip(time_slots, values)
+    ]
+    db.session.add_all(beliefs)
+    db.session.commit()
+
     yield dict(
-        production=production_capacity_sensor, consumption=consumption_capacity_sensor
+        production=production_capacity_sensor,
+        consumption=consumption_capacity_sensor,
+        power_capacity=power_capacity_sensor,
     )

flexmeasures/data/migrations/versions/6938f16617ab_add_search_index_for_beliefs.py

@@ -0,0 +1,32 @@
+"""add search index for beliefs
+
+Revision ID: 6938f16617ab
+Revises: c349f52c700d
+Create Date: 2024-03-01 09:55:34.910868
+
+"""
+from alembic import op
+
+
+# revision identifiers, used by Alembic.
+revision = "6938f16617ab"
+down_revision = "c349f52c700d"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    with op.batch_alter_table("timed_belief", schema=None) as batch_op:
+        batch_op.create_index(
+            "timed_belief_search_session_idx",
+            ["event_start", "sensor_id", "source_id"],
+            unique=False,
+            postgresql_include=["belief_horizon"],
+        )
+
+
+def downgrade():
+    with op.batch_alter_table("timed_belief", schema=None) as batch_op:
+        batch_op.drop_index(
+            "timed_belief_search_session_idx", postgresql_include=["belief_horizon"]
+        )

flexmeasures/data/migrations/versions/ad98460751d9_remove_obsolete_tables.py

@@ -22,18 +22,21 @@
 
 
 def upgrade():
-    tables_with_data = []
-    for table in [
+    tables = [
         "price",
         "power",
         "market",
         "market_type",
         "weather",
         "asset",
         "weather_sensor",
-    ]:
+    ]
+
+    #  check for existing data
+    tables_with_data = []
+    inspect = sa.inspect(db.engine)
+    for table in tables:
         try:
-            inspect = sa.inspect(db.engine)
             if inspect.has_table(table):
                 result = db.session.execute(
                     sa.text(f"SELECT 1 FROM {table};")
@@ -52,33 +55,27 @@ def upgrade():
             abort=True,
         )
 
+    # drop indexes
     with op.batch_alter_table("power", schema=None) as batch_op:
-        batch_op.drop_index("power_datetime_idx")
-        batch_op.drop_index("power_sensor_id_idx")
+        batch_op.drop_index("power_datetime_idx", if_exists=True)
+        batch_op.drop_index("power_sensor_id_idx", if_exists=True)
 
     with op.batch_alter_table("asset_type", schema=None) as batch_op:
-        batch_op.drop_index("asset_type_can_curtail_idx")
-        batch_op.drop_index("asset_type_can_shift_idx")
+        batch_op.drop_index("asset_type_can_curtail_idx", if_exists=True)
+        batch_op.drop_index("asset_type_can_shift_idx", if_exists=True)
 
     with op.batch_alter_table("weather", schema=None) as batch_op:
-        batch_op.drop_index("weather_datetime_idx")
-        batch_op.drop_index("weather_sensor_id_idx")
+        batch_op.drop_index("weather_datetime_idx", if_exists=True)
+        batch_op.drop_index("weather_sensor_id_idx", if_exists=True)
 
     with op.batch_alter_table("price", schema=None) as batch_op:
-        batch_op.drop_index("price_datetime_idx")
-        batch_op.drop_index("price_sensor_id_idx")
-
-    op.drop_table("asset")
-    op.drop_table("power")
-    op.drop_table("asset_type")
-
-    op.drop_table("weather_sensor")
-    op.drop_table("weather")
-    op.drop_table("weather_sensor_type")
+        batch_op.drop_index("price_datetime_idx", if_exists=True)
+        batch_op.drop_index("price_sensor_id_idx", if_exists=True)
 
-    op.drop_table("market")
-    op.drop_table("market_type")
-    op.drop_table("price")
+    # drop tables
+    for table in tables:
+        if inspect.has_table(table):
+            op.drop_table(table)
 
 
 def downgrade():

flexmeasures/data/models/planning/storage.py

@@ -122,16 +122,19 @@ def _prepare(self, skip_validation: bool = False) -> tuple:  # noqa: C901
                 "Power capacity is not defined in the sensor attributes or the flex-model."
             )
 
-        if isinstance(power_capacity_in_mw, ur.Quantity):
-            power_capacity_in_mw = power_capacity_in_mw.magnitude
-
-        if not (
-            isinstance(power_capacity_in_mw, float)
-            or isinstance(power_capacity_in_mw, int)
+        if isinstance(power_capacity_in_mw, float) or isinstance(
+            power_capacity_in_mw, int
         ):
-            raise ValueError(
-                "The only supported types for the power capacity are int and float."
-            )
+            power_capacity_in_mw = ur.Quantity(f"{power_capacity_in_mw} MW")
+
+        power_capacity_in_mw = get_continuous_series_sensor_or_quantity(
+            quantity_or_sensor=power_capacity_in_mw,
+            actuator=sensor,
+            unit="MW",
+            query_window=(start, end),
+            resolution=resolution,
+            beliefs_before=belief_time,
+        )
 
         # Check for known prices or price forecasts, trimming planning window accordingly
         up_deviation_prices, (start, end) = get_prices(

flexmeasures/data/models/planning/tests/test_solver.py

@@ -1475,6 +1475,49 @@ def test_battery_power_capacity_as_sensor(
     assert all(device_constraints["derivative max"].values == expected_consumption)
 
 
+def test_battery_bothways_power_capacity_as_sensor(
+    db, add_battery_assets, add_inflexible_device_forecasts, capacity_sensors
+):
+    """Check that the charging and discharging power capacities are limited by the power capacity."""
+    epex_da, battery = get_sensors_from_db(
+        db, add_battery_assets, battery_name="Test battery"
+    )
+
+    tz = pytz.timezone("Europe/Amsterdam")
+    start = tz.localize(datetime(2015, 1, 2))
+    end = tz.localize(datetime(2015, 1, 2, 7, 45))
+    resolution = timedelta(minutes=15)
+    soc_at_start = 10
+
+    flex_model = {
+        "soc-at-start": soc_at_start,
+        "roundtrip-efficiency": "100%",
+        "prefer-charging-sooner": False,
+    }
+
+    flex_model["power-capacity"] = {"sensor": capacity_sensors["production"].id}
+    flex_model["consumption-capacity"] = {"sensor": capacity_sensors["consumption"].id}
+    flex_model["production-capacity"] = {
+        "sensor": capacity_sensors["power_capacity"].id
+    }
+
+    scheduler: Scheduler = StorageScheduler(
+        battery, start, end, resolution, flex_model=flex_model
+    )
+
+    data_to_solver = scheduler._prepare()
+    device_constraints = data_to_solver[5][0]
+
+    max_capacity = (
+        capacity_sensors["power_capacity"]
+        .search_beliefs(event_starts_after=start, event_ends_before=end)
+        .event_value.values
+    )
+
+    assert all(device_constraints["derivative min"].values >= -max_capacity)
+    assert all(device_constraints["derivative max"].values <= max_capacity)
+
+
 def get_efficiency_problem_device_constraints(
     extra_flex_model, efficiency_sensors, add_battery_assets, db
 ) -> pd.DataFrame:

flexmeasures/data/models/planning/utils.py

@@ -373,7 +373,7 @@ def get_continuous_series_sensor_or_quantity(
     resolution: timedelta,
     beliefs_before: datetime | None = None,
     fallback_attribute: str | None = None,
-    max_value: float | int = np.nan,
+    max_value: float | int | pd.Series = np.nan,
 ) -> pd.Series:
     """Creates a time series from a quantity or sensor within a specified window,
     falling back to a given `fallback_attribute` and making sure no values exceed `max_value`.
@@ -409,6 +409,12 @@ def get_continuous_series_sensor_or_quantity(
     return time_series
 
 
-def nanmin_of_series_and_value(s: pd.Series, value: float) -> pd.Series:
+def nanmin_of_series_and_value(s: pd.Series, value: float | pd.Series) -> pd.Series:
     """Perform a nanmin between a Series and a float."""
+    if isinstance(value, pd.Series):
+        # Avoid strange InvalidIndexError on .clip due to different "dtype"
+        # pd.testing.assert_index_equal(value.index, s.index)
+        # [left]:  datetime64[ns, +0000]
+        # [right]: datetime64[ns, UTC]
+        value = value.tz_convert("UTC")
     return s.fillna(value).clip(upper=value)

flexmeasures/data/models/time_series.py

@@ -531,7 +531,7 @@ def find_closest(
             account_id=account_id_filter,
         )
         if n == 1:
-            return db.session.scalar(query.limit(1)).first()
+            return db.session.scalars(query.limit(1)).first()
         else:
             return db.session.scalars(query.limit(n)).all()
 

flexmeasures/ui/utils/breadcrumb_utils.py

@@ -1,8 +1,9 @@
 from __future__ import annotations
 
+from sqlalchemy import select
 from flexmeasures import Sensor, Asset, Account
 from flexmeasures.utils.flexmeasures_inflection import human_sorted
-from flask import url_for
+from flask import url_for, current_app
 
 
 def get_breadcrumb_info(entity: Sensor | Asset | Account | None) -> dict:
@@ -72,8 +73,14 @@ def get_siblings(entity: Sensor | Asset | Account | None) -> list[dict]:
     if isinstance(entity, Asset):
         if entity.parent_asset is not None:
             sibling_assets = entity.parent_asset.child_assets
-        else:
+        elif entity.owner is not None:
             sibling_assets = entity.owner.generic_assets
+        else:
+            session = current_app.db.session
+            sibling_assets = session.scalars(
+                select(Asset).filter(Asset.account_id.is_(None))
+            ).all()
+
         siblings = [
             {
                 "url": url_for("AssetCrudUI:get", id=asset.id),

requirements/3.10/app.txt

@@ -323,7 +323,7 @@ tabulate==0.9.0
     # via -r requirements/app.in
 threadpoolctl==3.2.0
     # via scikit-learn
-timely-beliefs[forecast]==2.0.0
+timely-beliefs[forecast]==2.1.0
     # via -r requirements/app.in
 timetomodel==0.7.3
     # via -r requirements/app.in

requirements/3.11/app.txt

@@ -321,7 +321,7 @@ tabulate==0.9.0
     # via -r requirements/app.in
 threadpoolctl==3.2.0
     # via scikit-learn
-timely-beliefs[forecast]==2.0.0
+timely-beliefs[forecast]==2.1.0
     # via -r requirements/app.in
 timetomodel==0.7.3
     # via -r requirements/app.in

requirements/3.8/app.txt

@@ -336,7 +336,7 @@ tabulate==0.9.0
     # via -r requirements/app.in
 threadpoolctl==3.2.0
     # via scikit-learn
-timely-beliefs[forecast]==2.0.0
+timely-beliefs[forecast]==2.1.0
     # via -r requirements/app.in
 timetomodel==0.7.3
     # via -r requirements/app.in