Merge bcbfa8d into 583fc87

documentation/api/change_log.rst

@@ -5,6 +5,11 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
+v3.0-10 | 2023-06-12
+""""""""""""""""""""
+
+- Introduced the ``storage-efficiency`` field to the ``flex-model``field for `/sensors/<id>/schedules/trigger` (POST).
+
 v3.0-9 | 2023-04-26
 """""""""""""""""""
 

documentation/api/notation.rst

@@ -185,27 +185,36 @@ This means that API and CLI users don't have to send the whole flex model every
 
 Here are the three types of flexibility models you can expect to be built-in:
 
-1) For storage devices (e.g. batteries, charge points, electric vehicle batteries connected to charge points), the schedule deals with the state of charge (SOC).
+1) For **storage devices** (e.g. batteries, and :abbr:`EV (electric vehicle)` batteries connected to charge points), the schedule deals with the state of charge (SOC).
 
-    The possible flexibility parameters are:
+   The possible flexibility parameters are:
 
-    - ``soc-at-start`` (defaults to 0)
-    - ``soc-unit`` (kWh or MWh)
-    - ``soc-min`` (defaults to 0)
-    - ``soc-max`` (defaults to max soc target)
-    - ``soc-targets`` (defaults to NaN values)
-    - ``roundtrip-efficiency`` (defaults to 100%)
-    - ``prefer-charging-sooner`` (defaults to True, also signals a preference to discharge later)
+   - ``soc-at-start`` (defaults to 0)
+   - ``soc-unit`` (kWh or MWh)
+   - ``soc-min`` (defaults to 0)
+   - ``soc-max`` (defaults to max soc target)
+   - ``soc-minima`` (defaults to NaN values)
+   - ``soc-maxima`` (defaults to NaN values)
+   - ``soc-targets`` (defaults to NaN values)
+   - ``roundtrip-efficiency`` (defaults to 100%)
+   - ``storage-efficiency`` (defaults to 100%) [#]_
+   - ``prefer-charging-sooner`` (defaults to True, also signals a preference to discharge later)
 
-  For some examples, see the `[POST] /sensors/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint docs.
+    .. [#] The storage efficiency (e.g. 95% or 0.95) to use for the schedule is applied over each time step equal to the sensor resolution. For example, a storage efficiency of 95 percent per (absolute) day, for scheduling a 1-hour resolution sensor, should be passed as a storage efficiency of :math:`0.95^{1/24} = 0.997865`.
 
-2) Shiftable process
+   For some examples, see the `[POST] /sensors/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint docs.
+
+2) For **shiftable processes**
 
    .. todo:: A simple algorithm exists, needs integration into FlexMeasures and asset type clarified.
 
-3) Heat pumps
-
-   .. todo:: Also work in progress, needs model for heat loss compensation.
+3) For **buffer devices** (e.g. thermal energy storage systems connected to heat pumps), use the same flexibility parameters described above for storage devices. Here are some tips to model a buffer with these parameters:
+
+   - Describe the thermal energy content in kWh or MWh.
+   - Set ``soc-minima`` to the accumulative usage forecast.
+   - Set ``roundtrip-efficiency`` to the square of the conversion efficiency. [#]_
+
+    .. [#] Setting a roundtrip efficiency of higher than 1 is not supported. We plan to implement a separate field for :abbr:`COP (coefficient of performance)` values.
 
 In addition, folks who write their own custom scheduler (see :ref:`plugin_customization`) might also require their custom flexibility model.
 That's no problem, FlexMeasures will let the scheduler decide which flexibility model is relevant and how it should be validated. 

documentation/changelog.rst

@@ -9,10 +9,10 @@ v0.14.0 | June XX, 2023
 New features
 -------------
 
-* Add multiple maxima and minima constraints into `StorageScheduler` [see `PR #680 <https://www.github.com/FlexMeasures/flexmeasures/pull/680>`_]
-* Add CLI command ``flexmeasures add report`` [see `PR #659 <https://www.github.com/FlexMeasures/flexmeasures/pull/659>`_]
-* Add CLI command ``flexmeasures show reporters`` [see `PR #686 <https://www.github.com/FlexMeasures/flexmeasures/pull/686>`_]
-* Add CLI command ``flexmeasures show schedulers`` [see `PR #708 <https://github.com/FlexMeasures/flexmeasures/pull/708>`_]
+* Allow setting a storage efficiency using the new ``storage-efficiency`` field when calling `/sensors/<id>/schedules/trigger` (POST) through the API (within the ``flex-model`` field), or when calling ``flexmeasures add schedule for-storage`` through the CLI [see `PR #679 <https://www.github.com/FlexMeasures/flexmeasures/pull/679>`_]
+* Allow setting multiple :abbr:`SoC (state of charge)` maxima and minima constraints for the `StorageScheduler`, using the new ``soc-minima`` and ``soc-maxima`` fields when calling `/sensors/<id>/schedules/trigger` (POST) through the API (within the ``flex-model`` field) [see `PR #680 <https://www.github.com/FlexMeasures/flexmeasures/pull/680>`_]
+* New CLI command ``flexmeasures add report`` to calculate a custom report from sensor data and save the results to the database, with the option to export them to a CSV or Excel file [see `PR #659 <https://www.github.com/FlexMeasures/flexmeasures/pull/659>`_]
+* New CLI commands ``flexmeasures show reporters`` and ``flexmeasures show schedulers`` to list available reporters and schedulers, respectively, including any defined in registered plugins [see `PR #686 <https://www.github.com/FlexMeasures/flexmeasures/pull/686>`_ and `PR #708 <https://github.com/FlexMeasures/flexmeasures/pull/708>`_]
 
 Bugfixes
 -----------
@@ -146,7 +146,7 @@ Bugfixes
 * The CLI command ``flexmeasures show beliefs`` now supports plotting time series data that includes NaN values, and provides better support for plotting multiple sensors that do not share the same unit [see `PR #516 <https://www.github.com/FlexMeasures/flexmeasures/pull/516>`_ and `PR #539 <https://www.github.com/FlexMeasures/flexmeasures/pull/539>`_]
 * Fixed JSON wrapping of return message for `/sensors/data` (GET) [see `PR #543 <https://www.github.com/FlexMeasures/flexmeasures/pull/543>`_]
 * Consistent CLI/UI support for asset lat/lng positions up to 7 decimal places (previously the UI rounded to 4 decimal places, whereas the CLI allowed more than 4) [see `PR #522 <https://www.github.com/FlexMeasures/flexmeasures/pull/522>`_]
-* Stop trimming the planning window in response to price availability, which is a problem when SoC targets occur outside of the available price window, by making a simplistic assumption about future prices [see `PR #538 <https://www.github.com/FlexMeasures/flexmeasures/pull/538>`_]
+* Stop trimming the planning window in response to price availability, which is a problem when :abbr:`SoC (state of charge)` targets occur outside of the available price window, by making a simplistic assumption about future prices [see `PR #538 <https://www.github.com/FlexMeasures/flexmeasures/pull/538>`_]
 * Faster loading of initial charts and calendar date selection [see `PR #533 <https://www.github.com/FlexMeasures/flexmeasures/pull/533>`_]
 
 Infrastructure / Support
@@ -177,7 +177,7 @@ v0.11.3 | November 2, 2022
 
 Bugfixes
 -----------
-* Fix scheduling with imperfect efficiencies, which resulted in exceeding the device's lower SoC limit. [see `PR #520 <https://www.github.com/FlexMeasures/flexmeasures/pull/520>`_]
+* Fix scheduling with imperfect efficiencies, which resulted in exceeding the device's lower :abbr:`SoC (state of charge)` limit. [see `PR #520 <https://www.github.com/FlexMeasures/flexmeasures/pull/520>`_]
 * Fix scheduler for Charge Points when taking into account inflexible devices [see `PR #517 <https://www.github.com/FlexMeasures/flexmeasures/pull/517>`_]
 * Prevent rounding asset lat/long positions to 4 decimal places when editing an asset in the UI [see `PR #522 <https://www.github.com/FlexMeasures/flexmeasures/pull/522>`_]
 

documentation/conf.py

@@ -46,7 +46,7 @@
     "sphinx_rtd_theme",
     "sphinx.ext.intersphinx",
     "sphinx.ext.coverage",
-    "sphinx.ext.imgmath",
+    "sphinx.ext.mathjax",
     "sphinx.ext.ifconfig",
     "sphinx.ext.todo",
     "sphinx_copybutton",

flexmeasures/api/v3_0/sensors.py

@@ -273,6 +273,7 @@ def trigger_schedule(  # noqa: C901
         To guarantee a minimum SOC in the period prior to 4.00pm, local minima constraints are imposed (via soc-minima)
         at 2.00pm and 3.00pm, for 15kWh and 20kWh, respectively.
         Roundtrip efficiency for use in scheduling is set to 98%.
+        Storage efficiency is set to 99.99%, denoting the state of charge left after each time step equal to the sensor's resolution.
         Aggregate consumption (of all devices within this EMS) should be priced by sensor 9,
         and aggregate production should be priced by sensor 10,
         where the aggregate power flow in the EMS is described by the sum over sensors 13, 14 and 15
@@ -306,6 +307,7 @@ def trigger_schedule(  # noqa: C901
                     "soc-min": 10,
                     "soc-max": 25,
                     "roundtrip-efficiency": 0.98,
+                    "storage-efficiency": 0.9999,
                 },
                 "flex-context": {
                     "consumption-price-sensor": 9,

flexmeasures/api/v3_0/tests/test_sensor_schedules.py

@@ -221,13 +221,19 @@ def test_trigger_and_get_schedule(
         roundtrip_efficiency = (
             float(message["roundtrip-efficiency"].replace("%", "")) / 100.0
         )
+        storage_efficiency = (
+            float(message["storage-efficiency"].replace("%", "")) / 100.0
+        )
         soc_targets = message.get("soc-targets")
     else:
         start_soc = message["flex-model"]["soc-at-start"] / 1000  # in MWh
         roundtrip_efficiency = (
             float(message["flex-model"]["roundtrip-efficiency"].replace("%", ""))
             / 100.0
         )
+        storage_efficiency = (
+            float(message["flex-model"]["storage-efficiency"].replace("%", "")) / 100.0
+        )
         soc_targets = message["flex-model"].get("soc-targets")
     resolution = sensor.event_resolution
     if soc_targets:
@@ -271,6 +277,7 @@ def test_trigger_and_get_schedule(
             start_soc,
             up_efficiency=roundtrip_efficiency**0.5,
             down_efficiency=roundtrip_efficiency**0.5,
+            storage_efficiency=storage_efficiency,
             decimal_precision=6,
         )
         print(consumption_schedule)

flexmeasures/api/v3_0/tests/utils.py

@@ -61,6 +61,7 @@ def message_for_trigger_schedule(
         "soc-max": 40,  # in kWh, according to soc-unit
         "soc-unit": "kWh",
         "roundtrip-efficiency": "98%",
+        "storage-efficiency": "99.99%",
     }
     if with_targets:
         if realistic_targets:

flexmeasures/cli/data_add.py

@@ -49,6 +49,7 @@
     LongitudeField,
     SensorIdField,
 )
+from flexmeasures.data.schemas.scheduling.storage import EfficiencyField
 from flexmeasures.data.schemas.sensors import SensorSchema
 from flexmeasures.data.schemas.units import QuantityField
 from flexmeasures.data.schemas.generic_assets import (
@@ -1012,11 +1013,22 @@ def create_schedule(ctx):
 @click.option(
     "--roundtrip-efficiency",
     "roundtrip_efficiency",
-    type=QuantityField("%", validate=validate.Range(min=0, max=1)),
+    type=EfficiencyField(),
     required=False,
     default=1,
     help="Round-trip efficiency (e.g. 85% or 0.85) to use for the schedule. Defaults to 100% (no losses).",
 )
+@click.option(
+    "--storage-efficiency",
+    "storage_efficiency",
+    type=EfficiencyField(),
+    required=False,
+    default=1,
+    help="Storage efficiency (e.g. 95% or 0.95) to use for the schedule,"
+    " applied over each time step equal to the sensor resolution."
+    " For example, a storage efficiency of 99 percent per (absolute) day, for scheduling a 1-hour resolution sensor, should be passed as a storage efficiency of 0.99**(1/24)."
+    " Defaults to 100% (no losses).",
+)
 @click.option(
     "--as-job",
     is_flag=True,
@@ -1036,6 +1048,7 @@ def add_schedule_for_storage(
     soc_min: ur.Quantity | None = None,
     soc_max: ur.Quantity | None = None,
     roundtrip_efficiency: ur.Quantity | None = None,
+    storage_efficiency: ur.Quantity | None = None,
     as_job: bool = False,
 ):
     """Create a new schedule for a storage asset.
@@ -1091,6 +1104,8 @@ def add_schedule_for_storage(
         soc_max = convert_units(soc_max.magnitude, str(soc_max.units), "MWh", capacity=capacity_str)  # type: ignore
     if roundtrip_efficiency is not None:
         roundtrip_efficiency = roundtrip_efficiency.magnitude / 100.0
+    if storage_efficiency is not None:
+        storage_efficiency = storage_efficiency.magnitude / 100.0
 
     scheduling_kwargs = dict(
         start=start,
@@ -1104,6 +1119,7 @@ def add_schedule_for_storage(
             "soc-max": soc_max,
             "soc-unit": "MWh",
             "roundtrip-efficiency": roundtrip_efficiency,
+            "storage-efficiency": storage_efficiency,
         },
         flex_context={
             "consumption-price-sensor": consumption_price_sensor.id,

flexmeasures/data/models/planning/linear_optimization.py

@@ -21,6 +21,7 @@
 from pyomo.opt import SolverFactory, SolverResults
 
 from flexmeasures.data.models.planning.utils import initialize_series
+from flexmeasures.utils.calculations import apply_stock_changes_and_losses
 
 infinity = float("inf")
 
@@ -31,6 +32,7 @@ def device_scheduler(  # noqa C901
     commitment_quantities: List[pd.Series],
     commitment_downwards_deviation_price: Union[List[pd.Series], List[float]],
     commitment_upwards_deviation_price: Union[List[pd.Series], List[float]],
+    initial_stock: float = 0,
 ) -> Tuple[List[pd.Series], float, SolverResults]:
     """This generic device scheduler is able to handle an EMS with multiple devices,
     with various types of constraints on the EMS level and on the device level,
@@ -43,6 +45,7 @@ def device_scheduler(  # noqa C901
         max: maximum stock assuming an initial stock of zero (e.g. in MWh or boxes)
         min: minimum stock assuming an initial stock of zero
         equal: exact amount of stock (we do this by clamping min and max)
+        efficiency: amount of stock left at the next datetime (the rest is lost)
         derivative max: maximum flow (e.g. in MW or boxes/h)
         derivative min: minimum flow
         derivative equals: exact amount of flow (we do this by clamping derivative min and derivative max)
@@ -171,6 +174,16 @@ def ems_derivative_min_select(m, j):
         else:
             return v
 
+    def device_efficiency(m, d, j):
+        """Assume perfect efficiency if no efficiency information is available."""
+        try:
+            eff = device_constraints[d]["efficiency"].iloc[j]
+        except KeyError:
+            return 1
+        if np.isnan(eff):
+            return 1
+        return eff
+
     def device_derivative_down_efficiency(m, d, j):
         """Assume perfect efficiency if no efficiency information is available."""
         try:
@@ -206,6 +219,7 @@ def device_derivative_up_efficiency(m, d, j):
     )
     model.ems_derivative_max = Param(model.j, initialize=ems_derivative_max_select)
     model.ems_derivative_min = Param(model.j, initialize=ems_derivative_min_select)
+    model.device_efficiency = Param(model.d, model.j, initialize=device_efficiency)
     model.device_derivative_down_efficiency = Param(
         model.d, model.j, initialize=device_derivative_down_efficiency
     )
@@ -228,14 +242,24 @@ def device_derivative_up_efficiency(m, d, j):
 
     # Add constraints as a tuple of (lower bound, value, upper bound)
     def device_bounds(m, d, j):
-        """Apply efficiencies to conversion from flow to stock change and vice versa."""
-        return (
-            m.device_min[d, j],
-            sum(
+        """Apply conversion efficiencies to conversion from flow to stock change and vice versa,
+        and apply storage efficiencies to stock levels from one datetime to the next."""
+        stock_changes = [
+            (
                 m.device_power_down[d, k] / m.device_derivative_down_efficiency[d, k]
                 + m.device_power_up[d, k] * m.device_derivative_up_efficiency[d, k]
-                for k in range(0, j + 1)
-            ),
+            )
+            for k in range(0, j + 1)
+        ]
+        efficiencies = [m.device_efficiency[d, k] for k in range(0, j + 1)]
+        return (
+            m.device_min[d, j],
+            [
+                stock - initial_stock
+                for stock in apply_stock_changes_and_losses(
+                    initial_stock, stock_changes, efficiencies
+                )
+            ][-1],
             m.device_max[d, j],
         )
 

flexmeasures/data/models/planning/storage.py

@@ -34,6 +34,7 @@ class StorageScheduler(Scheduler):
         "equals",
         "max",
         "min",
+        "efficiency",
         "derivative equals",
         "derivative max",
         "derivative min",
@@ -73,6 +74,7 @@ def compute(self, skip_validation: bool = False) -> pd.Series | None:
         soc_minima = self.flex_model.get("soc_minima")
         soc_maxima = self.flex_model.get("soc_maxima")
         roundtrip_efficiency = self.flex_model.get("roundtrip_efficiency")
+        storage_efficiency = self.flex_model.get("storage_efficiency")
         prefer_charging_sooner = self.flex_model.get("prefer_charging_sooner", True)
 
         consumption_price_sensor = self.flex_context.get("consumption_price_sensor")
@@ -126,7 +128,7 @@ def compute(self, skip_validation: bool = False) -> pd.Series | None:
             down_deviation_prices.loc[start : end - resolution]["event_value"]
         ]
 
-        # Set up device _constraints: only one scheduled flexible device for this EMS (at index 0), plus the forecasted inflexible devices (at indices 1 to n).
+        # Set up device constraints: only one scheduled flexible device for this EMS (at index 0), plus the forecasted inflexible devices (at indices 1 to n).
         device_constraints = [
             initialize_df(StorageScheduler.COLUMNS, start, end, resolution)
             for i in range(1 + len(inflexible_device_sensors))
@@ -170,6 +172,9 @@ def compute(self, skip_validation: bool = False) -> pd.Series | None:
         )
         device_constraints[0]["derivative up efficiency"] = roundtrip_efficiency**0.5
 
+        # Apply storage efficiency (accounts for losses over time)
+        device_constraints[0]["efficiency"] = storage_efficiency
+
         # check that storage constraints are fulfilled
         if not skip_validation:
             constraint_violations = validate_storage_constraints(
@@ -199,6 +204,7 @@ def compute(self, skip_validation: bool = False) -> pd.Series | None:
             commitment_quantities,
             commitment_downwards_deviation_price,
             commitment_upwards_deviation_price,
+            initial_stock=soc_at_start * (timedelta(hours=1) / resolution),
         )
         if scheduler_results.solver.termination_condition == "infeasible":
             # Fallback policy if the problem was unsolvable
@@ -268,7 +274,19 @@ def deserialize_flex_config(self):
             elif self.sensor.unit in ("MW", "kW"):
                 self.flex_model["soc-unit"] = self.sensor.unit + "h"
 
+        # Check for storage efficiency
+        # todo: simplify to: `if self.flex_model.get("storage-efficiency") is None:`
+        if (
+            "storage-efficiency" not in self.flex_model
+            or self.flex_model["storage-efficiency"] is None
+        ):
+            # Get default from sensor, or use 100% otherwise
+            self.flex_model["storage-efficiency"] = self.sensor.get_attribute(
+                "storage_efficiency", 1
+            )
+
         # Check for round-trip efficiency
+        # todo: simplify to: `if self.flex_model.get("roundtrip-efficiency") is None:`
         if (
             "roundtrip-efficiency" not in self.flex_model
             or self.flex_model["roundtrip-efficiency"] is None