Merge branch 'main' into update-piptools

documentation/api/change_log.rst

@@ -3,7 +3,7 @@
 API change log
 ===============
 
-.. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
+.. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace.
 
 v3.0-11 | 2023-08-02
 """"""""""""""""""""

documentation/changelog.rst

@@ -3,9 +3,12 @@
 FlexMeasures Changelog
 **********************
 
-v0.15.0 | July XX, 2023
+v0.15.0 | August 9, 2023
 ============================
 
+.. note:: Read more on these features on `the FlexMeasures blog <https://flexmeasures.io/015-process-scheduling-heatmap/>`__.
+
+
 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
 
 .. warning:: Upgrading to this version requires installing the LP/MILP solver HiGHS using ``pip install highspy``.
@@ -15,18 +18,18 @@ v0.15.0 | July XX, 2023
 New features
 -------------
 
+* Add `ProcessScheduler` class to optimize the starting time of processes one of the policies developed (INFLEXIBLE, SHIFTABLE and BREAKABLE), accessible via the CLI command `flexmeasures add schedule for-process` [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_ and `PR #768 <https://www.github.com/FlexMeasures/flexmeasures/pull/768>`_]
 * Users can select a new chart type (daily heatmap) on the sensor page of the UI, showing how sensor values are distributed over the time of day [see `PR #715 <https://www.github.com/FlexMeasures/flexmeasures/pull/715>`_]
+* Added API endpoints `/sensors/<id>` (GET) for fetching a single sensor, `/sensors` (POST) for adding a sensor, `/sensors/<id>` (PATCH) for updating a sensor and `/sensors/<id>` (DELETE) for deleting a sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_] and [see `PR #767 <https://www.github.com/FlexMeasures/flexmeasures/pull/767>`_] and [see `PR #773 <https://www.github.com/FlexMeasures/flexmeasures/pull/773>`_] and [see `PR #784 <https://www.github.com/FlexMeasures/flexmeasures/pull/784>`_]
 * Users are warned in the UI on when the data they are seeing includes one or more Daylight Saving Time (DST) transitions, and heatmaps (see previous feature) visualize these transitions intuitively [see `PR #723 <https://www.github.com/FlexMeasures/flexmeasures/pull/723>`_]
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times [see `PR #734 <https://www.github.com/FlexMeasures/flexmeasures/pull/734>`_]
 * Make it a lot easier to read off the color legend on the asset page, especially when showing many sensors, as they will now be ordered from top to bottom in the same order as they appear in the chart (as defined in the ``sensors_to_show`` attribute), rather than alphabetically [see `PR #742 <https://www.github.com/FlexMeasures/flexmeasures/pull/742>`_]
 * Users on FlexMeasures servers in play mode (``FLEXMEASURES_MODE = "play"``) can use the ``sensors_to_show`` attribute to show any sensor on their asset pages, rather than only sensors registered to assets in their own account or to public assets [see `PR #740 <https://www.github.com/FlexMeasures/flexmeasures/pull/740>`_]
 * Having percentages within the [0, 100] domain is such a common use case that we now always include it in sensor charts with % units, making it easier to read off individual charts and also to compare across charts [see `PR #739 <https://www.github.com/FlexMeasures/flexmeasures/pull/739>`_]
 * DataSource table now allows storing arbitrary attributes as a JSON (without content validation), similar to the Sensor and GenericAsset tables [see `PR #750 <https://www.github.com/FlexMeasures/flexmeasures/pull/750>`_]
-* Added API endpoints `/sensors/<id>` for fetching a single sensor, `/sensors` (POST) for adding a sensor, `/sensors/<id>` (PATCH) for updating a sensor and `/sensors/<id>` (DELETE) for deleting a sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_] and [see `PR #767 <https://www.github.com/FlexMeasures/flexmeasures/pull/767>`_] and [see `PR #773 <https://www.github.com/FlexMeasures/flexmeasures/pull/773>`_] and [see `PR #784 <https://www.github.com/FlexMeasures/flexmeasures/pull/784>`_]
+* Users will be able to see (e.g. in the UI) exactly which reporter created the report (saved as sensor data), and hosts will be able to identify exactly which configuration was used to create a given report [see `PR #751 <https://www.github.com/FlexMeasures/flexmeasures/pull/751>`_ and `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
+* The CLI `flexmeasures add report` now allows passing `config` and `parameters` in YAML format as files or editable via the system's default editor [see `PR #752 <https://www.github.com/FlexMeasures/flexmeasures/pull/752>`_ and `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
 * The CLI now allows to set lists and dicts as asset & sensor attributes (formerly only single values) [see `PR #762 <https://www.github.com/FlexMeasures/flexmeasures/pull/762>`_]
-* Add `ProcessScheduler` class to optimize the starting time of processes one of the policies developed (INFLEXIBLE, SHIFTABLE and BREAKABLE), accessible via the CLI command `flexmeasures add schedule for-process` [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_ and `PR #768 <https://www.github.com/FlexMeasures/flexmeasures/pull/768>`_]
-* Users will be able to see (e.g. in the UI) exactly which reporter created the report (saved as sensor data), and hosts will be able to identify exactly which configuration was used to create a given report [see `PR #751 <https://www.github.com/FlexMeasures/flexmeasures/pull/751>`_, `PR #752 <https://www.github.com/FlexMeasures/flexmeasures/pull/752>`_ and `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
-* The CLI `flexmeasures add report` now allows passing `config` and `parameters` in YAML format as files or editable via the system's default editor [see `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
 
 Bugfixes
 -----------
@@ -38,9 +41,9 @@ Infrastructure / Support
 
 * Add support for profiling Flask API calls using ``pyinstrument`` (if installed). Can be enabled by setting the environment variable ``FLEXMEASURES_PROFILE_REQUESTS`` to ``True`` [see `PR #722 <https://www.github.com/FlexMeasures/flexmeasures/pull/722>`_]
 * The endpoint `[POST] /health/ready <api/v3_0.html#get--api-v3_0-health-ready>`_ returns the status of the Redis connection, if configured [see `PR #699 <https://www.github.com/FlexMeasures/flexmeasures/pull/699>`_]
-* Document the `device_scheduler` linear program [see `PR #764 <https://www.github.com/FlexMeasures/flexmeasures/pull/764>`_].
-* Add support for `HiGHS <https://highs.dev/>`_ solver [see `PR #766 <https://www.github.com/FlexMeasures/flexmeasures/pull/766>`_].
-* Add support for installing FlexMeasures under Python 3.11 [see `PR #771 <https://www.github.com/FlexMeasures/flexmeasures/pull/771>`_].
+* Document the `device_scheduler` linear program [see `PR #764 <https://www.github.com/FlexMeasures/flexmeasures/pull/764>`_]
+* Add support for `HiGHS <https://highs.dev/>`_ solver [see `PR #766 <https://www.github.com/FlexMeasures/flexmeasures/pull/766>`_]
+* Add support for installing FlexMeasures under Python 3.11 [see `PR #771 <https://www.github.com/FlexMeasures/flexmeasures/pull/771>`_]
 * Start keeping sets of pinned requirements per supported Python version. Also fixes recent Docker build problem. [see `PR #776 <https://www.github.com/FlexMeasures/flexmeasures/pull/776>`_]
 * Removed obsolete code dealing with deprecated data models (e.g. assets, markets and weather sensors), and sunset the fm0 scheme for entity addresses [see `PR #695 <https://www.github.com/FlexMeasures/flexmeasures/pull/695>`_ and `project 11 <https://www.github.com/FlexMeasures/flexmeasures/projects/11>`_]
 

documentation/cli/change_log.rst

@@ -4,15 +4,15 @@
 FlexMeasures CLI Changelog
 **********************
 
-since v0.15.0 | July XX, 2023
+since v0.15.0 | August 9, 2023
 =================================
 
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times.
 * Add ``flexmeasures add schedule for-process`` to create a new process schedule for a given power sensor.
 * Add support for describing ``config`` and ``parameters`` in YAML for the command ``flexmeasures add report``, editable in user's code editor using the flags ``--edit-config`` or ``--edit-parameters``.
 * Add ``--kind process`` option to create the asset and sensors for the ``ProcessScheduler`` tutorial.
 
-since v0.14.1 | June XX, 2023
+since v0.14.1 | June 20, 2023
 =================================
 
 * Avoid saving any :abbr:`NaN (not a number)` values to the database, when calling ``flexmeasures add report``.

flexmeasures/api/v3_0/tests/test_sensors_api.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import pytest
+
 from flask import url_for
 
 from flexmeasures.data.models.time_series import TimedBelief
@@ -88,6 +89,7 @@ def test_post_a_sensor(client, setup_api_test_data):
     sensor: Sensor = Sensor.query.filter_by(name="power").one_or_none()
     assert sensor is not None
     assert sensor.unit == "kWh"
+    assert sensor.attributes["capacity_in_mw"] == 0.0074
 
 
 def test_post_sensor_to_asset_from_unrelated_account(client, setup_api_test_data):
@@ -117,12 +119,14 @@ def test_patch_sensor(client, setup_api_test_data):
         headers={"content-type": "application/json", "Authorization": auth_token},
         json={
             "name": "Changed name",
+            "attributes": '{"test_attribute": "test_attribute_value"}',
         },
     )
     assert response.json["name"] == "Changed name"
     new_sensor = Sensor.query.filter(Sensor.name == "Changed name").one_or_none()
     assert new_sensor.name == "Changed name"
     assert Sensor.query.filter(Sensor.name == "some gas sensor").one_or_none() is None
+    assert new_sensor.attributes["test_attribute"] == "test_attribute_value"
 
 
 @pytest.mark.parametrize(

flexmeasures/api/v3_0/tests/utils.py

@@ -46,6 +46,7 @@ def get_sensor_post_data(generic_asset_id: int = 2) -> dict:
         "event_resolution": "PT1H",
         "unit": "kWh",
         "generic_asset_id": generic_asset_id,
+        "attributes": '{"capacity_in_mw": 0.0074, "max_soc_in_mwh": 0.04, "min_soc_in_mwh": 0.008}',
     }
     return post_data
 

flexmeasures/app.py

@@ -133,7 +133,7 @@ def create(  # noqa C901
     )  # use copy to avoid mutating app.reporters
     app.data_generators["scheduler"] = schedulers
 
-    # deprecation of app.reporters
+    # deprecated: app.reporters and app.schedulers
     app.reporters = reporters
     app.schedulers = schedulers
 
@@ -143,7 +143,7 @@ def get_reporters():
         )
         return app.data_generators["reporter"]
 
-    setattr(app, "reporters", property(get_reporters))
+    setattr(app, "reporters", get_reporters())
 
     # add auth policy
 
@@ -188,7 +188,7 @@ def before_request():
             try:
                 import pyinstrument  # noqa F401
 
-                g.profiler = pyinstrument.Profiler()
+                g.profiler = pyinstrument.Profiler(async_mode="disabled")
                 g.profiler.start()
             except ImportError:
                 pass

flexmeasures/data/schemas/sensors.py

@@ -1,5 +1,7 @@
 from marshmallow import Schema, fields, validates, ValidationError
 
+import json
+
 from flexmeasures.data import ma
 from flexmeasures.data.models.generic_assets import GenericAsset
 from flexmeasures.data.models.time_series import Sensor
@@ -12,6 +14,17 @@
 from flexmeasures.data.schemas.times import DurationField
 
 
+class JSON(fields.Field):
+    def _deserialize(self, value, attr, data, **kwargs) -> dict:
+        try:
+            return json.loads(value)
+        except ValueError:
+            raise ValidationError("Not a valid JSON string.")
+
+    def _serialize(self, value, attr, data, **kwargs) -> str:
+        return json.dumps(value)
+
+
 class SensorSchemaMixin(Schema):
     """
     Base sensor schema.
@@ -32,6 +45,7 @@ class Meta:
     timezone = ma.auto_field()
     event_resolution = DurationField(required=True)
     entity_address = fields.String(dump_only=True)
+    attributes = JSON(required=False)
 
     @validates("unit")
     def validate_unit(self, unit: str):

flexmeasures/ui/templates/base.html

@@ -265,13 +265,16 @@
     }
 
     function checkSourceMasking(data) {
+        var sourceWarn = document.getElementById("sourcewarn");
+        if (sourceWarn == null)
+            return
         var uniqueSourceIds = getUniqueValues(data, 'source.id');
         if (chartType == 'daily_heatmap' && uniqueSourceIds.length > 1) {
-            document.getElementById('sourcewarn').style.display = 'block';
-            document.getElementById('sourcewarn').innerHTML = 'Please note that only data from the most prevalent source is shown.';
+            sourceWarn.style.display = 'block';
+            sourceWarn.innerHTML = 'Please note that only data from the most prevalent source is shown.';
         }
         else {
-            document.getElementById('sourcewarn').style.display = 'none';
+            sourceWarn.style.display = 'none';
         }
     }
 

flexmeasures/utils/plugin_utils.py

@@ -16,7 +16,7 @@
 from flexmeasures.utils.coding_utils import get_classes_module
 
 
-def register_plugins(app: Flask):
+def register_plugins(app: Flask):  # noqa: C901
     """
     Register FlexMeasures plugins as Blueprints.
     This is configured by the config setting FLEXMEASURES_PLUGINS.
@@ -113,13 +113,14 @@ def register_plugins(app: Flask):
         plugin_reporters = get_classes_module(module.__name__, Reporter)
         plugin_schedulers = get_classes_module(module.__name__, Scheduler)
 
-        # for legacy, we keep reporters and schedulers
-        app.reporters.update(plugin_reporters)
-        app.schedulers.update(plugin_schedulers)
-
         # add DataGenerators
-        app.data_generators["scheduler"].update(plugin_schedulers)
-        app.data_generators["reporter"].update(plugin_reporters)
+        # for legacy, we still maintain app.reporters and app.schedulers
+        if plugin_reporters:
+            app.data_generators["reporter"].update(plugin_reporters)
+            app.reporters.update(plugin_reporters)
+        if plugin_schedulers:
+            app.data_generators["scheduler"].update(plugin_schedulers)
+            app.schedulers.update(plugin_schedulers)
 
         app.config["LOADED_PLUGINS"][plugin_name] = plugin_version
     app.logger.info(f"Loaded plugins: {app.config['LOADED_PLUGINS']}")

requirements/3.10/app.txt

@@ -75,7 +75,7 @@ flask==2.1.2
     #   flask-wtf
     #   rq-dashboard
     #   sentry-sdk
-flask-classful @ git+https://github.com/superseed/flask-classful@werkzeug-2.2
+flask-classful==0.14.2
     # via -r requirements/app.in
 flask-cors==4.0.0
     # via -r requirements/app.in

requirements/3.11/app.txt

@@ -73,7 +73,7 @@ flask==2.1.2
     #   flask-wtf
     #   rq-dashboard
     #   sentry-sdk
-flask-classful @ git+https://github.com/superseed/flask-classful@werkzeug-2.2
+flask-classful==0.14.2
     # via -r requirements/app.in
 flask-cors==4.0.0
     # via -r requirements/app.in

requirements/3.8/app.txt

@@ -77,7 +77,7 @@ flask==2.1.2
     #   flask-wtf
     #   rq-dashboard
     #   sentry-sdk
-flask-classful @ git+https://github.com/superseed/flask-classful@werkzeug-2.2
+flask-classful==0.14.2
     # via -r requirements/app.in
 flask-cors==4.0.0
     # via -r requirements/app.in

requirements/3.9/app.txt

@@ -75,7 +75,7 @@ flask==2.1.2
     #   flask-wtf
     #   rq-dashboard
     #   sentry-sdk
-flask-classful @ git+https://github.com/superseed/flask-classful@werkzeug-2.2
+flask-classful == 0.14.2
     # via -r requirements/app.in
 flask-cors==4.0.0
     # via -r requirements/app.in

requirements/app.in

@@ -52,7 +52,7 @@ Flask-Security-Too>=5.0, <5.2
 # On repeated call to API with auth token, user is anonymous.
 # See test_get_one_user or test_api_task_run_post_unauthorized_wrong_role for a good example
 Flask-Login<0.6.2
-Flask-Classful @ git+https://github.com/superseed/flask-classful@werkzeug-2.2
+Flask-Classful
 # <0.15: https://github.com/marshmallow-code/flask-marshmallow/issues/262
 Flask-Marshmallow<0.15
 Flask-Cors