Merge remote-tracking branch 'origin/main' into sensor-matrix-chart

.vscode/spellright.dict

@@ -257,3 +257,8 @@ dataframe
 dataframes
 args
 docstrings
+Auth
+ctx_loader
+ctx_arg_name
+ctx_arg_pos
+dataset

documentation/api/change_log.rst

@@ -5,6 +5,12 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
+
+v3.0-12 | 2023-07-31
+"""""""""""""""""""
+
+- Added REST endpoint for adding a sensor: `/sensors` (POST)
+
 v3.0-11 | 2023-07-20
 """"""""""""""""""""
 

documentation/changelog.rst

@@ -8,17 +8,20 @@ v0.15.0 | July XX, 2023
 
 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
 
+.. warning:: If your server is running in play mode (``FLEXMEASURES_MODE = "play"``), users will be able to see sensor data from any account [see `PR #740 <https://www.github.com/FlexMeasures/flexmeasures/pull/740>`_].
+
 New features
 -------------
 
 * Users can select a new chart type (daily heatmap) on the sensor page of the UI, showing how sensor values are distributed over the time of day [see `PR #715 <https://www.github.com/FlexMeasures/flexmeasures/pull/715>`_]
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times [see `PR #734 <https://www.github.com/FlexMeasures/flexmeasures/pull/734>`_]
 * Make it a lot easier to read off the color legend on the asset page, especially when showing many sensors, as they will now be ordered from top to bottom in the same order as they appear in the chart (as defined in the ``sensors_to_show`` attribute), rather than alphabetically [see `PR #742 <https://www.github.com/FlexMeasures/flexmeasures/pull/742>`_]
+* Users on FlexMeasures servers in play mode (``FLEXMEASURES_MODE = "play"``) can use the ``sensors_to_show`` attribute to show any sensor on their asset pages, rather than only sensors registered to assets in their own account or to public assets [see `PR #740 <https://www.github.com/FlexMeasures/flexmeasures/pull/740>`_]
 * Having percentages within the [0, 100] domain is such a common use case that we now always include it in sensor charts with % units, making it easier to read off individual charts and also to compare across charts [see `PR #739 <https://www.github.com/FlexMeasures/flexmeasures/pull/739>`_]
 * DataSource table now allows storing arbitrary attributes as a JSON (without content validation), similar to the Sensor and GenericAsset tables [see `PR #750 <https://www.github.com/FlexMeasures/flexmeasures/pull/750>`_]
-* Added API endpoint `/sensor/<id>` for fetching a single sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_]
+* Added API endpoints `/sensors/<id>` for fetching a single sensor and `/sensors` (POST) for adding a sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_] and [see `PR #767 <https://www.github.com/FlexMeasures/flexmeasures/pull/767>`_]
 * The CLI now allows to set lists and dicts as asset & sensor attributes (formerly only single values) [see `PR #762 <https://www.github.com/FlexMeasures/flexmeasures/pull/762>`_]
-* Add `ProcessScheduler` class, which optimizes the starting time of processes using one of the following policies: INFLEXIBLE, SHIFTABLE and BREAKABLE [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_]
+* Add `ProcessScheduler` class to optimize the starting time of processes one of the policies developed (INFLEXIBLE, SHIFTABLE and BREAKABLE), accessible via the CLI command `flexmeasures add schedule for-process` [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_ and `PR #768 <https://www.github.com/FlexMeasures/flexmeasures/pull/768>`_]
 
 Bugfixes
 -----------

documentation/cli/change_log.rst

@@ -8,6 +8,8 @@ since v0.15.0 | July XX, 2023
 =================================
 
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times.
+* Add ``flexmeasures add schedule for-process`` to create a new process schedule for a given power sensor.
+
 
 since v0.14.1 | June XX, 2023
 =================================

documentation/cli/commands.rst

@@ -36,6 +36,7 @@ of which some are referred to in this documentation.
 ``flexmeasures add source``                       Add a new data source.
 ``flexmeasures add forecasts``                    Create forecasts.
 ``flexmeasures add schedule for-storage``         Create a charging schedule for a storage asset.
+``flexmeasures add schedule for-process``         Create a schedule for a process asset.
 ``flexmeasures add holidays``                     Add holiday annotations to accounts and/or assets.
 ``flexmeasures add annotation``                   Add annotation to accounts, assets and/or sensors.
 ``flexmeasures add toy-account``                  Create a toy account, for tutorials and trying things.

documentation/dev/auth.rst

@@ -30,7 +30,7 @@ You, as the endpoint author, need to make sure this is checked. Here is an examp
         {"the_resource": ResourceIdField(data_key="resource_id")},
         location="path",
     )
-    @permission_required_for_context("read", arg_name="the_resource")
+    @permission_required_for_context("read", ctx_arg_name="the_resource")
     @as_json
     def view(resource_id: int, resource: Resource):
         return dict(name=resource.name)

documentation/host/modes.rst

@@ -39,3 +39,4 @@ Small features
 - [API] Posted UDI events are not enforced to be consecutive.
 - [API] Names in ``GetConnectionResponse`` are the connections' unique database names rather than their display names (this feature is planned to be deprecated).
 - [UI] The dashboard plot showing the latest power value is not enforced to lie in the past (in case of simulating future values).
+- [UI] On the asset page, the ``sensors_to_show`` attribute can be used to show any sensor from any account, rather than only sensors from assets owned by the user's organization.

flexmeasures/api/dev/sensors.py

@@ -51,7 +51,7 @@ class SensorAPI(FlaskView):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="sensor")
+    @permission_required_for_context("read", ctx_arg_name="sensor")
     def get_chart(self, id: int, sensor: Sensor, **kwargs):
         """GET from /sensor/<id>/chart
 
@@ -88,7 +88,7 @@ def get_chart(self, id: int, sensor: Sensor, **kwargs):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="sensor")
+    @permission_required_for_context("read", ctx_arg_name="sensor")
     def get_chart_data(self, id: int, sensor: Sensor, **kwargs):
         """GET from /sensor/<id>/chart_data
 
@@ -121,7 +121,7 @@ def get_chart_data(self, id: int, sensor: Sensor, **kwargs):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="sensor")
+    @permission_required_for_context("read", ctx_arg_name="sensor")
     def get_chart_annotations(self, id: int, sensor: Sensor, **kwargs):
         """GET from /sensor/<id>/chart_annotations
 
@@ -150,7 +150,7 @@ def get_chart_annotations(self, id: int, sensor: Sensor, **kwargs):
         {"sensor": SensorIdField(data_key="id")},
         location="path",
     )
-    @permission_required_for_context("read", arg_name="sensor")
+    @permission_required_for_context("read", ctx_arg_name="sensor")
     def get(self, id: int, sensor: Sensor):
         """GET from /sensor/<id>
 
@@ -173,7 +173,7 @@ class AssetAPI(FlaskView):
         {"asset": AssetIdField(data_key="id")},
         location="path",
     )
-    @permission_required_for_context("read", arg_name="asset")
+    @permission_required_for_context("read", ctx_arg_name="asset")
     def get(self, id: int, asset: GenericAsset):
         """GET from /asset/<id>
 

flexmeasures/api/dev/tests/conftest.py

@@ -1,5 +1,6 @@
 import pytest
 
+from flexmeasures import User
 from flexmeasures.api.v3_0.tests.conftest import add_incineration_line
 from flexmeasures.data.models.time_series import Sensor
 
@@ -10,7 +11,7 @@ def setup_api_test_data(db, setup_roles_users, setup_generic_assets):
     Set up data for API dev tests.
     """
     print("Setting up data for API dev tests on %s" % db.engine)
-    add_incineration_line(db, setup_roles_users["Test Supplier User"])
+    add_incineration_line(db, User.query.get(setup_roles_users["Test Supplier User"]))
 
 
 @pytest.fixture(scope="function")
@@ -23,4 +24,6 @@ def setup_api_fresh_test_data(
     print("Setting up fresh data for API dev tests on %s" % fresh_db.engine)
     for sensor in Sensor.query.all():
         fresh_db.delete(sensor)
-    add_incineration_line(fresh_db, setup_roles_users_fresh_db["Test Supplier User"])
+    add_incineration_line(
+        fresh_db, User.query.get(setup_roles_users_fresh_db["Test Supplier User"])
+    )

flexmeasures/api/v3_0/accounts.py

@@ -70,7 +70,7 @@ def index(self):
 
     @route("/<id>", methods=["GET"])
     @use_kwargs({"account": AccountIdField(data_key="id")}, location="path")
-    @permission_required_for_context("read", arg_name="account")
+    @permission_required_for_context("read", ctx_arg_name="account")
     @as_json
     def get(self, id: int, account: Account):
         """API endpoint to get an account.

flexmeasures/api/v3_0/assets.py

@@ -42,7 +42,7 @@ class AssetAPI(FlaskView):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="account")
+    @permission_required_for_context("read", ctx_arg_name="account")
     @as_json
     def index(self, account: Account):
         """List all assets owned by a certain account.
@@ -103,7 +103,7 @@ def public(self):
 
     @route("", methods=["POST"])
     @permission_required_for_context(
-        "create-children", arg_loader=AccountIdField.load_current
+        "create-children", ctx_loader=AccountIdField.load_current
     )
     @use_args(asset_schema)
     def post(self, asset_data: dict):
@@ -144,7 +144,7 @@ def post(self, asset_data: dict):
 
     @route("/<id>", methods=["GET"])
     @use_kwargs({"asset": AssetIdField(data_key="id")}, location="path")
-    @permission_required_for_context("read", arg_name="asset")
+    @permission_required_for_context("read", ctx_arg_name="asset")
     @as_json
     def fetch_one(self, id, asset):
         """Fetch a given asset.
@@ -180,7 +180,7 @@ def fetch_one(self, id, asset):
     @route("/<id>", methods=["PATCH"])
     @use_args(partial_asset_schema)
     @use_kwargs({"db_asset": AssetIdField(data_key="id")}, location="path")
-    @permission_required_for_context("update", arg_name="db_asset")
+    @permission_required_for_context("update", ctx_arg_name="db_asset")
     @as_json
     def patch(self, asset_data: dict, id: int, db_asset: GenericAsset):
         """Update an asset given its identifier.
@@ -236,7 +236,7 @@ def patch(self, asset_data: dict, id: int, db_asset: GenericAsset):
 
     @route("/<id>", methods=["DELETE"])
     @use_kwargs({"asset": AssetIdField(data_key="id")}, location="path")
-    @permission_required_for_context("delete", arg_name="asset")
+    @permission_required_for_context("delete", ctx_arg_name="asset")
     @as_json
     def delete(self, id: int, asset: GenericAsset):
         """Delete an asset given its identifier.
@@ -278,7 +278,7 @@ def delete(self, id: int, asset: GenericAsset):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="asset")
+    @permission_required_for_context("read", ctx_arg_name="asset")
     def get_chart(self, id: int, asset: GenericAsset, **kwargs):
         """GET from /assets/<id>/chart
 
@@ -303,7 +303,7 @@ def get_chart(self, id: int, asset: GenericAsset, **kwargs):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="asset")
+    @permission_required_for_context("read", ctx_arg_name="asset")
     def get_chart_data(self, id: int, asset: GenericAsset, **kwargs):
         """GET from /assets/<id>/chart_data
 

flexmeasures/api/v3_0/sensors.py

@@ -30,6 +30,7 @@
 from flexmeasures.auth.decorators import permission_required_for_context
 from flexmeasures.data import db
 from flexmeasures.data.models.user import Account
+from flexmeasures.data.models.generic_assets import GenericAsset
 from flexmeasures.data.models.time_series import Sensor
 from flexmeasures.data.queries.utils import simplify_index
 from flexmeasures.data.schemas.sensors import SensorSchema, SensorIdField
@@ -64,7 +65,7 @@ class SensorAPI(FlaskView):
         },
         location="query",
     )
-    @permission_required_for_context("read", arg_name="account")
+    @permission_required_for_context("read", ctx_arg_name="account")
     @as_json
     def index(self, account: Account):
         """API endpoint to list all sensors of an account.
@@ -498,7 +499,7 @@ def get_schedule(self, sensor: Sensor, job_id: str, duration: timedelta, **kwarg
 
     @route("/<id>", methods=["GET"])
     @use_kwargs({"sensor": SensorIdField(data_key="id")}, location="path")
-    @permission_required_for_context("read", arg_name="sensor")
+    @permission_required_for_context("read", ctx_arg_name="sensor")
     @as_json
     def fetch_one(self, id, sensor):
         """Fetch a given sensor.
@@ -515,7 +516,7 @@ def fetch_one(self, id, sensor):
                 "name": "some gas sensor",
                 "unit": "m³/h",
                 "entity_address": "ea1.2023-08.localhost:fm1.1",
-                "event_resolution": 10,
+                "event_resolution": "PT10M",
                 "generic_asset_id": 4,
                 "timezone": "UTC",
             }
@@ -529,4 +530,49 @@ def fetch_one(self, id, sensor):
         :status 403: INVALID_SENDER
         :status 422: UNPROCESSABLE_ENTITY
         """
+
         return sensor_schema.dump(sensor), 200
+
+    @route("", methods=["POST"])
+    @use_args(sensor_schema)
+    @permission_required_for_context(
+        "create-children",
+        ctx_arg_pos=1,
+        ctx_arg_name="generic_asset_id",
+        ctx_loader=GenericAsset,
+        pass_ctx_to_loader=True,
+    )
+    def post(self, sensor_data: dict):
+        """Create new asset.
+
+        .. :quickref: Sensor; Create a new Sensor
+
+        This endpoint creates a new Sensor.
+
+        **Example request**
+
+        .. sourcecode:: json
+
+            {
+                "name": "power",
+                "event_resolution": "PT1H",
+                "unit": "kWh",
+                "generic_asset_id": 1,
+            }
+
+
+        The newly posted sensor is returned in the response.
+
+        :reqheader Authorization: The authentication token
+        :reqheader Content-Type: application/json
+        :resheader Content-Type: application/json
+        :status 201: CREATED
+        :status 400: INVALID_REQUEST
+        :status 401: UNAUTHORIZED
+        :status 403: INVALID_SENDER
+        :status 422: UNPROCESSABLE_ENTITY
+        """
+        sensor = Sensor(**sensor_data)
+        db.session.add(sensor)
+        db.session.commit()
+        return sensor_schema.dump(sensor), 201

flexmeasures/api/v3_0/tests/conftest.py

@@ -17,7 +17,9 @@ def setup_api_test_data(
     Set up data for API v3.0 tests.
     """
     print("Setting up data for API v3.0 tests on %s" % db.engine)
-    sensors = add_incineration_line(db, setup_roles_users["Test Supplier User"])
+    sensors = add_incineration_line(
+        db, User.query.get(setup_roles_users["Test Supplier User"])
+    )
     return sensors
 
 
@@ -32,7 +34,7 @@ def setup_api_fresh_test_data(
     for sensor in Sensor.query.all():
         fresh_db.delete(sensor)
     sensors = add_incineration_line(
-        fresh_db, setup_roles_users_fresh_db["Test Supplier User"]
+        fresh_db, User.query.get(setup_roles_users_fresh_db["Test Supplier User"])
     )
     return sensors
 

flexmeasures/api/v3_0/tests/test_sensor_data.py

@@ -44,7 +44,9 @@ def test_get_sensor_data(
 ):
     """Check the /sensors/data endpoint for fetching 1 hour of data of a 10-minute resolution sensor."""
     sensor = setup_api_test_data["some gas sensor"]
-    source: Source = setup_roles_users["Test Supplier User"].data_source[0]
+    source: Source = User.query.get(
+        setup_roles_users["Test Supplier User"]
+    ).data_source[0]
     assert sensor.event_resolution == timedelta(minutes=10)
     message = {
         "sensor": f"ea1.2021-01.io.flexmeasures:fm1.{sensor.id}",
@@ -76,7 +78,9 @@ def test_get_instantaneous_sensor_data(
 ):
     """Check the /sensors/data endpoint for fetching 1 hour of data of an instantaneous sensor."""
     sensor = setup_api_test_data["some temperature sensor"]
-    source: Source = setup_roles_users["Test Supplier User"].data_source[0]
+    source: Source = User.query.get(
+        setup_roles_users["Test Supplier User"]
+    ).data_source[0]
     assert sensor.event_resolution == timedelta(minutes=0)
     message = {
         "sensor": f"ea1.2021-01.io.flexmeasures:fm1.{sensor.id}",