Merge branch 'main' into 928-fix-discrepancies-between-db-and-models

Signed-off-by: GustaafL <41048720+GustaafL@users.noreply.github.com>

documentation/api/change_log.rst

@@ -5,6 +5,11 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL (e.g. `/api/v3_0`), allowing developers to upgrade at their own pace.
 
+v3.0-14 | 2023-12-07
+""""""""""""""""""""
+
+- Fix API version listing (GET /api/v3_0) for hosts running on Python 3.8.
+
 v3.0-13 | 2023-10-31
 """"""""""""""""""""
 

documentation/changelog.rst

@@ -11,28 +11,34 @@ v0.18.0 | December XX, 2023
 New features
 -------------
 
+* Better navigation experience through listings (sensors / assets / users / accounts) in the :abbr:`UI (user interface)`, by heading to the selected entity upon a click (or CTRL + click) anywhere within a row [see `PR #923 <https://github.com/FlexMeasures/flexmeasures/pull/923>`_]
+* New flexmeasures configuration setting `FLEXMEASURES_ENFORCE_SECURE_CONTENT_POLICY` for upgrading insecure `http` requests to secured requests `https` [see `PR #920 <https://github.com/FlexMeasures/flexmeasures/pull/920>`_]
+
 Infrastructure / Support
 ----------------------
 
 * Align database and models of `annotations`, `data_sources`, and `timed_belief` [see `PR #929 <https://github.com/FlexMeasures/flexmeasures/pull/929>`_]
 * Remove obsolete database tables `price`, `power`, `market`, `market_type`, `weather`, `asset`, and `weather_sensor` [see `PR #921 <https://github.com/FlexMeasures/flexmeasures/pull/921>`_]
 * New documentation section on constructing a flex model for :abbr:`V2G (vehicle-to-grid)` [see `PR #885 <https://github.com/FlexMeasures/flexmeasures/pull/885>`_]
+* Allow charts in plugins to show currency codes (such as EUR) as currency symbols (€) [see `PR #922 <https://github.com/FlexMeasures/flexmeasures/pull/922>`_]
 
 Bugfixes
 -----------
 
-* Show `Assets`, `Users`, `Tasks` and `Accounts` pages in the navigation bar for the `admin-reader` role [see `PR #900 <https://github.com/FlexMeasures/flexmeasures/pull/900>`_]
 * Give `admin-reader` role access to the RQ Scheduler dashboard [see `PR #901 <https://github.com/FlexMeasures/flexmeasures/pull/901>`_]
+* Assets without a geographical position (i.e. no lat/lng coordinates) can be edited through the UI [see `PR #924 <https://github.com/FlexMeasures/flexmeasures/pull/924>`_]
 
 
-v0.17.1 | November 22, 2023
+v0.17.1 | December 7, 2023
 ============================
 
 Bugfixes
 -----------
 
+* Show `Assets`, `Users`, `Tasks` and `Accounts` pages in the navigation bar for the `admin-reader` role [see `PR #900 <https://github.com/FlexMeasures/flexmeasures/pull/900>`_]
+* Reduce worker logs when datetime exceeds the end of the schedule [see `PR #918 <https://github.com/FlexMeasures/flexmeasures/pull/918>`_]
 * Fix infeasible problem due to incorrect estimation of the big-M value [see `PR #905 <https://github.com/FlexMeasures/flexmeasures/pull/905>`_]
-
+* Fix API version listing (GET /api/v3_0) for hosts running on Python 3.8 [see `PR #917 <https://github.com/FlexMeasures/flexmeasures/pull/917>`_]
 
 
 v0.17.0 | November 8, 2023

documentation/configuration.rst

@@ -104,7 +104,7 @@ FLEXMEASURES_PROFILE_REQUESTS
 
 If True, the processing time of requests are profiled.
 
-The overall time used by requests are logged to the console. In addiition, if `pyinstrument` is installed, then a profiling report is made (of time being spent in different function calls) for all Flask API endpoints.
+The overall time used by requests are logged to the console. In addition, if `pyinstrument` is installed, then a profiling report is made (of time being spent in different function calls) for all Flask API endpoints.
 
 The profiling results are stored in the ``profile_reports`` folder in the instance directory.
 
@@ -384,7 +384,9 @@ You can use this setting to overwrite that URI and point the tests to an (empty)
 Security
 --------
 
-This is only a selection of the most important settings.
+Settings to ensure secure handling of credentials and data.
+
+For Flask-Security and Flask-Cors (setting names start with "SECURITY" or "CORS"), this is only a selection of the most important settings.
 See `the Flask-Security Docs <https://flask-security-too.readthedocs.io/en/stable/configuration.html>`_ as well as the `Flask-CORS docs <https://flask-cors.readthedocs.io/en/latest/configuration.html>`_ for all possibilities.
 
 SECRET_KEY (**)
@@ -453,6 +455,13 @@ Allows users to make authenticated requests. If true, injects the Access-Control
 Default: ``True``
 
 
+FLEXMEASURES_ENFORCE_SECURE_CONTENT_POLICY
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When ``FLEXMEASURES_ENFORCE_SECURE_CONTENT_POLICY`` is set to ``True``, the ``<meta>`` tag with the ``Content-Security-Policy`` directive, specifically ``upgrade-insecure-requests``, is included in the HTML head. This directive instructs the browser to upgrade insecure requests from ``http`` to ``https``. One example of a use case for this is if you have a load balancer in front of FlexMeasures, which is secured with a certificate and only accepts https.
+
+Default: ``False``
+
 
 .. _mail-config:
 

flexmeasures/api/v3_0/public.py

@@ -31,7 +31,7 @@ def index(self):
                 methods: str = "/".join(
                     [m for m in rule.methods if m not in ("OPTIONS", "HEAD")]
                 )
-                stripped_url = url.removeprefix(self.route_base)
+                stripped_url = removeprefix(url, self.route_base)
                 full_url = (
                     request.url_root.removesuffix("/") + url
                     if url.startswith("/")
@@ -75,3 +75,15 @@ def quickref_directive(content):
             break
 
     return description
+
+
+def removeprefix(text: str, prefix: str) -> str:
+    """Remove a prefix from a text.
+
+    todo: use text.removeprefix(prefix) instead of this method, after dropping support for Python 3.8
+          See https://docs.python.org/3.9/library/stdtypes.html#str.removeprefix
+    """
+    if text.startswith(prefix):
+        return text[len(prefix) :]
+    else:
+        return text

flexmeasures/data/models/planning/storage.py

@@ -480,7 +480,6 @@ def ensure_soc_min_max(self):
 
 
 class StorageFallbackScheduler(MetaStorageScheduler):
-
     __version__ = "1"
     __author__ = "Seita"
 
@@ -630,6 +629,7 @@ def build_device_soc_values(
     if isinstance(soc_values, pd.Series):  # some tests prepare it this way
         device_values = soc_values
     else:
+        disregarded_datetimes = []
         device_values = initialize_series(
             np.nan,
             start=start_of_schedule,
@@ -645,14 +645,22 @@ def build_device_soc_values(
             )  # otherwise DST would be problematic
             if soc_datetime > end_of_schedule:
                 # Skip too-far-into-the-future target
+                disregarded_datetimes += [soc_datetime]
                 max_server_horizon = get_max_planning_horizon(resolution)
-                current_app.logger.warning(
-                    f"Disregarding target datetime {soc_datetime}, because it exceeds {end_of_schedule}. Maximum scheduling horizon is {max_server_horizon}."
-                )
                 continue
 
             device_values.loc[soc_datetime] = soc
 
+        if disregarded_datetimes:
+            if len(disregarded_datetimes) == 1:
+                current_app.logger.warning(
+                    f"Disregarding 1 target datetime {disregarded_datetimes[0]}, because it exceeds {end_of_schedule}. Maximum scheduling horizon is {max_server_horizon}."
+                )
+            else:
+                current_app.logger.warning(
+                    f"Disregarding {len(disregarded_datetimes)} target datetimes from {min(disregarded_datetimes)} until {max(disregarded_datetimes)}, because they exceed {end_of_schedule}. Maximum scheduling horizon is {max_server_horizon}."
+                )
+
         # soc_values are at the end of each time slot, while prices are indexed by the start of each time slot
         device_values = device_values[start_of_schedule + resolution : end_of_schedule]
 
@@ -904,7 +912,6 @@ def sanitize_expression(expression: str, columns: list) -> tuple[str, list]:
     columns_involved = []
 
     for column in columns:
-
         if re.search(get_pattern_match_word(column), _expression):
             columns_involved.append(column)
 

flexmeasures/data/models/planning/tests/test_solver.py

@@ -1,6 +1,7 @@
 from datetime import datetime, timedelta
 import pytest
 import pytz
+import logging
 
 import numpy as np
 import pandas as pd
@@ -13,6 +14,7 @@
     StorageScheduler,
     add_storage_constraints,
     validate_storage_constraints,
+    build_device_soc_values,
 )
 from flexmeasures.data.models.planning.linear_optimization import device_scheduler
 from flexmeasures.data.models.planning.tests.utils import check_constraints
@@ -1255,3 +1257,43 @@ def set_if_not_none(dictionary, key, value):
 
     assert all(ems_constraints["derivative min"] == expected_site_production_capacity)
     assert all(ems_constraints["derivative max"] == expected_site_consumption_capacity)
+
+
+@pytest.mark.parametrize(
+    ["soc_values", "log_message"],
+    [
+        (
+            [
+                {"datetime": datetime(2023, 5, 19, tzinfo=pytz.utc), "value": 1.0},
+                {"datetime": datetime(2023, 5, 22, tzinfo=pytz.utc), "value": 1.0},
+                {"datetime": datetime(2023, 5, 23, tzinfo=pytz.utc), "value": 1.0},
+                {"datetime": datetime(2023, 5, 21, tzinfo=pytz.utc), "value": 1.0},
+            ],
+            "Disregarding 3 target datetimes from 2023-05-21 00:00:00+00:00 until 2023-05-23 00:00:00+00:00, because they exceed 2023-05-20 00:00:00+00:00",
+        ),
+        (
+            [
+                {"datetime": datetime(2023, 5, 19, tzinfo=pytz.utc), "value": 1.0},
+                {"datetime": datetime(2023, 5, 23, tzinfo=pytz.utc), "value": 1.0},
+            ],
+            "Disregarding 1 target datetime 2023-05-23 00:00:00+00:00, because it exceeds 2023-05-20 00:00:00+00:00",
+        ),
+    ],
+)
+def test_build_device_soc_values(caplog, soc_values, log_message):
+    caplog.set_level(logging.WARNING)
+    soc_at_start = 3.0
+    start_of_schedule = datetime(2023, 5, 18, tzinfo=pytz.utc)
+    end_of_schedule = datetime(2023, 5, 20, tzinfo=pytz.utc)
+    resolution = timedelta(minutes=5)
+
+    with caplog.at_level(logging.WARNING):
+        device_values = build_device_soc_values(
+            soc_values=soc_values,
+            soc_at_start=soc_at_start,
+            start_of_schedule=start_of_schedule,
+            end_of_schedule=end_of_schedule,
+            resolution=resolution,
+        )
+    print(device_values)
+    assert log_message in caplog.text

flexmeasures/data/schemas/scheduling/storage.py

@@ -60,7 +60,6 @@ def __init__(self, *args, **kwargs):
 
     @validates("value")
     def validate_value(self, _value):
-
         if self.value_validator is not None:
             self.value_validator(_value)
 

flexmeasures/ui/crud/assets.py

@@ -8,7 +8,7 @@
 from flask_wtf import FlaskForm
 from flask_security import login_required, current_user
 from wtforms import StringField, DecimalField, SelectField
-from wtforms.validators import DataRequired
+from wtforms.validators import DataRequired, optional
 from flexmeasures.auth.policy import user_has_admin_access
 
 from flexmeasures.data import db
@@ -40,11 +40,13 @@ class AssetForm(FlaskForm):
     name = StringField("Name")
     latitude = DecimalField(
         "Latitude",
+        validators=[optional()],
         places=None,
         render_kw={"placeholder": "--Click the map or enter a latitude--"},
     )
     longitude = DecimalField(
         "Longitude",
+        validators=[optional()],
         places=None,
         render_kw={"placeholder": "--Click the map or enter a longitude--"},
     )
@@ -65,8 +67,10 @@ def validate_on_submit(self):
     def to_json(self) -> dict:
         """turn form data into a JSON we can POST to our internal API"""
         data = copy.copy(self.data)
-        data["longitude"] = float(data["longitude"])
-        data["latitude"] = float(data["latitude"])
+        if data.get("longitude") is not None:
+            data["longitude"] = float(data["longitude"])
+        if data.get("latitude") is not None:
+            data["latitude"] = float(data["latitude"])
 
         if "csrf_token" in data:
             del data["csrf_token"]

flexmeasures/ui/static/css/flexmeasures.css

@@ -556,6 +556,8 @@ i:not(.supersize):before, i:not(.supersize):after {
 /* icon on the left (use left-icon class) */
 :not(.map-icon) > i.left-icon {
     left: 40px;
+    /* Prevent user agent stylesheets from treating this tag as an italics tag */
+    font-style: normal;
 }
 :not(.map-icon) > i.left-icon:after, :not(.map-icon) > i.left-icon:before {
     position: absolute;
@@ -818,6 +820,13 @@ i.icon-wind:hover:before {
 
 /* --- Tables ---- */
 
+/* Clickable list items */
+
+table.dataTable.nav-on-click tbody tr:hover {
+    background-color: var(--nav-current-hover-background-color) !important;
+    cursor: pointer;
+}
+
 /* scrolling */
 .floatThead-wrapper table {
     width: 100%;