Merge remote-tracking branch 'origin/main' into 954-document-how-to-l…

…oad-plugins-into-docker-container

NOTICE

@@ -1,3 +1,3 @@
 FlexMeasures
-Copyright 2018-2022 Seita Energy Flexibility
+Copyright 2018-2024 Seita Energy Flexibility
 Apache 2.0 license

documentation/changelog.rst

@@ -21,23 +21,34 @@ Bugfixes
 -----------
 
 
-v0.18.1 | January XX, 2024
+v0.18.2 | January xx, 2024
 ============================
 
+* Automatically update table navigation in the UI without requiring users to hard refresh their browser [see `PR #961 <https://github.com/FlexMeasures/flexmeasures/pull/961>`_]
+
+
+v0.18.1 | January 15, 2024
+============================
+
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+
 Bugfixes
 -----------
 
+* Fix database migrations meant to clean up deprecated tables [see `PR #960 <https://github.com/FlexMeasures/flexmeasures/pull/960>`_]
 * Allow showing beliefs (plot and file export) via the CLI for sensors with non-unique names [see `PR #947 <https://github.com/FlexMeasures/flexmeasures/pull/947>`_]
 * Added Redis credentials to the Docker Compose configuration for the web server to ensure proper interaction with the Redis queue [see `PR #945 <https://github.com/FlexMeasures/flexmeasures/pull/945>`_]
 * Fix API version listing (GET /api/v3_0) for hosts running on Python 3.8 [see `PR #917 <https://github.com/FlexMeasures/flexmeasures/pull/917>`_ and `PR #950 <https://github.com/FlexMeasures/flexmeasures/pull/950>`_]
+* Fix the validation of the option ``--parent-asset`` of command ``flexmeasures add asset`` [see `PR #959 <https://github.com/FlexMeasures/flexmeasures/pull/959>`_]
+
 
 
 v0.18.0 | December 23, 2023
 ============================
 
 .. note:: Read more on these features on `the FlexMeasures blog <https://flexmeasures.io/018-better-use-of-future-knowledge/>`__.
 
-.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``). If this fails, update to ``flexmeasures==0.18.1`` first (and then run ``flexmeasures db upgrade``).
 
 New features
 -------------
@@ -46,7 +57,7 @@ New features
 * Introduce a breadcrumb to navigate through assets and sensor pages using its child-parent relationship [see `PR #930 <https://github.com/FlexMeasures/flexmeasures/pull/930>`_]
 * Define device-level power constraints as sensors to create schedules with changing power limits. [see `PR #897 <https://github.com/FlexMeasures/flexmeasures/pull/897>`_]
 * Allow to provide external storage usage or gain components using the ``soc-usage`` and ``soc-gain`` fields of the `flex-model` [see `PR #906 <https://github.com/FlexMeasures/flexmeasures/pull/906>`_]
-* Define time-varying charging and discharging efficiencies as sensors or as constant values which allows to define the :COP:`Coefficient of Performance`  [see `PR #933 <https://github.com/FlexMeasures/flexmeasures/pull/933>`_].
+* Define time-varying charging and discharging efficiencies as sensors or as constant values which allows to define the :abbr:`COP (coefficient of performance)` [see `PR #933 <https://github.com/FlexMeasures/flexmeasures/pull/933>`_]
 
 Infrastructure / Support
 ----------------------
@@ -94,7 +105,7 @@ Infrastructure / Support
 ----------------------
 
 - Introduce a new one-to-many relation between assets, allowing the definition of an asset's parent (which is also an asset). This hierarchical relationship enables assets to be related in a structured manner. [see `PR #855 <https://github.com/FlexMeasures/flexmeasures/pull/855>`_ and `PR #874 <https://github.com/FlexMeasures/flexmeasures/pull/874>`_]
-- Introduce a new format for the output of ``Scheduler`` to prepare for multiple outputs [see `PR #879 <https://github.com/FlexMeasures/flexmeasures/pull/879>`_].
+- Introduce a new format for the output of ``Scheduler`` to prepare for multiple outputs [see `PR #879 <https://github.com/FlexMeasures/flexmeasures/pull/879>`_]
 
 
 v0.16.1 | October 2, 2023

documentation/cli/change_log.rst

@@ -4,11 +4,18 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v0.18.1 | January 15, 2023
+=======================================
+
+* Fix the validation of the option ``--parent-asset`` of command ``flexmeasures add asset``.
+
+
 since v0.17.0 | November 8, 2023
 =======================================
 
 * Add ``--consultancy`` option to ``flexmeasures add account`` to create a consultancy relationship with another account.
 
+
 since v0.16.0 | September 29, 2023
 =======================================
 

flexmeasures/cli/data_add.py

@@ -67,7 +67,6 @@
 from flexmeasures.data.schemas.generic_assets import (
     GenericAssetSchema,
     GenericAssetTypeSchema,
-    GenericAssetIdField,
 )
 from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
 from flexmeasures.data.models.user import User
@@ -331,10 +330,10 @@ def add_sensor(**args):
     type=str,
     help="Description (useful to explain acronyms, for example).",
 )
-def add_asset_type(**args):
+def add_asset_type(**kwargs):
     """Add an asset type."""
-    check_errors(GenericAssetTypeSchema().validate(args))
-    generic_asset_type = GenericAssetType(**args)
+    check_errors(GenericAssetTypeSchema().validate(kwargs))
+    generic_asset_type = GenericAssetType(**kwargs)
     db.session.add(generic_asset_type)
     db.session.commit()
     click.secho(
@@ -372,9 +371,9 @@ def add_asset_type(**args):
 )
 @click.option(
     "--parent-asset",
-    "parent_asset",
+    "parent_asset_id",
     required=False,
-    type=GenericAssetIdField(),
+    type=int,
     help="Parent of this asset. The entity needs to exists on the database.",
 )
 def add_asset(**args):

flexmeasures/data/migrations/versions/ad98460751d9_remove_obsolete_tables.py

@@ -41,6 +41,7 @@ def upgrade():
         except ProgrammingError as exception:
             print(f"Table {table} not found, continuing...")
             print(exception)
+    db.session.close()  # https://stackoverflow.com/a/26346280/13775459
 
     if tables_with_data:
         click.confirm(

flexmeasures/data/migrations/versions/c349f52c700d_update_data_sources.py

@@ -18,6 +18,9 @@
 
 def upgrade():
     # The name of the data_source should be 120 String, this was not correctly set in an earlier revision of the db.
+    op.execute(
+        sa.text("UPDATE data_source SET attributes = '{}' WHERE attributes IS NULL;")
+    )
     with op.batch_alter_table("data_source", schema=None) as batch_op:
         batch_op.alter_column(
             "name",
@@ -34,15 +37,21 @@ def upgrade():
         )
 
     # This constraint is renamed to include the full name of the `data_source` table.
-    with op.batch_alter_table("timed_belief", schema=None) as batch_op:
-        batch_op.drop_constraint(
-            "timed_belief_source_id_source_fkey", type_="foreignkey"
-        )
-        batch_op.create_foreign_key(
-            batch_op.f("timed_belief_source_id_data_source_fkey"),
-            "data_source",
-            ["source_id"],
-            ["id"],
+    if foreign_key_exists("timed_belief", "timed_belief_source_id_source_fkey"):
+        with op.batch_alter_table("timed_belief", schema=None) as batch_op:
+            batch_op.drop_constraint(
+                "timed_belief_source_id_source_fkey", type_="foreignkey"
+            )
+            batch_op.create_foreign_key(
+                batch_op.f("timed_belief_source_id_data_source_fkey"),
+                "data_source",
+                ["source_id"],
+                ["id"],
+            )
+    else:
+        # already renamed
+        assert foreign_key_exists(
+            "timed_belief", "timed_belief_source_id_data_source_fkey"
         )
 
 
@@ -71,3 +80,17 @@ def downgrade():
             type_=sa.VARCHAR(length=80),
             existing_nullable=False,
         )
+
+
+def foreign_key_exists(table_name, fk_name) -> bool:
+    # Get the current connection
+    connection = op.get_bind()
+
+    # Create an Inspector
+    insp = sa.inspect(connection)
+
+    # Get the foreign keys for the specified table
+    foreign_keys = insp.get_foreign_keys(table_name)
+
+    # Check if the specified foreign key name exists
+    return any(fk["name"] == fk_name for fk in foreign_keys)

flexmeasures/data/schemas/generic_assets.py

@@ -71,6 +71,15 @@ def validate_generic_asset_type(self, generic_asset_type_id: int):
                 f"GenericAssetType with id {generic_asset_type_id} doesn't exist."
             )
 
+    @validates("parent_asset_id")
+    def validate_parent_asset(self, parent_asset_id: int | None):
+        if parent_asset_id is not None:
+            parent_asset = GenericAsset.query.get(parent_asset_id)
+            if not parent_asset:
+                raise ValidationError(
+                    f"Parent GenericAsset with id {parent_asset_id} doesn't exist."
+                )
+
     @validates("account_id")
     def validate_account(self, account_id: int | None):
         if account_id is None and (

flexmeasures/ui/templates/base.html

@@ -33,7 +33,7 @@
     <link href="https://cdn.jsdelivr.net/npm/ion-rangeslider@2.3.1/css/ion.rangeSlider.css"
         rel="stylesheet" />
     <!-- Custom CSS -->
-    <link href="{{ url_for('flexmeasures_ui.static', filename='css/flexmeasures.css') }}" rel="stylesheet" />
+    <link href="{{ url_for('flexmeasures_ui.static', filename='css/flexmeasures.css') }}?v={{ flexmeasures_version }}" rel="stylesheet" />
     {% if active_page == "tasks" %}
     <link href="{{ url_for('rq_dashboard.static', filename='css/main.css') }}" rel="stylesheet">
     <link href="{{ url_for('flexmeasures_ui.static', filename='css/external/rq-dashboard-bootstrap.min.css') }}" rel="stylesheet" />
@@ -660,7 +660,7 @@
 
     <!-- Custom scripts -->
     <script src="{{ url_for('flexmeasures_ui.static', filename='js/swiped-events.min.js') }}"></script>
-    <script src="{{ url_for('flexmeasures_ui.static', filename='js/flexmeasures.js') }}"></script>
+    <script src="{{ url_for('flexmeasures_ui.static', filename='js/flexmeasures.js') }}?v={{ flexmeasures_version }}"></script>
 
     {% endblock scripts %}