Merge remote-tracking branch 'origin/feature/reporting/add-attributes…

…-column-data-source' into feature/reporting/add-attributes-column-data-source

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,2 +1,27 @@
+## Description
+
+Summary of the changes introduced in this PR. Try to use bullet points as much as possible.
+
+## Look & Feel
+
+This section can contain example pictures for UI, Input/Output for CLI, Request / Response for API endpoint, etc.
+
+## How to test
+
+Steps to test it or name of the tests functions.
+
+The library [flexmeasures-client](https://github.com/FlexMeasures/flexmeasures-client/) can be useful to showcase new features. For example,
+it can be used to set some example data to be used in a new UI feature.
+
+## Further Improvements
+
+Potential improvements to be done in the same PR or follow up Issues/Discussions/PRs.
+
+## Related Items
+
+Mention if this PR closes an Issue or Project.
+
+---
+
 - [ ] I agree to contribute to the project under Apache 2 License. 
-- [ ] To the best of my knowledge, the proposed patch is not based on a code under GPL or other license that is incompatible with FlexMeasures
+- [ ] To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

.gitignore

@@ -37,3 +37,5 @@ db_schema.png
 
 .coverage
 htmlcov
+test/*
+profile_reports/*

documentation/changelog.rst

@@ -9,20 +9,32 @@ v0.15.0 | July XX, 2023
 New features
 -------------
 
+* Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times [see `PR #734 <https://www.github.com/FlexMeasures/flexmeasures/pull/734>`_]
+* Make it a lot easier to read off the color legend on the asset page, especially when showing many sensors, as they will now be ordered from top to bottom in the same order as they appear in the chart (as defined in the ``sensors_to_show`` attribute), rather than alphabetically [see `PR #742 <https://www.github.com/FlexMeasures/flexmeasures/pull/742>`_]
+* Having percentages within the [0, 100] domain is such a common use case that we now always include it in sensor charts with % units, making it easier to read off individual charts and also to compare across charts [see `PR #739 <https://www.github.com/FlexMeasures/flexmeasures/pull/739>`_]
+
 Bugfixes
 -----------
 
 Infrastructure / Support
 ----------------------
 
+* Add support for profiling Flask API calls using ``pyinstrument`` (if installed). Can be enabled by setting the environment variable ``FLEXMEASURES_PROFILE_REQUESTS`` to ``True`` [see `PR #722 <https://www.github.com/FlexMeasures/flexmeasures/pull/722>`_]
+
 
-v0.14.1 | June XX, 2023
+v0.14.1 | June 26, 2023
 ============================
 
 Bugfixes
 -----------
 
 * Relax constraint validation of `StorageScheduler` to accommodate violations caused by floating point precision [see `PR #731 <https://www.github.com/FlexMeasures/flexmeasures/pull/731>`_]
+* Avoid saving any :abbr:`NaN (not a number)` values to the database, when calling ``flexmeasures add report`` [see `PR #735 <https://www.github.com/FlexMeasures/flexmeasures/pull/735>`_]
+* Fix browser console error when loading asset or sensor page with only a single data point [see `PR #732 <https://www.github.com/FlexMeasures/flexmeasures/pull/732>`_]
+* Fix showing multiple sensors with bare 3-letter currency code as their units (e.g. EUR) in one chart [see `PR #738 <https://www.github.com/FlexMeasures/flexmeasures/pull/738>`_]
+* Fix defaults for the ``--start-offset`` and ``--end-offset`` options to ``flexmeasures add report``, which weren't being interpreted in the local timezone of the reporting sensor [see `PR #744 <https://www.github.com/FlexMeasures/flexmeasures/pull/744>`_]
+* Relax constraint for overlaying plot traces for sensors with various resolutions, making it possible to show e.g. two price sensors in one chart, where one of them records hourly prices and the other records quarter-hourly prices [see `PR #743 <https://www.github.com/FlexMeasures/flexmeasures/pull/743>`_]
+* Resolve bug where different page loads would potentially influence the time axis of each other's charts, by avoiding mutation of shared field definitions [see `PR #746 <https://www.github.com/FlexMeasures/flexmeasures/pull/746>`_]
 
 
 v0.14.0 | June 15, 2023

documentation/cli/change_log.rst

@@ -4,6 +4,17 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v0.15.0 | July XX, 2023
+=================================
+
+* Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times.
+
+since v0.14.1 | June XX, 2023
+=================================
+
+* Avoid saving any :abbr:`NaN (not a number)` values to the database, when calling ``flexmeasures add report``.
+* Fix defaults for the ``--start-offset`` and ``--end-offset` options to ``flexmeasures add report``, which weren't being interpreted in the local timezone of the reporting sensor.
+
 since v0.14.0 | June 15, 2023
 =================================
 

documentation/configuration.rst

@@ -98,7 +98,15 @@ Default: ``"migrations/dumps"``
 FLEXMEASURES_PROFILE_REQUESTS
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Whether to turn on a feature which times requests made through FlexMeasures. Interesting for developers.
+If True, the processing time of requests are profiled.
+
+The overall time used by requests are logged to the console. In addiition, if `pyinstrument` is installed, then a profiling report is made (of time being spent in different function calls) for all Flask API endpoints.
+
+The profiling results are stored in the ``profile_reports`` folder in the instance directory.
+
+Note: Profile reports for API endpoints are overwritten on repetition of the same request.
+
+Interesting for developers.
 
 Default: ``False``
 

flexmeasures/app.py

@@ -5,6 +5,9 @@
 from __future__ import annotations
 
 import time
+import os
+from pathlib import Path
+from datetime import date
 
 from flask import Flask, g, request
 from flask.cli import load_dotenv
@@ -97,6 +100,17 @@ def create(  # noqa C901
     if app.env not in ("documentation", "development"):
         SSLify(app)
 
+    # Prepare profiling, if needed
+
+    if app.config.get("FLEXMEASURES_PROFILE_REQUESTS", False):
+        Path("profile_reports").mkdir(parents=True, exist_ok=True)
+        try:
+            import pyinstrument  # noqa F401
+        except ImportError:
+            app.logger.warning(
+                "FLEXMEASURES_PROFILE_REQUESTS is True, but pyinstrument not installed ― I cannot produce profiling reports for requests."
+            )
+
     # Register database and models, including user auth security handlers
 
     from flexmeasures.data import register_at as register_db_at
@@ -150,6 +164,13 @@ def create(  # noqa C901
     def before_request():
         if app.config.get("FLEXMEASURES_PROFILE_REQUESTS", False):
             g.start = time.time()
+            try:
+                import pyinstrument  # noqa F401
+
+                g.profiler = pyinstrument.Profiler()
+                g.profiler.start()
+            except ImportError:
+                pass
 
     @app.teardown_request
     def teardown_request(exception=None):
@@ -159,5 +180,22 @@ def teardown_request(exception=None):
                 app.logger.info(
                     f"[PROFILE] {str(round(diff, 2)).rjust(6)} seconds to serve {request.url}."
                 )
+                if not hasattr(g, "profiler"):
+                    return app
+                g.profiler.stop()
+                output_html = g.profiler.output_html(timeline=True)
+                endpoint = request.endpoint
+                if endpoint is None:
+                    endpoint = "unknown"
+                today = date.today()
+                profile_filename = f"pyinstrument_{endpoint}.html"
+                profile_output_path = Path(
+                    "profile_reports", today.strftime("%Y-%m-%d")
+                )
+                profile_output_path.mkdir(parents=True, exist_ok=True)
+                with open(
+                    os.path.join(profile_output_path, profile_filename), "w+"
+                ) as f:
+                    f.write(output_html)
 
     return app

flexmeasures/cli/data_add.py

@@ -5,7 +5,7 @@
 from __future__ import annotations
 
 from datetime import datetime, timedelta
-from typing import Optional, Type
+from typing import Type
 import json
 from pathlib import Path
 from io import TextIOBase
@@ -66,7 +66,7 @@
 )
 from flexmeasures.data.services.utils import get_or_create_model
 from flexmeasures.utils import flexmeasures_inflection
-from flexmeasures.utils.time_utils import server_now, get_timezone, apply_offset_chain
+from flexmeasures.utils.time_utils import server_now, apply_offset_chain
 from flexmeasures.utils.unit_utils import convert_units, ur
 from flexmeasures.data.utils import save_to_db
 from flexmeasures.data.models.reporting import Reporter
@@ -374,7 +374,7 @@ def add_source(name: str, model: str, version: str, source_type: str):
     "sensor",
     required=True,
     type=SensorIdField(),
-    help="Sensor to which the beliefs pertain.",
+    help="Record the beliefs under this sensor. Follow up with the sensor's ID. ",
 )
 @click.option(
     "--source",
@@ -1165,7 +1165,7 @@ def add_schedule_for_storage(
     "sensor",
     type=SensorIdField(),
     required=True,
-    help="ID of the sensor used to save the report."
+    help="Sensor used to save the report. Follow up with the sensor's ID. "
     " If needed, use `flexmeasures add sensor` to create a new sensor first.",
 )
 @click.option(
@@ -1230,8 +1230,7 @@ def add_schedule_for_storage(
     "--timezone",
     "timezone",
     required=False,
-    default="UTC",
-    help="Timezone as string, e.g. 'UTC' or 'Europe/Amsterdam' (defaults to FLEXMEASURES_TIMEZONE config setting)",
+    help="Timezone as string, e.g. 'UTC' or 'Europe/Amsterdam' (defaults to the timezone of the sensor used to save the report).",
 )
 @click.option(
     "--dry-run",
@@ -1243,26 +1242,26 @@ def add_report(  # noqa: C901
     reporter_class: str,
     sensor: Sensor,
     reporter_config: TextIOBase,
-    start: Optional[datetime] = None,
-    end: Optional[datetime] = None,
-    start_offset: Optional[str] = None,
-    end_offset: Optional[str] = None,
-    resolution: Optional[timedelta] = None,
-    output_file: Optional[Path] = None,
+    start: datetime | None = None,
+    end: datetime | None = None,
+    start_offset: str | None = None,
+    end_offset: str | None = None,
+    resolution: timedelta | None = None,
+    output_file: Path | None = None,
     dry_run: bool = False,
-    timezone: str | pytz.BaseTzInfo = get_timezone(),
+    timezone: str | None = None,
 ):
     """
     Create a new report using the Reporter class and save the results
     to the database or export them as CSV or Excel file.
     """
 
-    # parse timezone into a BaseTzInfo object
-    if isinstance(timezone, str):
+    # compute now in the timezone local to the output sensor
+    if timezone is not None:
         check_timezone(timezone)
-        timezone = pytz.timezone(zone=timezone)
-
-    now = timezone.localize(datetime.now())
+    now = pytz.timezone(
+        zone=timezone if timezone is not None else sensor.timezone
+    ).localize(datetime.now())
 
     # apply offsets, if provided
     if start_offset is not None:
@@ -1347,10 +1346,10 @@ def add_report(  # noqa: C901
             "Report computation done, but the report is empty.", **MsgStyle.WARN
         )
 
-    # save the report it's not running in dry mode
+    # save the report if it's not running in dry mode
     if not dry_run:
         click.echo("Saving report to the database...")
-        save_to_db(result)
+        save_to_db(result.dropna())
         db.session.commit()
         click.secho(
             "Success. The report has been saved to the database.",

flexmeasures/cli/data_delete.py

@@ -307,18 +307,26 @@ def delete_nan_beliefs(sensor_id: int | None = None):
 @with_appcontext
 @click.option(
     "--id",
-    "sensor",
+    "sensors",
     type=SensorIdField(),
     required=True,
-    help="Delete a single sensor and its (time series) data. Follow up with the sensor's ID.",
+    multiple=True,
+    help="Delete a sensor and its (time series) data. Follow up with the sensor's ID. "
+    "This argument can be given multiple times",
 )
 def delete_sensor(
-    sensor: Sensor,
+    sensors: list[Sensor],
 ):
-    """Delete a sensor and all beliefs about it."""
-    n = TimedBelief.query.filter(TimedBelief.sensor_id == sensor.id).delete()
-    db.session.delete(sensor)
-    click.confirm(f"Delete {sensor.__repr__()}, along with {n} beliefs?", abort=True)
+    """Delete sensors and their (time series) data."""
+    n = TimedBelief.query.filter(
+        TimedBelief.sensor_id.in_(sensor.id for sensor in sensors)
+    ).delete()
+    for sensor in sensors:
+        db.session.delete(sensor)
+    click.confirm(
+        f"Delete {', '.join(sensor.__repr__() for sensor in sensors)}, along with {n} beliefs?",
+        abort=True,
+    )
     db.session.commit()