Merge branch 'main' into feature/planning/separate-fallback

Signed-off-by: Victor <victor@seita.nl>

documentation/changelog.rst

@@ -11,15 +11,16 @@ New features
 
 * Introduce new reporter to compute profit/loss due to electricity flows: `ProfitOrLossReporter` [see `PR #808 <https://github.com/FlexMeasures/flexmeasures/pull/808>`_ and `PR #844 <https://github.com/FlexMeasures/flexmeasures/pull/844>`_]
 * Charts visible in the UI can be exported to PNG or SVG formats in a more automated fashion, using the new CLI command flexmeasures show chart [see `PR #833 <https://github.com/FlexMeasures/flexmeasures/pull/833>`_]
-* API users can ask for a schedule to take into account an explicit ``power-capacity`` (flex-model) and/or ``site-power-capacity`` (flex-context), thereby overriding any existing defaults for their asset [see `PR #850 <https://github.com/FlexMeasures/flexmeasures/pull/850>`_]
+* Chart data visible in the UI can be exported to CSV format [see `PR #849 <https://github.com/FlexMeasures/flexmeasures/pull/849>`_]
 * Sensor charts showing instantaneous observations can be interpolated by setting the ``interpolate`` sensor attribute to one of the `supported Vega-Lite interpolation methods <https://vega.github.io/vega-lite/docs/area.html#properties>`_ [see `PR #851 <https://github.com/FlexMeasures/flexmeasures/pull/851>`_]
 * Scheduling data better distinguishes (e.g. in chart tooltips) when a schedule was the result of a fallback mechanism, by splitting of the fallback mechanism from the main scheduler (as a separate job) [see `PR #846 <https://github.com/FlexMeasures/flexmeasures/pull/846>`_]
+* API users can ask for a schedule to take into account an explicit ``power-capacity`` (flex-model) and/or ``site-power-capacity`` (flex-context), thereby overriding any existing defaults for their asset [see `PR #850 <https://github.com/FlexMeasures/flexmeasures/pull/850>`_]
 
 Infrastructure / Support
 ----------------------
 
 * Allow additional datetime conversions to quantitative time units, specifically, from timezone-naive and/or dayfirst datetimes, which can be useful when importing data [see `PR #831 <https://github.com/FlexMeasures/flexmeasures/pull/831>`_]
-* Add a new tutorial to explain the use of the `AggregatorReporter` to compute the headroom and the `ProfitOrLossReporter` to compute the cost of running a process [see `PR #825 <https://github.com/FlexMeasures/flexmeasures/pull/825>`_]
+* Add a new tutorial to explain the use of the `AggregatorReporter` to compute the headroom and the `ProfitOrLossReporter` to compute the cost of running a process [see `PR #825 <https://github.com/FlexMeasures/flexmeasures/pull/825>`_ and `PR #856 <https://github.com/FlexMeasures/flexmeasures/pull/856>`_]
 * Script to update dependencies across supported Python versions [see `PR #843 <https://github.com/FlexMeasures/flexmeasures/pull/843>`_]
 * Test all supported Python versions in our CI pipeline (GitHub Actions) [see `PR #847 <https://github.com/FlexMeasures/flexmeasures/pull/847>`_]
 * Have our CI pipeline (GitHub Actions) build the Docker image and make a schedule [see `PR #800 <https://github.com/FlexMeasures/flexmeasures/pull/800>`_]

documentation/tut/toy-example-expanded.rst

@@ -103,6 +103,20 @@ First, during the sunny hours of the day, when solar power is being send to the
 
 Second, charging of the battery is also changed a bit (around 10am), as less can be discharged later.
 
+Moreover, we can use reporters to compute the capacity headroom (see :ref:`tut_toy_schedule_process` for more details). The image below shows that the scheduler is respecting the capacity limits.
+
+.. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-headroom-pv.png
+    :align: center
+|
+
+In the case of the scheduler that we ran in the previous tutorial, which did not yet consider the PV, the discharge power would have exceeded the headroom:
+
+.. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-headroom-nopv.png
+    :align: center
+|
+
+.. note:: You can add arbitrary sensors to a chart using the attribute ``sensors_to_show``. See :ref:`view_asset-data` for more.
+
 We hope this part of the tutorial shows how to incorporate a limited grid connection rather easily with FlexMeasures. There are more ways to model such settings, but this is a straightforward one.
 
 This tutorial showed a quick way to add an inflexible load (like solar power) and a grid connection. In :ref:`tut_toy_schedule_process`, we'll turn to something different: the optimal timing of processes with fixed energy work and duration.

documentation/tut/toy-example-reporter.rst

@@ -148,8 +148,9 @@ we are going to create share the same `config`. The `config` defines the price s
 that it will return costs as positive values and revenue as negative values.
 
 Still, we need to define the parameters. The three reports share the same structure for the parameters with the following fields:
-* `input`: sensor that stores the power/energy flow. The number of sensors is limited to 1.
-* `output`: sensor to store the report. We can provide sensors with different resolutions to store the same results at different time scales.
+
+    - `input`: sensor that stores the power/energy flow. The number of sensors is limited to 1.
+    - `output`: sensor to store the report. We can provide sensors with different resolutions to store the same results at different time scales.
 
 .. note::
     It's possible to define the `config` and `parameters` in JSON or YAML formats.
@@ -242,5 +243,6 @@ Check the results `here <http://localhost:5000/sensor/11/>`_. The image should b
     :align: center
 |
 
+
 Now, we can compare the results of the reports to the ones we computed manually in :ref:`this table <table-process>`). Keep in mind that the
 report is showing the profit of each 15min period and adding them all shows that it matches with our previous results.

flexmeasures/ui/static/css/flexmeasures.css

@@ -1825,4 +1825,8 @@ div.heading-group {
   top: 0px;
   height: 50px;
   border-width: 0 0 0 16.6666666667px;
+}
+
+#exportToCSVAction {
+  cursor: pointer;
 }

flexmeasures/ui/static/js/data-utils.js

@@ -61,3 +61,91 @@ function getValueByNestedKey(obj, key) {
     return value;
 }
 
+// From https://stackoverflow.com/a/49332027/13775459
+function toISOLocal(d) {
+    var z  = n =>  ('0' + n).slice(-2);
+    var zz = n => ('00' + n).slice(-3);
+    var off = d.getTimezoneOffset();
+    var sign = off > 0? '-' : '+';
+    off = Math.abs(off);
+
+    return d.getFullYear() + '-' +
+    z(d.getMonth()+1) + '-' +
+    z(d.getDate()) + 'T' +
+    z(d.getHours()) + ':'  +
+    z(d.getMinutes()) + ':' +
+    z(d.getSeconds()) + '.' +
+    zz(d.getMilliseconds()) +
+    sign + z(off/60|0) + ':' + z(off%60);
+}
+
+// Create a function to convert data to CSV
+export function convertToCSV(data) {
+    if (data.length === 0) {
+        return "";
+    }
+
+    // Extract column names from the first object in the data array
+    const columns = Object.keys(data[0]);
+
+    // Create the header row
+    const headerRow = columns.join(',') + '\n';
+
+    // Create the data rows
+    const dataRows = data.map(row => {
+        const rowData = columns.map(col => {
+            const value = row[col];
+            if (typeof value === 'object' && value !== null) {
+                return value.description || '';
+            } else if (col === 'event_start' || col === 'belief_time') {
+                // Check if the column is a timestamp column
+                const timestamp = parseInt(value);
+                if (!isNaN(timestamp)) {
+                    const date = new Date(timestamp); // Convert to Date
+                    // Format the date in ISO8601 format and localize to the specified timezone
+                    // return date.toISOString();  // Use this instead of toISOLocal to get UTC instead
+                    return toISOLocal(date);
+                }
+            } else if (col === 'belief_horizon') {
+                // Check if the column is 'belief_horizon' (duration in ms)
+                const durationMs = parseInt(value);
+                if (!isNaN(durationMs)) {
+                    // Check if the duration is zero
+                    if (durationMs === 0) {
+                        return 'PT0H';
+                    }
+
+                    // Check if the duration is negative
+                    const isNegative = durationMs < 0;
+
+                    // Calculate absolute duration in seconds
+                    const absDurationSeconds = Math.abs(durationMs) / 1000;
+
+                    // Calculate hours, minutes, and seconds
+                    const hours = Math.floor(absDurationSeconds / 3600);
+                    const minutes = Math.floor((absDurationSeconds % 3600) / 60);
+                    const seconds = Math.floor(absDurationSeconds % 60);
+
+                    // Format the duration as ISO8601 duration
+                    let iso8601Duration = isNegative ? '-PT' : 'PT';
+                    if (hours > 0) {
+                        iso8601Duration += hours + 'H';
+                    }
+                    if (minutes > 0) {
+                        iso8601Duration += minutes + 'M';
+                    }
+                    if (seconds > 0) {
+                        iso8601Duration += seconds + 'S';
+                    }
+
+                    return iso8601Duration;
+                }
+            }
+            return value;
+        });
+        return rowData.join(',');
+    });
+
+    // Combine the header row and data rows
+    return "data:text/csv;charset=utf-8," + headerRow + dataRows.join('\n');
+}

flexmeasures/ui/templates/base.html

@@ -200,7 +200,7 @@
     <!-- Render Charts -->
     <script type="module" type="text/javascript">
 
-    import { getUniqueValues } from "{{ url_for('flexmeasures_ui.static', filename='js/data-utils.js') }}";
+    import { getUniqueValues, convertToCSV } from "{{ url_for('flexmeasures_ui.static', filename='js/data-utils.js') }}";
     import { subtract, thisMonth, lastNMonths, countDSTTransitions, getOffsetBetweenTimezonesForDate } from "{{ url_for('flexmeasures_ui.static', filename='js/daterange-utils.js') }}";
     import { partition, updateBeliefs, beliefTimedelta, setAbortableTimeout} from "{{ url_for('flexmeasures_ui.static', filename='js/replay-utils.js') }}";
 
@@ -282,6 +282,29 @@
 
         await vegaEmbed('#'+elementId, chartSpecsPath + 'dataset_name=' + datasetName + '&width=container&include_sensor_annotations=false&include_asset_annotations=false&chart_type=' + chartType, {{ chart_options | safe }})
         .then(function (result) {
+
+            // Create a custom menu item for exporting to CSV
+            const exportToCSVAction = document.createElement('a');
+            exportToCSVAction.id = 'exportToCSVAction';
+            exportToCSVAction.href = '#';
+            exportToCSVAction.download = datasetName + '.csv';
+            exportToCSVAction.textContent = 'Save as CSV';
+            exportToCSVAction.addEventListener('mousedown', async function (event) {
+                event.preventDefault();
+                const chartData = vegaView.data(datasetName);
+                const csvContent = convertToCSV(chartData);
+                const encodedUri = encodeURI(csvContent);
+                exportToCSVAction.href = encodedUri;
+            });
+
+            // Append menu item to chart actions (hamburger menu)
+            const vegaActions = document.querySelector('.vega-actions');
+            if (vegaActions) {
+                vegaActions.appendChild(exportToCSVAction);
+            } else {
+                console.log('Warning: CSV export functionality is not available, because no div with class=vega-actions was found to append export action to');
+            }
+
             // result.view is the Vega View, chartSpecsPath is the original Vega-Lite specification
             vegaView = result.view;
             if (previousResult) {

flexmeasures/ui/utils/chart_defaults.py

@@ -1,7 +1,7 @@
 chart_options = dict(
     mode="vega-lite",
     renderer="svg",
-    actions={"export": True, "source": True, "editor": True},
+    actions={"export": True, "source": False, "compiled": False, "editor": False},
     theme="light",
     tooltip={"theme": "light"},
 )

flexmeasures/ui/utils/view_utils.py

@@ -74,7 +74,15 @@ def render_flexmeasures_template(html_filename: str, **variables):
         current_user.is_authenticated and current_user.username or ""
     )
     variables["js_versions"] = current_app.config.get("FLEXMEASURES_JS_VERSIONS")
-    variables["chart_options"] = json.dumps(chart_options)
+
+    # Chart options passed to vega-embed
+    options = chart_options.copy()
+    if "sensor_id" in variables:
+        options["downloadFileName"] = f"sensor-{variables['sensor_id']}"
+    elif "asset" in variables:
+        asset = variables["asset"]
+        options["downloadFileName"] = f"asset-{asset.id}-{asset.name}"
+    variables["chart_options"] = json.dumps(options)
 
     variables["menu_logo"] = current_app.config.get("FLEXMEASURES_MENU_LOGO_PATH")
     variables["extra_css"] = current_app.config.get("FLEXMEASURES_EXTRA_CSS_PATH")