Merge branch 'main' into 632-make-pip-installable-on-windows

Signed-off-by: Nicolas Höning <nicolas@seita.nl>
FlexMeasures · Aug 7, 2023 · d241ee4 · d241ee4
2 parents 3dc2792 + 2866237
commit d241ee4
Show file tree

Hide file tree

Showing 131 changed files with 4,958 additions and 5,937 deletions.
diff --git a/.github/workflows/lint-and-test.yml b/.github/workflows/lint-and-test.yml
@@ -49,7 +49,6 @@ jobs:
             ${{ runner.os }}-pip-
       - run: |
           ci/setup-postgres.sh
-          sudo apt-get -y install coinor-cbc
       - name: Install FlexMeasures & exact dependencies for tests
         run: make install-for-test
         if: github.event_name == 'push' && steps.cache.outputs.cache-hit != 'true'

diff --git a/.vscode/spellright.dict b/.vscode/spellright.dict
@@ -257,3 +257,8 @@ dataframe
 dataframes
 args
 docstrings
+Auth
+ctx_loader
+ctx_arg_name
+ctx_arg_pos
+dataset
diff --git a/Dockerfile b/Dockerfile
@@ -12,7 +12,7 @@ WORKDIR /app
 COPY requirements /app/requirements
 
 # py dev tooling
-RUN python3 -m pip install --no-cache-dir  --upgrade pip && python3 --version && pip3 install --no-cache-dir --upgrade setuptools && pip3 install --no-cache-dir -r requirements/app.txt -r requirements/dev.txt -r requirements/test.txt
+RUN python3 -m pip install --no-cache-dir  --upgrade pip && python3 --version && pip3 install --no-cache-dir --upgrade setuptools && pip3 install highspy && pip3 install --no-cache-dir -r requirements/app.txt -r requirements/dev.txt -r requirements/test.txt
 
 # Copy code and meta/config data
 COPY setup.* .flaskenv wsgi.py /app/

diff --git a/Makefile b/Makefile
@@ -65,7 +65,7 @@ install-flexmeasures:
 	pip install -e .
 
 install-pip-tools:
-	pip3 install -q "pip-tools>=6.4"
+	pip3 install -q "pip-tools>=7.0"
 
 install-docs-dependencies:
 	pip install -r requirements/docs.txt

diff --git a/ci/install-cbc-from-source.sh b/ci/install-cbc-from-source.sh
@@ -1,9 +1,10 @@
 #!/bin/bash
 
-######################################################################
+#################################################################################
 # This script installs the Cbc solver from source
-# (for cases where you can't install the coinor-cbc package)
-######################################################################
+# (for cases where you can't install the coinor-cbc package via package managers)
+# Note: We use 2.9 here, but 2.10 has also been working well in our CI pipeline.
+#################################################################################
 
 # Install to this dir
 SOFTWARE_DIR=/home/seita/software

diff --git a/ci/run_mypy.sh b/ci/run_mypy.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 set -e
 pip install --upgrade 'mypy>=0.902'
-pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools types-tabulate
+pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools types-tabulate types-PyYAML
 # We are checking python files which have type hints, and leave out bigger issues we made issues for
 # * data/scripts: We'll remove legacy code: https://trello.com/c/1wEnHOkK/7-remove-custom-data-scripts
 # * data/models and data/services: https://trello.com/c/rGxZ9h2H/540-makequery-call-signature-is-incoherent

diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst
@@ -5,10 +5,13 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
-v3.0-11 | 2023-07-20
+v3.0-11 | 2023-08-02
 """"""""""""""""""""
 
 - Added REST endpoint for fetching one sensor: `/sensors/<id>` (GET)
+- Added REST endpoint for adding a sensor: `/sensors` (POST)
+- Added REST endpoint for patching a sensor: `/sensors/<id>` (PATCH)
+- Added REST endpoint for deleting a sensor: `/sensors/<id>` (DELETE)
 
 v3.0-10 | 2023-06-12
 """"""""""""""""""""

diff --git a/documentation/api/notation.rst b/documentation/api/notation.rst
@@ -94,7 +94,7 @@ It uses the fact that all FlexMeasures sensors have unique IDs.
 
 The ``fm0`` scheme is the original scheme.
 It identified different types of sensors (such as grid connections, weather sensors and markets) in different ways.
-The ``fm0`` scheme has been deprecated and is no longer supported officially.
+The ``fm0`` scheme has been sunset since API version 3.
 
 
 Timeseries
@@ -202,9 +202,14 @@ Here are the three types of flexibility models you can expect to be built-in:
 
    For some examples, see the `[POST] /sensors/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint docs.
 
-2) For **shiftable processes**
+2) For **processes**
 
-   .. todo:: A simple and proven algorithm exists, but is awaiting proper integration into FlexMeasures, see `PR 729 <https://github.com/FlexMeasures/flexmeasures/pull/729>`_.
+    - ``power``: nominal power of the load.
+    - ``duration``: time that the load last.
+    - ``optimization_sense``: objective of the scheduler, to maximize or minimize.
+    - ``time_restrictions``: time periods in which the load cannot be schedule to.
+    - ``process_type``: INFLEXIBLE, BREAKABLE or SHIFTABLE.
+
 
 3) For **buffer devices** (e.g. thermal energy storage systems connected to heat pumps), use the same flexibility parameters described above for storage devices. Here are some tips to model a buffer with these parameters:
 
@@ -388,22 +393,17 @@ For example, to obtain data originating from data source 42, include the followi
 
 Data source IDs can be found by hovering over data in charts.
 
-.. note:: Older API version (< 3) accepted user IDs (integers), account roles (strings) and lists thereof, instead of data source IDs (integers).
-
-
 .. _units:
 
 Units
 ^^^^^
 
-From API version 3 onwards, we are much more flexible with sent units.
+The FlexMeasures API is quite flexible with sent units.
 A valid unit for timeseries data is any unit that is convertible to the configured sensor unit registered in FlexMeasures.
 So, for example, you can send timeseries data with "W" unit to a "kW" sensor.
 And if you wish to do so, you can even send a timeseries with "kWh" unit to a "kW" sensor.
 In this case, FlexMeasures will convert the data using the resolution of the timeseries.
 
-For API versions 1 and 2, the unit sent needs to be an exact match with the sensor unit, and only "MW" is allowed for power sensors.
-
 .. _signs:
 
 Signs of power values

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -8,27 +8,40 @@ v0.15.0 | July XX, 2023
 
 .. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
 
+.. warning:: Upgrading to this version requires installing the LP/MILP solver HiGHS using ``pip install highspy``.
+
+.. warning:: If your server is running in play mode (``FLEXMEASURES_MODE = "play"``), users will be able to see sensor data from any account [see `PR #740 <https://www.github.com/FlexMeasures/flexmeasures/pull/740>`_].
+
 New features
 -------------
 
+* Users can select a new chart type (daily heatmap) on the sensor page of the UI, showing how sensor values are distributed over the time of day [see `PR #715 <https://www.github.com/FlexMeasures/flexmeasures/pull/715>`_]
+* Users are warned in the UI on when the data they are seeing includes one or more Daylight Saving Time (DST) transitions, and heatmaps (see previous feature) visualize these transitions intuitively [see `PR #723 <https://www.github.com/FlexMeasures/flexmeasures/pull/723>`_]
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times [see `PR #734 <https://www.github.com/FlexMeasures/flexmeasures/pull/734>`_]
 * Make it a lot easier to read off the color legend on the asset page, especially when showing many sensors, as they will now be ordered from top to bottom in the same order as they appear in the chart (as defined in the ``sensors_to_show`` attribute), rather than alphabetically [see `PR #742 <https://www.github.com/FlexMeasures/flexmeasures/pull/742>`_]
+* Users on FlexMeasures servers in play mode (``FLEXMEASURES_MODE = "play"``) can use the ``sensors_to_show`` attribute to show any sensor on their asset pages, rather than only sensors registered to assets in their own account or to public assets [see `PR #740 <https://www.github.com/FlexMeasures/flexmeasures/pull/740>`_]
 * Having percentages within the [0, 100] domain is such a common use case that we now always include it in sensor charts with % units, making it easier to read off individual charts and also to compare across charts [see `PR #739 <https://www.github.com/FlexMeasures/flexmeasures/pull/739>`_]
 * DataSource table now allows storing arbitrary attributes as a JSON (without content validation), similar to the Sensor and GenericAsset tables [see `PR #750 <https://www.github.com/FlexMeasures/flexmeasures/pull/750>`_]
-* Added API endpoint `/sensor/<id>` for fetching a single sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_]
+* Added API endpoints `/sensors/<id>` for fetching a single sensor, `/sensors` (POST) for adding a sensor, `/sensors/<id>` (PATCH) for updating a sensor and `/sensors/<id>` (DELETE) for deleting a sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_] and [see `PR #767 <https://www.github.com/FlexMeasures/flexmeasures/pull/767>`_] and [see `PR #773 <https://www.github.com/FlexMeasures/flexmeasures/pull/773>`_] and [see `PR #784 <https://www.github.com/FlexMeasures/flexmeasures/pull/784>`_]
 * The CLI now allows to set lists and dicts as asset & sensor attributes (formerly only single values) [see `PR #762 <https://www.github.com/FlexMeasures/flexmeasures/pull/762>`_]
+* Add `ProcessScheduler` class to optimize the starting time of processes one of the policies developed (INFLEXIBLE, SHIFTABLE and BREAKABLE), accessible via the CLI command `flexmeasures add schedule for-process` [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_ and `PR #768 <https://www.github.com/FlexMeasures/flexmeasures/pull/768>`_]
+* Users will be able to see (e.g. in the UI) exactly which reporter created the report (saved as sensor data), and hosts will be able to identify exactly which configuration was used to create a given report [see `PR #751 <https://www.github.com/FlexMeasures/flexmeasures/pull/751>`_, `PR #752 <https://www.github.com/FlexMeasures/flexmeasures/pull/752>`_ and `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
+* The CLI `flexmeasures add report` now allows passing `config` and `parameters` in YAML format as files or editable via the system's default editor [see `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
 
 Bugfixes
 -----------
 
+* Add binary constraint to avoid energy leakages during periods with negative prices [see `PR #770 <https://www.github.com/FlexMeasures/flexmeasures/pull/770>`_]
+
 Infrastructure / Support
 ----------------------
 
 * Add support for profiling Flask API calls using ``pyinstrument`` (if installed). Can be enabled by setting the environment variable ``FLEXMEASURES_PROFILE_REQUESTS`` to ``True`` [see `PR #722 <https://www.github.com/FlexMeasures/flexmeasures/pull/722>`_]
 * The endpoint `[POST] /health/ready <api/v3_0.html#get--api-v3_0-health-ready>`_ returns the status of the Redis connection, if configured [see `PR #699 <https://www.github.com/FlexMeasures/flexmeasures/pull/699>`_]
 * Document the `device_scheduler` linear program [see `PR #764 <https://www.github.com/FlexMeasures/flexmeasures/pull/764>`_].
-
-/api/v3_0/health/ready
+* Add support for `HiGHS <https://highs.dev/>`_ solver [see `PR #766 <https://www.github.com/FlexMeasures/flexmeasures/pull/766>`_].
+* Add support for installing FlexMeasures under Python 3.11 [see `PR #771 <https://www.github.com/FlexMeasures/flexmeasures/pull/771>`_].
+* Removed obsolete code dealing with deprecated data models (e.g. assets, markets and weather sensors), and sunset the fm0 scheme for entity addresses [see `PR #695 <https://www.github.com/FlexMeasures/flexmeasures/pull/695>`_ and `project 11 <https://www.github.com/FlexMeasures/flexmeasures/projects/11>`_]
 
 v0.14.2 | July 25, 2023
 ============================

diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
@@ -8,6 +8,9 @@ since v0.15.0 | July XX, 2023
 =================================
 
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times.
+* Add ``flexmeasures add schedule for-process`` to create a new process schedule for a given power sensor.
+* Add support for describing ``config`` and ``parameters`` in YAML for the command ``flexmeasures add report``, editable in user's code editor using the flags ``--edit-config`` or ``--edit-parameters``.
+* Add ``--kind process`` option to create the asset and sensors for the ``ProcessScheduler`` tutorial.
 
 since v0.14.1 | June XX, 2023
 =================================

diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst
@@ -36,6 +36,7 @@ of which some are referred to in this documentation.
 ``flexmeasures add source``                       Add a new data source.
 ``flexmeasures add forecasts``                    Create forecasts.
 ``flexmeasures add schedule for-storage``         Create a charging schedule for a storage asset.
+``flexmeasures add schedule for-process``         Create a schedule for a process asset.
 ``flexmeasures add holidays``                     Add holiday annotations to accounts and/or assets.
 ``flexmeasures add annotation``                   Add annotation to accounts, assets and/or sensors.
 ``flexmeasures add toy-account``                  Create a toy account, for tutorials and trying things.

diff --git a/documentation/concepts/benefits.rst b/documentation/concepts/benefits.rst
@@ -34,4 +34,6 @@ The platform operator (as ESCo or Aggregator) and asset owners can share the pro
 FlexMeasures plans on providing basic accounting for this.
 
 
-.. note:: Read more on flexibility opportunities and activations, as well as profit sharing on :ref:`benefits_of_flex`
+..
+    <This cross reference has been commented out while we rewrite the benefits_of_flex page to fit out new data model and UI>
+    note:: Read more on flexibility opportunities and activations, as well as profit sharing on :ref:`benefits_of_flex`
diff --git a/documentation/concepts/device_scheduler.rst b/documentation/concepts/device_scheduler.rst
@@ -48,6 +48,7 @@ Symbol                              Variable in the Code
 :math:`P^{ems}_{min}(j)`              ems_derivative_min                                 Minimum flow of the EMS during time period :math:`j`.
 :math:`P^{ems}_{max}(j)`              ems_derivative_max                                 Maximum flow of the EMS during time period :math:`j`.
 :math:`Commitment(c,j)`               commitment_quantity                                Commitment c (at EMS level) over time step :math:`j`.
+:math:`M`                             M                                                  Large constant number, upper bound of :math:`Power_{up}(d,j)` and :math:`|Power_{down}(d,j)|`
 ================================ ================================================ ==============================================================================================================  
 
 
@@ -62,6 +63,7 @@ Symbol                              Variable in the Code
 :math:`P_{up}(d,j)`                   device_power_up                                    Upwards power of device :math:`d` during time period :math:`j`.
 :math:`P_{down}(d,j)`                 device_power_down                                  Downwards power of device :math:`d` during time period :math:`j`.
 :math:`P^{ems}(j)`                    ems_power                                          Aggregated power of all the devices during time period :math:`j`.
+:math:`\sigma(d,j)`                   device_power_sign                                  Upwards power activation if :math:`\sigma(d,j)=1`, downwards power activation otherwise.
 ================================ ================================================ ==============================================================================================================  
 
 Cost function
@@ -154,6 +156,22 @@ Device bounds
     0 \leq P_{up}(d,j)\leq max(P_{max}(d,j),0)
 
 
+Upwards/Downwards activation selection
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Avoid simultaneous upwards and downwards activation during the same time period.
+
+.. math:: 
+  :name: device_up_derivative_sign
+
+    P_{up}(d,j) \leq M \cdot \sigma(d,j)
+
+.. math:: 
+  :name: device_down_derivative_sign
+
+    -P_{down}(d,j) \leq M \cdot (1-\sigma(d,j))
+
+
 Grid constraints
 ^^^^^^^^^^^^^^^^^
 

diff --git a/documentation/configuration.rst b/documentation/configuration.rst
@@ -55,7 +55,7 @@ Default: ``False``
 FLEXMEASURES_LP_SOLVER
 ^^^^^^^^^^^^^^^^^^^^^^
 
-The command to run the scheduling solver. This is the executable command which FlexMeasures calls via the `pyomo library <http://www.pyomo.org/>`_. Other values might be ``cplex`` or ``glpk``. Consult `their documentation <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ to learn more. 
+The command to run the scheduling solver. This is the executable command which FlexMeasures calls via the `pyomo library <http://www.pyomo.org/>`_. Other values might be ``cplex``, ``glpk`` or ``appsi_highs`` for `HiGHS <https://highs.dev/>`_. Consult `their documentation <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ to learn more. 
 
 Default: ``"cbc"``
 
@@ -179,7 +179,7 @@ For more fine-grained control, the entries can also be tuples of view names and
 
 .. note:: This fine-grained control requires FlexMeasures version 0.6.0
 
-Default: ``["dashboard", "analytics", "portfolio", "assets", "users"]``
+Default: ``["dashboard"]``
 
 
 FLEXMEASURES_MENU_LISTED_VIEW_ICONS

diff --git a/documentation/dev/auth.rst b/documentation/dev/auth.rst
@@ -30,7 +30,7 @@ You, as the endpoint author, need to make sure this is checked. Here is an examp
         {"the_resource": ResourceIdField(data_key="resource_id")},
         location="path",
     )
-    @permission_required_for_context("read", arg_name="the_resource")
+    @permission_required_for_context("read", ctx_arg_name="the_resource")
     @as_json
     def view(resource_id: int, resource: Resource):
         return dict(name=resource.name)

diff --git a/documentation/dev/introduction.rst b/documentation/dev/introduction.rst
@@ -48,16 +48,18 @@ Go into the ``flexmeasures`` folder and install all dependencies including the o
    $ cd flexmeasures
    $ make install-for-dev
 
-:ref:`Install the LP solver <install-lp-solver>`. On Unix the Cbc LP solver can be installed with:
+:ref:`Install the LP solver <install-lp-solver>`. On Linux, the HiGHS solver can be installed with:
 
 .. code-block:: bash
 
-   $ apt-get install coinor-cbc
+   $ pip install highspy
 
-.. note::
 
-    If you are on Windows, then running & developing FlexMeasures will not work 100%. For instance, the queueing only works if you install rq-win (https://github.com/michaelbrooks/rq-win) manually and the make tooling is difficult to get to work as well.
-    We recommend to use the Windows Sub-system for Linux (https://learn.microsoft.com/en-us/windows/wsl/install) or work via Docker-compose (https://flexmeasures.readthedocs.io/en/latest/dev/docker-compose.html).
+Alternatively, the CBC solver can be installed with:
+
+.. code-block:: bash
+
+   $ apt-get install coinor-cbc
 
 
 Configuration
@@ -136,6 +138,13 @@ Otherwise, you need to add some other user first. Here is how we add an admin:
 (The account-id you need in the 2nd command is printed by the 1st)
 
 
+.. note::
+
+    If you are on Windows, then running & developing FlexMeasures will not work 100%. For instance, the queueing only works if you install rq-win (https://github.com/michaelbrooks/rq-win) manually and the make tooling is difficult to get to work as well.
+    We recommend to use the Windows Sub-system for Linux (https://learn.microsoft.com/en-us/windows/wsl/install) or work via Docker-compose (https://flexmeasures.readthedocs.io/en/latest/dev/docker-compose.html).
+
+
+
 Logfile
 --------