Merge branch 'main' into 763-requirements

Dockerfile

@@ -12,7 +12,7 @@ WORKDIR /app
 COPY requirements /app/requirements
 
 # py dev tooling
-RUN python3 -m pip install --no-cache-dir  --upgrade pip && python3 --version && pip3 install --no-cache-dir --upgrade setuptools && pip3 install --no-cache-dir -r requirements/app.txt -r requirements/dev.txt -r requirements/test.txt
+RUN python3 -m pip install --no-cache-dir  --upgrade pip && python3 --version && pip3 install --no-cache-dir --upgrade setuptools && pip3 install highspy && pip3 install --no-cache-dir -r requirements/app.txt -r requirements/dev.txt -r requirements/test.txt
 
 # Copy code and meta/config data
 COPY setup.* .flaskenv wsgi.py /app/

documentation/api/notation.rst

@@ -202,9 +202,14 @@ Here are the three types of flexibility models you can expect to be built-in:
 
    For some examples, see the `[POST] /sensors/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint docs.
 
-2) For **shiftable processes**
+2) For **processes**
 
-   .. todo:: A simple and proven algorithm exists, but is awaiting proper integration into FlexMeasures, see `PR 729 <https://github.com/FlexMeasures/flexmeasures/pull/729>`_.
+    - ``power``: nominal power of the load.
+    - ``duration``: time that the load last.
+    - ``optimization_sense``: objective of the scheduler, to maximize or minimize.
+    - ``time_restrictions``: time periods in which the load cannot be schedule to.
+    - ``process_type``: INFLEXIBLE, BREAKABLE or SHIFTABLE.
+
 
 3) For **buffer devices** (e.g. thermal energy storage systems connected to heat pumps), use the same flexibility parameters described above for storage devices. Here are some tips to model a buffer with these parameters:
 

documentation/changelog.rst

@@ -17,6 +17,7 @@ New features
 * DataSource table now allows storing arbitrary attributes as a JSON (without content validation), similar to the Sensor and GenericAsset tables [see `PR #750 <https://www.github.com/FlexMeasures/flexmeasures/pull/750>`_]
 * Added API endpoint `/sensor/<id>` for fetching a single sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_]
 * The CLI now allows to set lists and dicts as asset & sensor attributes (formerly only single values) [see `PR #762 <https://www.github.com/FlexMeasures/flexmeasures/pull/762>`_]
+* Add `ProcessScheduler` class, which optimizes the starting time of processes using one of the following policies: INFLEXIBLE, SHIFTABLE and BREAKABLE [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_]
 
 Bugfixes
 -----------
@@ -27,10 +28,9 @@ Infrastructure / Support
 * Add support for profiling Flask API calls using ``pyinstrument`` (if installed). Can be enabled by setting the environment variable ``FLEXMEASURES_PROFILE_REQUESTS`` to ``True`` [see `PR #722 <https://www.github.com/FlexMeasures/flexmeasures/pull/722>`_]
 * The endpoint `[POST] /health/ready <api/v3_0.html#get--api-v3_0-health-ready>`_ returns the status of the Redis connection, if configured [see `PR #699 <https://www.github.com/FlexMeasures/flexmeasures/pull/699>`_]
 * Document the `device_scheduler` linear program [see `PR #764 <https://www.github.com/FlexMeasures/flexmeasures/pull/764>`_].
+* Add support for `HiGHS <https://highs.dev/>`_ solver [see `PR #766 <https://www.github.com/FlexMeasures/flexmeasures/pull/766>`_].
 
-/api/v3_0/health/ready
-
-v0.14.2 | July 21, 2023
+v0.14.2 | July 25, 2023
 ============================
 
 Bugfixes

documentation/configuration.rst

@@ -55,7 +55,7 @@ Default: ``False``
 FLEXMEASURES_LP_SOLVER
 ^^^^^^^^^^^^^^^^^^^^^^
 
-The command to run the scheduling solver. This is the executable command which FlexMeasures calls via the `pyomo library <http://www.pyomo.org/>`_. Other values might be ``cplex`` or ``glpk``. Consult `their documentation <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ to learn more. 
+The command to run the scheduling solver. This is the executable command which FlexMeasures calls via the `pyomo library <http://www.pyomo.org/>`_. Other values might be ``cplex``, ``glpk`` or ``appsi_highs`` for `HiGHS <https://highs.dev/>`_. Consult `their documentation <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ to learn more. 
 
 Default: ``"cbc"``
 

documentation/dev/introduction.rst

@@ -54,6 +54,11 @@ Go into the ``flexmeasures`` folder and install all dependencies including the o
 
    $ apt-get install coinor-cbc
 
+Alternatively, HiGHS solver can be installed with pip:
+
+.. code-block:: bash
+
+   $ pip install highspy
 
 Configuration
 ^^^^^^^^^^^^^^^^^^^^

documentation/host/data.rst

@@ -28,13 +28,18 @@ Install
 
 We believe FlexMeasures works with Postgres above version 9 and we ourselves have run it with versions up to 14.
 
-On Unix:
+On Linux:
 
 .. code-block:: bash
 
+   $ # On Ubuntu and Debian, you can install postgres like this:
    $ sudo apt-get install postgresql-12  # replace 12 with the version available in your packages
    $ pip install psycopg2-binary
 
+   $ # On Fedora, you can install postgres like this:
+   $ sudo dnf install postgresql postgresql-server
+   $ sudo postgresql-setup --initdb --unit postgresql
+
 
 On Windows:
 
@@ -45,6 +50,35 @@ On Windows:
 * ``conda install psycopg2``
 
 
+Using Docker Compose:
+
+
+Alternatively, you can use Docker Compose to run a postgres database. Use can use the following ``docker-compose.yml`` as a starting point:
+
+
+.. code-block:: yaml
+
+   version: '3.7'
+
+   services:
+     postgres:
+       image: postgres:latest
+       restart: always
+       environment:
+         POSTGRES_USER: flexmeasures
+         POSTGRES_PASSWORD: this-is-your-secret-choice
+         POSTGRES_DB: flexmeasures
+       ports:
+         - 5432:5432
+       volumes:
+         - ./postgres-data:/var/lib/postgresql/data
+       network_mode: host
+
+To run this, simply type ``docker-compose up`` in the directory where you saved the ``docker-compose.yml`` file. Pass the ``-d`` flag to run it in the background.
+
+This will create a postgres database in a directory ``postgres-data`` in your current working directory. You can change the password and database name to your liking. You can also change the port mapping to e.g. ``5433:5432`` if you already have a postgres database running on your host machine.
+
+
 Make sure postgres represents datetimes in UTC timezone
 ^^^^^^^^^^^^^
 
@@ -66,6 +100,8 @@ Then restart the postgres server.
 
     $ sudo service postgresql restart
 
+.. note:: If you are using Docker to run postgres, the ``timezone`` setting is already set to ``UTC`` by default.
+
 
 Create "flexmeasures" and "flexmeasures_test" databases and users
 ^^^^^^^^^^^^^

documentation/host/deployment.rst

@@ -48,10 +48,10 @@ Keep in mind that FlexMeasures is based on `Flask <https://flask.palletsprojects
 Install the linear solver on the server
 ---------------------------------------
 
-To compute schedules, FlexMeasures uses the `Cbc <https://github.com/coin-or/Cbc>`_ mixed integer linear optimization solver.
-It is used through `Pyomo <http://www.pyomo.org>`_\ , so in principle supporting a `different solver <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ would be possible.
+To compute schedules, FlexMeasures uses the `CBC <https://github.com/coin-or/Cbc>`_ (FlexMeasures solver by default) or `HiGHS <https://highs.dev/>`_ mixed integer linear optimization solver.
+Solvers are used through `Pyomo <http://www.pyomo.org>`_\ , so in principle supporting a `different solver <https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers>`_ would be possible.
 
-Cbc needs to be present on the server where FlexMeasures runs, under the ``cbc`` command.
+CBC needs to be present on the server where FlexMeasures runs, under the ``cbc`` command.
 
 You can install it on Debian like this:
 
@@ -66,3 +66,10 @@ pass a directory for the installation.
 
 In case you want to install a later version, adapt the version in the script. 
 
+HiGHS can be installed using pip:
+
+.. code-block:: bash
+
+   $ pip install highspy
+
+

documentation/index.rst

@@ -180,9 +180,9 @@ The platform operator of FlexMeasures can be an Aggregator.
     :caption: Tutorials
     :maxdepth: 1
 
+    tut/installation
     tut/toy-example-from-scratch
     tut/toy-example-expanded
-    tut/installation
     tut/posting_data
     tut/forecasting_scheduling
     tut/building_uis

documentation/tut/installation.rst

@@ -226,9 +226,12 @@ For FlexMeasures to be able to send email to users (e.g. for resetting passwords
 Install an LP solver
 ^^^^^^^^^^^^^^^^^^^^
 
-For planning balancing actions, the FlexMeasures platform uses a linear program solver. Currently that is the Cbc solver. See :ref:`solver-config` if you want to change to a different solver.
+For planning balancing actions, the FlexMeasures platform uses a linear program solver. Currently that is the CBC or HiGHS solvers. See :ref:`solver-config` if you want to change to a different solver.
 
-Installing Cbc can be done on Unix via:
+CBC
+*****
+
+Installing CBC can be done on Unix via:
 
 .. code-block:: bash
 
@@ -239,7 +242,18 @@ Installing Cbc can be done on Unix via:
 
 We provide a script for installing from source (without requiring ``sudo`` rights) in the `ci` folder.
 
-More information (e.g. for installing on Windows) on `the Cbc website <https://projects.coin-or.org/Cbc>`_.
+More information (e.g. for installing on Windows) on `the CBC website <https://projects.coin-or.org/Cbc>`_.
+
+HiGHS
+******
+
+HiGHS is a modern LP solver that aims at solving large problems. It can be installed using pip:
+
+.. code-block:: bash
+
+   $ pip install highspy
+
+More information (e.g. for installing on Windows) on `the HiGHS website <https://highs.dev/>`_.
 
 
 Install and configure Redis

documentation/tut/toy-example-from-scratch.rst

@@ -51,7 +51,7 @@ Install Flexmeasures and the database
             $ docker run --rm --name flexmeasures-tutorial-fm --env SQLALCHEMY_DATABASE_URI=postgresql://postgres:fm-db-passwd@flexmeasures-tutorial-db:5432/flexmeasures-db --env SECRET_KEY=notsecret --env FLASK_ENV=development --env LOGGING_LEVEL=INFO -d --network=flexmeasures_network -p 5000:5000 lfenergy/flexmeasures
             $ docker exec flexmeasures-tutorial-fm bash -c "flexmeasures db upgrade"
 
-        .. note:: A tip on Linux/macOS ― You might have the ``docker`` command, but need `sudo` rights to execute it. ``alias docker='sudo docker'`` enables you to still run this tutorial.
+        .. note:: A tip on Linux/macOS ― You might have the ``docker`` command, but need `sudo` rights to execute it. ``alias docker='sudo docker'`` enables you to still run this tutorial. Alternatively, you can add yourself to the ``docker`` group. See `here <https://docs.docker.com/engine/install/linux-postinstall/>`_ for more information.
 
         Now - what's *very important* to remember is this: The rest of this tutorial will happen *inside* the ``flexmeasures-tutorial-fm`` container! This is how you hop inside the container and run a terminal there:
 

flexmeasures/data/models/planning/linear_optimization.py

@@ -339,10 +339,16 @@ def cost_function(m):
     model.costs = Objective(rule=cost_function, sense=minimize)
 
     # Solve
+
+    # load_solutions=False to avoid a RuntimeError exception in appsi solvers when solving an infeasible problem.
     results = SolverFactory(current_app.config.get("FLEXMEASURES_LP_SOLVER")).solve(
-        model
+        model, load_solutions=False
     )
 
+    # load the results only if a feasible solution has been found
+    if len(results.solution) > 0:
+        model.solutions.load_from(results)
+
     planned_costs = value(model.costs)
     planned_power_per_device = []
     for d in model.d: