Make installable on MacOS (#1000)

* Update db setup text in toy example setup Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Fix commit message Signed-off-by: Nikolai Rozanov <nickolay@seita.nl> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Fix commit message Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Make runnable on macos + update docs accordingly Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Post review fixes. Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Post review fixes Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Add command for highs macOS installation Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Enhance test_closest_sensor to Support Variable Number of Closest Sensors (n) (#998) * use scalars instead of scalar having limit constraint Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> * update the test_closest_sensor, to test multiple sensor as params Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> * update the docstring Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> --------- Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * [CLI] add `--offspring` flag to command flexmeasures delete beliefs (#1003) * add --offspring flag to command flexmeasures delete beliefs Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: include offspring assets in prompt and done messages Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: less alarming message in case a time window was passed Signed-off-by: F.N. Claessen <felix@seita.nl> * Improve processing time for deleting beliefs via CLI (#1005) * speed up deleting beliefs via CLI Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> * docs: changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: typo Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> * fix: CLI message Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> Co-authored-by: F.N. Claessen <felix@seita.nl> Co-authored-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Introduce FLEXMEASURES_FORCE_HTTPS (#1008) * introduce FLEXMEASURES_FORCE_HTTPS so we can control better when we want the app to answer to https. Apply in the internal API to solve a load balancer situation. Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add warning to consider this setting before upgrading Signed-off-by: Nicolas Höning <nicolas@seita.nl> * build the URL right in the beginning of internal view functions Signed-off-by: Nicolas Höning <nicolas@seita.nl> --------- Signed-off-by: Nicolas Höning <nicolas@seita.nl> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * add forgotten changelog entry for #1008 Signed-off-by: Nicolas Höning <nicolas@seita.nl> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Changelog message Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * Changelog message fix Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * [CLI] add `--offspring` flag to command flexmeasures delete beliefs (#1003) * add --offspring flag to command flexmeasures delete beliefs Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: include offspring assets in prompt and done messages Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: less alarming message in case a time window was passed Signed-off-by: F.N. Claessen <felix@seita.nl> * Improve processing time for deleting beliefs via CLI (#1005) * speed up deleting beliefs via CLI Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> * docs: changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: typo Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> * fix: CLI message Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> Co-authored-by: F.N. Claessen <felix@seita.nl> Co-authored-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> * merge master Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> --------- Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com> Signed-off-by: Ahmad Wahid <ahmedwahid16101@gmail.com> Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Nicolas Höning <nicolas@seita.nl> Signed-off-by: nrozanov <50166318+nrozanov@users.noreply.github.com> Co-authored-by: Nikolai <nrozanov@iponweb.net> Co-authored-by: Ahmad Wahid <59763365+Ahmad-Wahid@users.noreply.github.com> Co-authored-by: Victor <victor@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> Co-authored-by: Nicolas Höning <nicolas@seita.nl>
FlexMeasures · Mar 14, 2024 · 7442a78 · 7442a78
1 parent ebb3409
commit 7442a78
Show file tree

Hide file tree

Showing 14 changed files with 92 additions and 21 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,7 +3,7 @@ dist
 
 raw_data
 instance
-venv
+*venv/
 .eggs
 .env
 .vscode
@@ -44,3 +44,7 @@ pyproject.toml
 poetry.lock
 .venv
 .vscode/
+
+.DS_Store
+
+.gitconfig.*
diff --git a/Makefile b/Makefile
@@ -1,10 +1,11 @@
 # Check Python major and minor version
 # For more information, see https://stackoverflow.com/a/22105036
 PYV = $(shell python -c "import sys;t='{v[0]}.{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)")
+HIGHS_DIR = "../HiGHS"
 
 # Note: use tabs
 # actions which are virtual, i.e. not a script
-.PHONY: install install-for-dev install-for-test install-deps install-flexmeasures run-local test freeze-deps upgrade-deps update-docs update-docs-pdf show-file-space show-data-model clean-db cli-autocomplete
+.PHONY: install install-for-dev install-for-test install-deps install-flexmeasures run-local test freeze-deps upgrade-deps update-docs update-docs-pdf show-file-space show-data-model clean-db cli-autocomplete build-highs-macos install-highs-macos
 
 
 # ---- Development ---
@@ -55,6 +56,29 @@ else
 	rm temp-test.in
 endif
 	make install-flexmeasures
+# Locally install HiGS on macOS
+	if [ "$(shell uname)" = "Darwin" ]; then \
+		make install-highs-macos; \
+	fi
+
+$(HIGHS_DIR):
+	if [ ! -d $(HIGHS_DIR) ]; then \
+		git clone https://github.com/ERGO-Code/HiGHS.git $(HIGHS_DIR); \
+	fi
+	brew install cmake;
+
+build-highs-macos: $(HIGHS_DIR)
+	cd $(HIGHS_DIR); \
+	git checkout latest; \
+	mkdir -p build; \
+	cd build; \
+	cmake ..; \
+	make; \
+	make install; \
+	cd ../../flexmeasures;
+
+install-highs-macos: build-highs-macos
+	pip install $(HIGHS_DIR) ; \
 
 install-deps:
 	make install-pip-tools

diff --git a/ci/update-packages.sh b/ci/update-packages.sh
@@ -29,8 +29,8 @@ if ! [ -x "$(command -v docker)" ]; then
   exit 1
 fi
 
-# Check if we can run docker without sudo
-if ! docker ps > /dev/null 2>&1; then
+# Check if we can run docker without sudo (check is not needed for Macos system)
+if ! docker ps > /dev/null 2>&1 && [[ "$(uname)" != "Darwin" ]]; then
   echo "Docker is not running without sudo. Please add your user to the docker group and try again."
   echo "You may use the following command to do so:"
   echo "sudo usermod -aG docker $USER"

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -22,6 +22,7 @@ Infrastructure / Support
 * Improve processing time for deleting beliefs via CLI [see `PR #1005 <https://github.com/FlexMeasures/flexmeasures/pull/1005>`_]
 * Support deleting beliefs via CLI for all offspring assets at once [see `PR #1003 <https://github.com/FlexMeasures/flexmeasures/pull/1003>`_]
 * Add setting ``FLEXMEASURES_FORCE_HTTPS`` to explicitly toggle if HTTPS should be used for all requests [see `PR #1008 <https://github.com/FlexMeasures/flexmeasures/pull/1008>`_]
+* Make flexmeasures installable locally on macOS [see `PR #1000 <https://github.com/FlexMeasures/flexmeasures/pull/1000>`_]
 
 
 v0.19.2 | March 1, 2024

diff --git a/documentation/dev/dependency-management.rst b/documentation/dev/dependency-management.rst
@@ -34,7 +34,7 @@ Now ― you probably have only one Python version installed. Let's say you add a
 
 .. code-block:: bash
 
-    $ cd ci; ./update-packages.sh
+    $ cd ci; ./update-packages.sh; cd ../
 
 This script will use docker to do these upgrades per Python version.
 

diff --git a/documentation/dev/setup-and-guidelines.rst b/documentation/dev/setup-and-guidelines.rst
@@ -54,12 +54,23 @@ Go into the ``flexmeasures`` folder and install all dependencies including the o
 
    $ pip install highspy
 
+On MacOS it will be installed locally by `make install-for-test` and no actions are required on your part
 
-Alternatively, the CBC solver can be installed with:
+Besides highs, the CBC solver is required for tests as well:
 
-.. code-block:: bash
+.. tabs::
+
+    .. tab:: Linux
+
+        .. code-block:: bash
+
+            $ apt-get install coinor-cbc
+
+    .. tab:: MacOS
+
+        .. code-block:: bash
 
-   $ apt-get install coinor-cbc
+            $ brew install cbc
 
 
 Configuration

diff --git a/documentation/host/data.rst b/documentation/host/data.rst
@@ -50,10 +50,23 @@ On Windows:
 * ``conda install psycopg2``
 
 
+On Macos:
+
+.. code-block:: bash
+
+   $ brew update
+   $ brew doctor
+   $ # Need to specify postgres version, in this example we use 13
+   $ brew install postgresql@13
+   $ brew link postgresql@13 --force
+   $ # Start postgres (you can change /usr/local/var/postgres to any directory you like)
+   $ pg_ctl -D /usr/local/var/postgres -l logfile start
+
+
 Using Docker Compose:
 
 
-Alternatively, you can use Docker Compose to run a postgres database. Use can use the following ``docker-compose.yml`` as a starting point:
+Alternatively, you can use Docker Compose to run a postgres database. You can use the following ``docker-compose.yml`` as a starting point:
 
 
 .. code-block:: yaml
@@ -96,9 +109,19 @@ Find the ``timezone`` setting and set it to 'UTC'.
 
 Then restart the postgres server.
 
-.. code-block:: bash
+.. tabs::
+
+   .. tab:: Linux
 
-    $ sudo service postgresql restart
+      .. code-block:: bash
+
+         $ sudo service postgresql restart
+
+   .. tab:: Macos
+
+      .. code-block:: bash
+
+         $ pg_ctl -D /usr/local/var/postgres -l logfile restart
 
 .. note:: If you are using Docker to run postgres, the ``timezone`` setting is already set to ``UTC`` by default.
 
@@ -120,14 +143,16 @@ Proceed to create a database as the postgres superuser (using your postgres user
    $ createuser --pwprompt -U postgres flexmeasures_test  # enter "flexmeasures_test" as password
    $ exit
 
+.. note:: In case you encounter the following "FAILS: sudo: unknown user postgres" you need to create "postgres" OS user with sudo rights first - better done via System preferences -> Users & Groups.
+
 
 Or, from within Postgres console:
 
 .. code-block:: sql
 
-   CREATE USER flexmeasures WITH UNENCRYPTED PASSWORD 'this-is-your-secret-choice';
+   CREATE USER flexmeasures WITH PASSWORD 'this-is-your-secret-choice';
    CREATE DATABASE flexmeasures WITH OWNER = flexmeasures;
-   CREATE USER flexmeasures_test WITH UNENCRYPTED PASSWORD 'flexmeasures_test';
+   CREATE USER flexmeasures_test WITH PASSWORD 'flexmeasures_test';
    CREATE DATABASE flexmeasures_test WITH OWNER = flexmeasures_test;
 
 
@@ -158,6 +183,8 @@ Add the following extensions while logged in as the postgres superuser:
    CREATE EXTENSION cube;
    CREATE EXTENSION earthdistance;
 
+.. note:: Lines from above should be run seperately
+
 
 If you have it, connect to the ``flexmeasures_test`` database and repeat creating these extensions there. Then ``exit``.
 
@@ -230,6 +257,7 @@ You can create users with the ``add user`` command. Check it out:
 
 .. code-block:: bash
 
+   $ flexmeasures add account --help
    $ flexmeasures add user --help
 
 
@@ -281,6 +309,8 @@ You can visualise the data model like this:
 This will generate a picture based on the model code.
 You can also generate picture based on the actual database, see inside the Makefile. 
 
+.. note:: If you encounter "error: externally-managed-environment" when running `make test` and you do it in venv, try `pip cache purge` or use pipx.
+
 Maintenance
 ----------------
 

diff --git a/documentation/tut/toy-example-setup.rst b/documentation/tut/toy-example-setup.rst
@@ -52,7 +52,7 @@ Install Flexmeasures and the database
             $ docker run --rm --name flexmeasures-tutorial-db -e POSTGRES_PASSWORD=fm-db-passwd -e POSTGRES_DB=flexmeasures-db -d --network=flexmeasures_network postgres:latest
             $ docker run --rm --name flexmeasures-tutorial-fm --env SQLALCHEMY_DATABASE_URI=postgresql://postgres:fm-db-passwd@flexmeasures-tutorial-db:5432/flexmeasures-db --env SECRET_KEY=notsecret --env FLEXMEASURES_ENV=development --env LOGGING_LEVEL=INFO -d --network=flexmeasures_network -p 5000:5000 lfenergy/flexmeasures
 
-        To upgrade the FlexMeasures database, execute:
+        To establish the FlexMeasures database structure, execute:
 
         .. code-block:: bash
 

diff --git a/flexmeasures/data/scripts/clean_database.sh b/flexmeasures/data/scripts/clean_database.sh
@@ -100,14 +100,14 @@ if is_database $1
 then
   echo "$1 database exists"
   read -r -p "Make a backup first? [y/N] " response
-  response=${response,,}    # make lowercase
+  response=$(tr '[:upper:]' '[:lower:]' <<< $response) # make lowercase
   if [[ "$response" =~ ^(yes|y)$ ]]; then
     echo "Making db dump ..."
     flexmeasures db-ops dump
   fi
 
   read -r -p "This will drop your database and re-create a clean one. Continue?[y/N] " response
-  response=${response,,} # make lowercase
+  response=$(tr '[:upper:]' '[:lower:]' <<< $response) # make lowercase
   if [[ "$response" =~ ^(yes|y)$ ]]; then
      if ! delete_database $1; then
        exit 1

diff --git a/requirements/3.10/app.txt b/requirements/3.10/app.txt
@@ -18,7 +18,7 @@ attrs==23.2.0
     #   referencing
 babel==2.14.0
     # via py-moneyed
-bcrypt==4.1.2
+bcrypt==4.0.1
     # via -r requirements/app.in
 blinker==1.7.0
     # via

diff --git a/requirements/3.11/app.txt b/requirements/3.11/app.txt
@@ -16,7 +16,7 @@ attrs==23.2.0
     #   referencing
 babel==2.14.0
     # via py-moneyed
-bcrypt==4.1.2
+bcrypt==4.0.1
     # via -r requirements/app.in
 blinker==1.7.0
     # via

diff --git a/requirements/3.8/app.txt b/requirements/3.8/app.txt
@@ -20,7 +20,7 @@ babel==2.14.0
     # via py-moneyed
 backports-zoneinfo==0.2.1
     # via workalendar
-bcrypt==4.1.2
+bcrypt==4.0.1
     # via -r requirements/app.in
 blinker==1.7.0
     # via

diff --git a/requirements/3.9/app.txt b/requirements/3.9/app.txt
@@ -18,7 +18,7 @@ attrs==23.2.0
     #   referencing
 babel==2.14.0
     # via py-moneyed
-bcrypt==4.1.2
+bcrypt==4.0.1
     # via -r requirements/app.in
 blinker==1.7.0
     # via

diff --git a/requirements/app.in b/requirements/app.in
@@ -15,7 +15,8 @@ inflect<=6.0.2
 pydantic<2
 humanize
 psycopg2-binary
-bcrypt
+# due to https://github.com/Flask-Middleware/flask-security/issues/907
+bcrypt<=4.0.1
 pytz
 # limit the numpy version to make it compatible with dependencies in timely-beliefs >=1.18 (libraries sktime, numba).
 numpy<1.25