Merged in feature/package-tools (pull request #64)

Use pip-tools to freeze our dependencies

* requirements.txt, add a Makefile, add pip-compile (update requirements.txt) as pre-commit step

* add pip-sync to Makefile and CI, document it

* add pinned depencencies to bitbucket CI

* revert

Approved-by: Felix Claessen

.pre-commit-config.yaml

@@ -1,4 +1,8 @@
 repos:
+-   repo: https://github.com/jazzband/pip-tools
+    rev: 5.0.0
+    hooks:
+    - id: pip-compile
 -   repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v1.3.0
     hooks:

Makefile

@@ -0,0 +1,27 @@
+# Note: use tabs
+# actions which are virtual, i.e. not a script
+.PHONY: install install-deps install-bvp freeze-deps show-file-space
+
+install: install-deps install-bvp
+
+install-deps:
+	pip install pip-tools
+	pip-sync
+
+freeze-deps:
+	pip-compile
+
+install-bvp:
+	python setup.py develop
+
+upgrade-db:
+	flask db current
+	flask db upgrade
+
+update-docs:
+	pip install sphinx sphinxcontrib.httpdomain
+	cd documentation; make clean; make html; cd ..
+
+show-file-space:
+	# Where is our file space going?
+	du --summarize --human-readable --total ./* ./.[a-zA-Z]* | sort -h

Readme.md

@@ -6,7 +6,6 @@ The *Balancing Valorisation Platform (BVP)* is a tool for scheduling balancing a
 Its purpose is to offer these balancing actions as one aggregated service to energy markets, realising the highest possible value for its users.
 
 
-
 ## Build & Run
 
 
@@ -21,9 +20,11 @@ Its purpose is to offer these balancing actions as one aggregated service to ene
 * Make a virtual environment: `python3.8 -m venv bvp-venv` or use a different tool like `mkvirtualenv`. You can also use
   an [Anaconda distribution](https://conda.io/docs/user-guide/tasks/manage-environments.html) as base with `conda create -n bvp-venv python=3.8`.
 * Activate it, e.g.: `source bvp-venv/bin/activate`
-* Install the `bvp` platform and dependencies:
+* Install dependencies and the `bvp` platform itself:
 
-      python setup.py [develop|install]
+      pip install pip-tools
+      pip-sync
+      python setup.py develop
 
 
 

bitbucket-pipelines.yml

@@ -16,6 +16,7 @@ pipelines:
           - apt-get update
           - apt-get -y install postgresql-client coinor-cbc
           - PGPASSWORD=a1test psql -h localhost -p 5432 -c "create extension if not exists cube; create extension if not exists earthdistance;" -U a1test a1test;
+          - make install-deps
           - python setup.py test
         services:
           - postgres
@@ -29,6 +30,7 @@ pipelines:
             - apt-get update
             - apt-get -y install postgresql-client coinor-cbc
             - PGPASSWORD=a1test psql -h localhost -p 5432 -c "create extension if not exists cube; create extension if not exists earthdistance;" -U a1test a1test;
+            - make install-deps
             - python setup.py test
           caches:
             - pip

ci/README.md

@@ -1,6 +1,6 @@
 # WSGI configuration
 
-Here is an example hot to serve this application as WSGI app:
+Here is an example how to serve this application as WSGI app:
 
 
     # This file contains the WSGI configuration required to serve up your
@@ -45,7 +45,7 @@ Git remote is added and the code is pushed. The below step pushes to the BVP sta
 ```
 git remote add pythonanywhere seita@ssh.pythonanywhere.com:/home/seita/bvp-staging
 
-git push -u pythonanywhere $BITBUCKET_BRANCH
+git push --follow-tags -u pythonanywhere $BITBUCKET_BRANCH
 ```
 
 # Install Post-Receive Hook
@@ -55,7 +55,7 @@ in the repo where you wish to deploy the final code. This will be triggered when
 push is received by the Bitbucket repo.
 
 The script below can be a Post Receive Hook (save as `hooks/post-receive` in your remote origin repo and update paths).
-It will force checkout the master branch,update dependencies, upgrade the database structure,
+It will force checkout the master branch, update dependencies, upgrade the database structure,
 update the documentation and finally touch the wsgi.py file.
 This last step is documented by PythonAnywhere as a way to soft restart the running application.
 
@@ -69,17 +69,19 @@ echo "CHECKING OUT CODE TO GIT WORK TREE ($PATH_TO_GIT_WORK_TREE) ..."
 GIT_WORK_TREE=$PATH_TO_GIT_WORK_TREE git checkout -f
 
 cd $PATH_TO_GIT_WORK_TREE
-PATH=$PATH_TO_VENV/bin:$PATH                                                                                                                                                                                  
+PATH=$PATH_TO_VENV/bin:$PATH
 
 echo "INSTALLING DEPENDENCIES ..."
-python setup.py develop
+make install-deps
+
+echo "INSTALLING BVP ..."
+make install-bvp
 
 echo "UPGRADING DATABASE STRUCTURE ..."
-flask db upgrade
+make upgrade-db
 
 echo "UPDATING DOCUMENTATION ..."
-pip install sphinx sphinxcontrib.httpdomain
-cd documentation; make clean; make html; cd ..
+make update-db
 
 echo "RESTARTING APPLICATION ..."
 touch $PATH_TO_WSGI

requirements.txt

@@ -0,0 +1,94 @@
+#
+# This file is autogenerated by pip-compile
+# To update, run:
+#
+#    pip-compile
+#
+--extra-index-url https://packagecloud.io/github/git-lfs/pypi/simple
+
+alembic==1.4.2            # via flask-migrate
+appdirs==1.4.4            # via pyomo
+arrow==0.15.7             # via rq-dashboard
+babel==2.8.0              # via flask-babelex
+bcrypt==3.1.7             # via bvp (setup.py)
+blinker==1.4              # via flask-mail, flask-principal
+bokeh==1.0.4              # via bvp (setup.py), pandas-bokeh
+certifi==2020.6.20        # via requests
+cffi==1.14.0              # via bcrypt
+chardet==3.0.4            # via requests
+click==7.1.2              # via bvp (setup.py), flask, rq
+colour==0.1.5             # via bvp (setup.py)
+cycler==0.10.0            # via matplotlib
+dnspython==1.16.0         # via email-validator
+email-validator==1.1.1    # via bvp (setup.py)
+flask-babelex==0.9.4      # via flask-security
+flask-classful==0.14.2    # via bvp (setup.py)
+flask-json==0.3.4         # via bvp (setup.py)
+flask-login==0.4.1        # via bvp (setup.py), flask-security
+flask-mail==0.9.1         # via bvp (setup.py), flask-security
+flask-marshmallow==0.13.0  # via bvp (setup.py)
+flask-migrate==2.5.3      # via bvp (setup.py)
+flask-principal==0.4.0    # via flask-security
+flask-security==3.0.0     # via bvp (setup.py)
+flask-sqlalchemy==2.4.3   # via bvp (setup.py), flask-migrate
+flask-sslify==0.1.5       # via bvp (setup.py)
+flask-wtf==0.14.3         # via bvp (setup.py), flask-security
+flask==1.1.2              # via bvp (setup.py), flask-babelex, flask-classful, flask-json, flask-login, flask-mail, flask-marshmallow, flask-migrate, flask-principal, flask-security, flask-sqlalchemy, flask-sslify, flask-wtf, rq-dashboard
+forecastiopy==0.22        # via bvp (setup.py)
+humanize==2.5.0           # via bvp (setup.py)
+idna==2.10                # via email-validator, requests
+inflect==4.1.0            # via bvp (setup.py)
+inflection==0.5.0         # via bvp (setup.py)
+iso8601==0.1.12           # via bvp (setup.py)
+isodate==0.6.0            # via bvp (setup.py)
+itsdangerous==1.1.0       # via flask, flask-security, flask-wtf
+jinja2==2.11.2            # via bokeh, flask, flask-babelex
+joblib==0.16.0            # via scikit-learn
+kiwisolver==1.2.0         # via matplotlib
+mako==1.1.3               # via alembic
+markupsafe==1.1.1         # via jinja2, mako, wtforms
+marshmallow-sqlalchemy==0.23.1  # via bvp (setup.py)
+marshmallow==3.6.1        # via flask-marshmallow, marshmallow-sqlalchemy
+matplotlib==3.2.2         # via timetomodel
+nose==1.3.7               # via pyutilib
+numpy==1.19.0             # via bokeh, bvp (setup.py), matplotlib, pandas, patsy, scikit-learn, scipy, statsmodels, timetomodel
+packaging==20.4           # via bokeh
+pandas-bokeh==0.4.3       # via bvp (setup.py)
+pandas==1.0.5             # via bvp (setup.py), pandas-bokeh, statsmodels, timetomodel
+passlib==1.7.2            # via flask-security
+patsy==0.5.1              # via statsmodels
+pillow==7.2.0             # via bokeh
+ply==3.11                 # via pyomo
+pscript==0.7.4            # via bvp (setup.py)
+psycopg2-binary==2.8.5    # via bvp (setup.py)
+py3dns==3.2.1             # via bvp (setup.py)
+pycparser==2.20           # via cffi
+pyomo==5.7                # via bvp (setup.py)
+pyparsing==2.4.7          # via matplotlib, packaging
+pysolar==0.8              # via bvp (setup.py)
+python-dateutil==2.8.1    # via alembic, arrow, bokeh, matplotlib, pandas, timetomodel
+python-dotenv==0.14.0     # via bvp (setup.py)
+python-editor==1.0.4      # via alembic
+pytz==2020.1              # via babel, bvp (setup.py), pandas, timetomodel, tzlocal
+pyutilib==6.0.0           # via pyomo
+pyyaml==5.3.1             # via bokeh
+redis==3.5.3              # via rq, rq-dashboard
+requests==2.24.0          # via forecastiopy
+rq-dashboard==0.6.1       # via bvp (setup.py)
+rq==1.4.3                 # via bvp (setup.py), rq-dashboard
+scikit-learn==0.23.1      # via sklearn
+scipy==1.5.1              # via scikit-learn, statsmodels, timetomodel
+six==1.15.0               # via bcrypt, bokeh, cycler, flask-marshmallow, isodate, packaging, patsy, pyomo, python-dateutil, pyutilib
+sklearn==0.0              # via timetomodel
+speaklater==1.3           # via flask-babelex
+sqlalchemy==1.3.18        # via alembic, flask-sqlalchemy, marshmallow-sqlalchemy, timetomodel
+statsmodels==0.11.1       # via timetomodel
+threadpoolctl==2.1.0      # via scikit-learn
+timetomodel==0.6.8        # via bvp (setup.py)
+tornado==6.0.4            # via bokeh
+tzlocal==2.1              # via bvp (setup.py)
+urllib3==1.25.9           # via requests
+validate-email==1.3       # via bvp (setup.py)
+werkzeug==1.0.1           # via flask
+wtforms==2.3.1            # via flask-wtf
+xlrd==1.2.0               # via bvp (setup.py)