-
Notifications
You must be signed in to change notification settings - Fork 34
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into export-to-csv
Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
- Loading branch information
Showing
13 changed files
with
247 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
--- | ||
name: build-docker-image | ||
|
||
on: | ||
pull_request: | ||
types: | ||
- opened | ||
- synchronize | ||
push: | ||
branches: | ||
- main | ||
|
||
jobs: | ||
build: | ||
name: Build Docker Image | ||
runs-on: ubuntu-latest | ||
services: | ||
postgres: | ||
env: | ||
POSTGRES_DB: flexmeasures_test | ||
POSTGRES_PASSWORD: flexmeasures_test | ||
POSTGRES_USER: flexmeasures_test | ||
image: postgres:latest | ||
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s | ||
--health-retries 5 | ||
ports: | ||
- 5432:5432 | ||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v3 | ||
- name: Build Docker Image | ||
run: docker build -t flexmeasures:latest -f Dockerfile . | ||
- name: Generate random secret key | ||
run: echo "SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex(24))')" | ||
>> .env | ||
- name: Export SQLALCHEMY_DATABASE_URI | ||
run: echo "SQLALCHEMY_DATABASE_URI=postgresql://flexmeasures_test:flexmeasures_test@127.0.0.1:5432/flexmeasures_test" | ||
>> .env | ||
- name: Keep running flexmeasures container in background | ||
run: docker run -t -d --env-file .env --network=host --name fm-container flexmeasures:latest | ||
- name: Execute database upgrade | ||
run: docker exec --env-file .env fm-container flexmeasures | ||
db upgrade | ||
- name: Add toy user | ||
run: docker exec --env-file .env fm-container flexmeasures | ||
add toy-account | ||
- name: Generate prices dummy data | ||
run: .github/workflows/generate-dummy-price.sh | ||
- name: Copy prices dummy data | ||
run: docker cp prices-tomorrow.csv fm-container:/app/prices-tomorrow.csv | ||
- name: Add beliefs | ||
run: docker exec --env-file .env fm-container flexmeasures | ||
add beliefs --sensor-id 1 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam | ||
- name: Export TOMORROW | ||
run: echo "TOMORROW=$(date --date="next day" '+%Y-%m-%d')" | ||
>> $GITHUB_ENV | ||
- name: Add schedule | ||
run: docker exec --env-file .env fm-container flexmeasures | ||
add schedule for-storage --sensor-id 2 --consumption-price-sensor 1 | ||
--start ${TOMORROW}T07:00+01:00 --duration PT12H | ||
--soc-at-start 50% --roundtrip-efficiency 90% |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
#!/bin/bash | ||
|
||
set -e | ||
set -x | ||
|
||
TOMORROW=$(date --date="next day" '+%Y-%m-%d') | ||
|
||
echo "Hour,Price | ||
${TOMORROW}T00:00:00,10 | ||
${TOMORROW}T01:00:00,11 | ||
${TOMORROW}T02:00:00,12 | ||
${TOMORROW}T03:00:00,15 | ||
${TOMORROW}T04:00:00,18 | ||
${TOMORROW}T05:00:00,17 | ||
${TOMORROW}T06:00:00,10.5 | ||
${TOMORROW}T07:00:00,9 | ||
${TOMORROW}T08:00:00,9.5 | ||
${TOMORROW}T09:00:00,9 | ||
${TOMORROW}T10:00:00,8.5 | ||
${TOMORROW}T11:00:00,10 | ||
${TOMORROW}T12:00:00,8 | ||
${TOMORROW}T13:00:00,5 | ||
${TOMORROW}T14:00:00,4 | ||
${TOMORROW}T15:00:00,4 | ||
${TOMORROW}T16:00:00,5.5 | ||
${TOMORROW}T17:00:00,8 | ||
${TOMORROW}T18:00:00,12 | ||
${TOMORROW}T19:00:00,13 | ||
${TOMORROW}T20:00:00,14 | ||
${TOMORROW}T21:00:00,12.5 | ||
${TOMORROW}T22:00:00,10 | ||
${TOMORROW}T23:00:00,7" > prices-tomorrow.csv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
ARG PYTHON_VERSION | ||
|
||
FROM python:${PYTHON_VERSION}-slim-bookworm as update | ||
|
||
# Install dependencies | ||
|
||
RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
build-essential \ | ||
git \ | ||
&& rm -rf /var/lib/apt/lists/* | ||
|
||
# Copy the source code | ||
|
||
COPY . /app | ||
WORKDIR /app | ||
|
||
CMD ["python", "--version"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
#!/bin/bash | ||
|
||
###################################################################### | ||
# This script sets up the docker environment for updating packages | ||
###################################################################### | ||
|
||
set -e | ||
set -x | ||
|
||
# Check if docker is installed | ||
if ! [ -x "$(command -v docker)" ]; then | ||
echo "Docker is not installed. Please install docker and try again." | ||
exit 1 | ||
fi | ||
|
||
# Check if we can run docker without sudo | ||
if ! docker ps > /dev/null 2>&1; then | ||
echo "Docker is not running without sudo. Please add your user to the docker group and try again." | ||
echo "You may use the following command to do so:" | ||
echo "sudo usermod -aG docker $USER" | ||
echo "You will need to log out and log back in for this to take effect." | ||
exit 1 | ||
fi | ||
|
||
SOURCE_DIR=$(pwd)/../ | ||
|
||
TEMP_DIR=$(mktemp -d) | ||
|
||
# Copy the build files to the temp directory | ||
cp -r ../ci $TEMP_DIR/ci | ||
cp -r ../requirements $TEMP_DIR/requirements | ||
cp -r ../Makefile $TEMP_DIR | ||
|
||
cd $TEMP_DIR | ||
|
||
PYTHON_VERSIONS=(3.8 3.9 3.10 3.11) | ||
|
||
for PYTHON_VERSION in "${PYTHON_VERSIONS[@]}" | ||
do | ||
# Check if container exists and remove it | ||
docker container inspect flexmeasures-update-packages-$PYTHON_VERSION > /dev/null 2>&1 && docker rm --force flexmeasures-update-packages-$PYTHON_VERSION | ||
# Build the docker image | ||
docker build --build-arg=PYTHON_VERSION=$PYTHON_VERSION -t flexmeasures-update-packages:$PYTHON_VERSION . -f ci/Dockerfile.update | ||
# Build flexmeasures | ||
# We are disabling running tests here, because we only want to update the packages. Running tests would require us to setup a postgres database, | ||
# which is not necessary for updating packages. | ||
docker run --name flexmeasures-update-packages-$PYTHON_VERSION -it flexmeasures-update-packages:$PYTHON_VERSION make upgrade-deps skip-test=yes | ||
# Copy the requirements to the source directory | ||
docker cp flexmeasures-update-packages-$PYTHON_VERSION:/app/requirements/$PYTHON_VERSION $SOURCE_DIR/requirements/ | ||
# Remove the container | ||
docker rm flexmeasures-update-packages-$PYTHON_VERSION | ||
# Remove the image | ||
docker rmi flexmeasures-update-packages:$PYTHON_VERSION | ||
done | ||
|
||
# Clean up docker builder cache | ||
echo "You can clean up the docker builder cache with the following command:" | ||
echo "docker builder prune --all -f" | ||
|
||
# Remove the temp directory | ||
rm -rf $TEMP_DIR |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
Dependency Management | ||
======================= | ||
|
||
Requirements | ||
------------- | ||
|
||
FlexMeasures is built on the shoulder of giants, namely other open source libraries. | ||
Look into the `requirements` folder to see what is required to run FlexMeasures (`app.in`) or to test it, or to build this documentation. | ||
|
||
The `.in` files specify our general demands, and in `.txt` files, we keep a set of pinned dependency versions, so we can all work on the same background (crucial to compare behavior of installations to each other). | ||
|
||
To install these pinned requirements, run: | ||
|
||
.. code-block:: bash | ||
$ make install-for-dev | ||
Check out `Makefile` for other useful commands, but this should get you going. | ||
|
||
To upgrade the pinned versions, we can run: | ||
|
||
|
||
.. code-block:: bash | ||
$ make upgrade-deps | ||
Python versions | ||
---------------- | ||
|
||
In addition, we support a range of Python versions (as you can see in the `requirements` folder. | ||
|
||
Now ― you probably have only one Python version installed. Let's say you add a dependency, or update the minimum required version. How to update the pinned sets of requirements across all Python versions? | ||
|
||
.. code-block:: bash | ||
$ cd ci; ./update-packages.sh | ||
This script will use docker to do these upgrades per Python version. | ||
|
||
Still, we'd also like to be able to test FlexMeasures across all these versions. | ||
We've added that capability to our CI pipeline (GitHub Actions), so you could clone it an make a PR, in order to run them. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
from typing import Any | ||
|
||
from altair.vegalite.schema import Interpolate | ||
|
||
|
||
def validate_special_attributes(key: str, value: Any): | ||
"""Validate attributes with a special meaning in FlexMeasures.""" | ||
if key == "interpolate": | ||
Interpolate.validate(value) |