PhaenoNet Cloud Functions

Data Processing Overview

The image can be modified in Excalidraw/Google Cloud Architecture Diagramming Tool

Develop in GitHub Codespaces

Launch the Codespace in GitHub using the codespace branch. It will checkout the master branch, initialize the submodules, and rebuild the container. This process may take some minutes to finish.

Deploy Cloud Functions

Use Github Action to manage cloud functions.

Set export scheduler

gcloud scheduler jobs update pubsub import_meteoswiss_data --project $PROJECT --schedule="5 1 * * *" --topic="import_meteoswiss_data" --message-body="none" --time-zone="Europe/Zurich" --description="Trigger cloud function to import meteoswiss data"

Update phaenonet-test data

Instructions to update data-sets in the Firestore test instance from production.

Production copyback

Use Github Action to create a clean test environment from the last production backup.

Copy data partial

Check if cloud functions should be deployed or not depending on the use-case and what data is imported. Alternatively there is an UI available in Firestore.

gcloud --project=phaenonet --account=firestore-backup@phaenonet.iam.gserviceaccount.com firestore export gs://phaenonet-[backup-daily|backup-archive]/[backup-folder] --collection-ids=[collection_ids]
gcloud --project=phaenonet-test --account=firestore-backup@phaenonet-test.iam.gserviceaccount.com firestore import gs://phaenonet-[backup-daily|backup-archive]/[backup-folder] --collection-ids=[collection_ids]

Execute GitHub actions localy

Use act -j <job>

Upgrade actions

Update python version

1. update and rebuild, publish docker images
2. rebuild devcontainer
3. update Pipfile on minor version change & rebuild environment
4. update RUNTIME variable in deploy-functions.yml
5. update container versions in main.yml (2x)

Rebuild pipenv on minor update

Edit Pipfile and set new version

pipenv --rm
PIPENV_VENV_IN_PROJECT=1 pipenv update
pipenv sync --dev

Related resources

• phaenonet-client
• data model documentation