This project contains all information necesarry to run the open source ESDL MapEditor and ESSIM toolsuite on your local machine or cloud infrastructure.
This software stack consists of the following open source components with the link to their github repositories:
- ESDL MapEditor and ESSIM
- Contents
- What can I do with the MapEditor and ESSIM?
- What is ESDL?
- Features
- The architecture of the Toolsuite
- Setting up and running the software stack
- Prerequisites
- Steps to follow
- Step 1. Configuring and starting the software for the base infrastructure
- Step 2. Creating user accounts
- Step 3. Configure role based access control for ESSIM dashboard
- Step 4. Create an API key in Grafana for the Panel Service
- Step 5. Start the MapEditor and ESSIM
- Step 6. Log in to the ESDL MapEditor
- Step 7. Upload some profiles
- ESDL MapEditor and ESSIM Tutorials
- Cloud deployment
- Details
- License
The ESDL MapEditor is a map-based energy system editor. You can use it to create ESDL based energy system descriptions, that can then be simulated with a growing number of ESDL-capable simulators, or processed with ESDL-capable tools. ESSIM is our ESDL based energy system simulator that gives insights in the hourly energy (im)balance of an energy system described in the ESDL language.
To get an impression of what the tools can do and how they look like, have look at the tutorials.
ESDL is a modelling language created to describe complete (hybrid) energy systems in one uniform format. It allows to describe information about the individual energy system components, how they are connected, how they are used (e.g. using energy production or consumption profiles), where they are physically located (on the map), what they cost (now and in future). Furthermore information about buildings in an area, energy potential, KPIs (on buildings, areas, or any assets) can be described. Possible applications are facilitating interoperability between different energy transition models and publishing open data on energy systems.
Click here for the ESDL documentation website
Click here for the ESDL class documentation website (a clickable ESDL model)
Click here for the ESDL github website
- Draw an energy system scenario by dragging and dropping energy assets on a map
- Connect components, set typical component characteristics (installed power, efficiencies, costs)
- Attach power or energy profiles to assets (demand and production profiles)
- Set control strategies (specifically needed for ESSIM simulations)
- Query external ESDL data sources, for example for solar or wind potential
- Visualize WMS layers with information that can be used to define your scenario
- Convert shapefiles into ESDL assets
- Query the boundary service for area borders (provinces, municipalities)
- Visualize results of simulations on the map (KPI dashboards, load animations, color areas based on KPI outcomes, load duration curves)
Click here for some more information on the ESDL MapEditor
- Simulates energy systems defined in ESDL and calculates energy balance over time
- Calculates optimal schedule of flexible producers and the effect of this schedule in terms of emissions, costs, load on the network
- Calculates schedules for conversion and storage in a similar manner
Click here for some more information on ESSIM
The architecture of the toolsuite provided is shown here (the gray components are open source solutions provided by others):
It consists of the following functionalities:
- ESDL MapEditor: map based scenario editor
- ESSIM: energy system simulator, to calculate hourly energy balance and give insights in effects of conversion and storage
- Identity & Access Management: using Keycloak, an open-source IAM solution, it provides user management, roles, groups, organisations, authentication and authorisation, role based access control
- ESDL Drive: cloud storage for ESDL files, with access control at user, group or organisation level (supports git-like versioning)
- Panel Service: service to create graphs from InfluxDB time series data
- Boundary Service: service that gives boundary information for provinces, municipalities, neighbourhoods, and so on
- Grafana: open-source analytics & monitoring solution for every database
- InfluxDB: open-source time series database solution
- PostgresDB with PostGIS extension: open-source relational database management system
- MongoDB: open-source document-oriented database program (NoSQL)
Although the software was designed to run in a hosted environment somewhere in the cloud or in your in-company datacenter, the software can be run on a local laptop or PC as well. The following steps describe the installation process on a local machine. In the cloud deployment chapter we'll give some directions for cloud deployment
The current stack uses docker and docker compose. The tested versions are currently:
| Software | Version |
|---|---|
| Docker engine | 26.1.3 |
| Docker Compose | 2.6.1 |
- Step 1. Configuring and starting the software for the base infrastructure
- Step 2. Creating user accounts
- Step 3. Configure role based access control for ESSIM dashboard
- Step 4. Create an API key in Grafana for the Panel Service
- Step 5. Start the MapEditor and ESSIM
- Step 6. Log in to the ESDL MapEditor
- Step 7. Upload some profiles
NOTE FOR WINDOWS USERS:
When cloning this repository using git for windows, file line endings are automatically converted from LF (Unix style) to CRLF (Windows style). This causes problems for the file BaseInfrastructure/postgres/init-database.sh as it is being mounted in one of the docker containers running linux. Please make sure that this file gets Unix style line endings, by converting it back using for example notepad++ or dos2unix, or configure git in such a way that it doesn't automatically convert line endings to windows style (Search for 'git autocrlf').
To configure the base infrastructure have a look at the .env.template file in de BaseInfrastructure folder and copy it to a .env file and adapt to your situation (defaults should work out of the box for a localhost installation, but are not secure).
Start the base infrastructure (databases, grafana, pgadmin, ...)
cd BaseInfrastructure
docker-compose upNote: use
docker-compose up -dto start the base infrastructure in the background.
Wait for the base infrastructure to be ready.
Using you webbrowser, go to http://localhost:8080/auth to login to keycloak (Note: don't forget the /auth, this is needed for newer Keycloak versions to be compatible with older versions).
Login with the admin credentials as specified in the docker-compose file or your adapted .env file.
Once you're logged in, you see a welcome for the master realm (e.g. to configure other realms, or change the admin password).
Select the ESDL Studio (esdl-mapeditor) realm using the drop-down on the top-left.
Click Users from the menu on the left
Click Add User and fill in the proper user information and click Create. Be sure to fill in an email address!
After clicking Save a number of tabs appear. Go to the Credentials tab, select Set password and set the password of the newly created user in the popup.
Go to the Attributes tab and add an attribute with role as key and with essim as value, and press Save.
The ESSIM dashboard is a Grafana based solution for viewing simulation results. Grafana supports multiple roles: Viewer, Editor and Admin. If you want to give some users other roles than Viewer, add the roles Editor and/or Admin to Keycloak and assign these roles to the appropriate user. When you don't do this, all users get the Viewer rights.
Go to the user that needs to become an Editor or Admin, go to the Role Mapping tab, and press the Assign Role button. A popup is displayed with all possible client roles in the realm. Search for essim-dashboard - Admin or essim-dashboard - Editor and check the role and press Assign to add the Editor or Admin role to the user (select only one).
Log in to Grafana, go to http://localhost:3000 and login with the credentials from the user with Admin rights you've just created.
Select the menu option 'API keys' in the settings menu.
Click the button 'New API key'
Fill in the details for the API key. Choose a name (e.g. 'panel-service') and make sure to give it the 'Admin' role.
Copy the generated API key.
Fill in this API key in the file ./ESDLMapEditor/panel-service.env (Replace the key that is already there).
GRAFANA_API_KEY=eyJrIjoiV3g0Z3pGUUxBNkhucXlySjhCRFczNXZwVXhiREhrRXciLCJuIjoicGFuZWwtc2VydmljZSIsImlkIjoxfQ==Also have a look at the file ./ESDLMapEditor/boundary-service.env and fill in the password from the ./BaseInfrastructure/.env file or do a source ./BaseInfrastructure/.env to configure the database password for the boundary service.
In another terminal window, start ESSIM:
cd ESSIM
docker-compose upIn yet another terminal window, start the ESDL MapEditor and accompanying services:
cd ESDLMapEditor
docker-compose upTo start ESDL Drive storage do the following in another terminal (or use -d option for each docker-compose command, to start in detached mode):
cd ESDLDrive
docker-compose upUsing your webbrowser go to http://localhost:8111
Press Start and log in using the credentials created in the previous step
You should see the following screen now:
The ESDL Mapeditor is ready to be used!
If you're installing this toolsuite to run ESSIM simulations or any other application that requires timeseries data, you need to upload some profiles. For that purpose we've created a profile manager. In the repository there is an example dataset with profiles created from publically available data (NEDU profiles for electricity and gas usage and KNMI solar profile).
NOTE: Recent versions of the ESDL MapEditor are integrated with the Energy Data Repository (EDR) and profiles present in the EDR are now also accessible from the ESDL MapEditor!
Click View and select Settings from the menu. The application settings dialog appears. Click Upload profiles. In the first drop down menu select:
- Personal profiles: to upload profiles that will become available only for the current user
- Standard profiles: to upload profiles that will become available for all users
- Project profiles for ...: to upload profiles related to a project, so that all project members can use these profiles
Drag the file ./Data/Profiles/standard_profiles.csv and drop it in the area indicated in the dialog, or click the button to select a file. Depending on your local machine's regional settings (Use '.' or ',' as the decimal seperator) you might need to choose the other csv file in the same directory.
After the uploading is finished, click Profiles plugin in the menu on the left. On the right side a window appears where you can view and edit the settings of the profiles you've just uploaded.
The profiles can now be used in the simulations.
In order to stop the running services, press, in the reverse order, Ctrl-C in the terminal windows. If you started docker compose up with the -d flag, you can do a
docker compose downin each of the folders.
A docker compose down -v in the BaseInfrastructure folder will also remove the data volumes created (e.g. the data stored in Postgres, Mongo and InfluxDB). This resets and removes all data in case you got stuck somewhere.
Please go here to find five different tutorials that explain how to work with the ESDL MapEditor and ESSIM
In order to run this software stack in a hosted environment, several services must be offered to the end-user:
- The MapEditor frontend: the main entry point for this software stack for end uses
- The ESSIM dashboard: to show ESSIM simulation results
- Keycloak (Identity & Access Management): to facilitate the login process
- The panel service: to visualize the profile data from within the MapEditor
Optionally the following services can be offered too:
- ESDL drive: although the functionality provided from within the MapEditor is more extensive
In our own hosted environment we use traefik as a reverse proxy in front on the above listed services, for two reasons:
- to terminate SSL traffic
- as a reverse proxy: to route HTTP traffic to the right container
Furthermore we use:
- docker swarm: to create a cluster of several virtual machines
- portainer: for container management
- docker registry: to locally push container images and make deployment in the swarm easier
Recently we have also successful setups in Kubernetes. Reach out for more details.
The four (or five) services listed in the beginning of this chapter must be accessible via seperate domain names (using a local or global DNS server) or seperate IP addresses. As the essim dashboard and the panel service both use grafana as their frontend, they could be treated as the same service if that's desirable. We tried running the services using sub-paths (e.g. https://mycompany.com/mapeditor and https://mycompany.com/essim-dashboard) but we were not very successful to get everything up and running.
The following picture shows how a deployment with a reverse proxy
Required changes:
- In
BaseInfrastructure/docker-compose.yml- Find
GF_SERVER_ROOT_URL: changelocalhostto the domain name for the ESSIM dashboard - Find
GF_AUTH_SIGNOUT_REDIRECT_URL: change two (!) occurences oflocalhostto the domain name for keycloak - Find
GF_AUTH_GENERIC_OAUTH_AUTH_URL: changelocalhostto the domain name for keycloak
- Find
- In
BaseInfrastructure/keycloak/esdl-mapeditor-realm.json(or login to keycloak and change using their web-based management interface)- Replace all occurences of
localhost:portto the respective domain names
- Replace all occurences of
- In
ESDLMapEditor/docker-compose.yml- Find
EXTERNAL_GRAFANA_URL: changelocalhostto the domain name for ESDL Mapeditor
- Find
- In
ESDLMapeditor/mapeditor_open_source.env:- Find
PANEL_SERVICE_EXTERNAL_URL: replacelocalhostwith the domain name of the panel service
- Find
- Find the file
./ESDLMapEditor/esdl_mapeditor/client_secrets_opensource.json. This file is mounted inside the container when started and contains the keycloak configuration.- Replace all occurences of
localhostto the respective domain names (one for the mapeditor and 2 for keycloak) - or mount a different version in ./ESDLMapeditor/docker-compose.yml
volumes: - "./esdl-mapeditor/client_secrets_opensource.json:/usr/src/app/credentials/client_secrets_opensource.json"
- Replace all occurences of
- In
BaseInfrastructure/keycloak/esdl-mapeditor-realm.json(or login to keycloak and change using their web-based management interface)- Replace all occurences of
localhost:portto the respective domain names
- Replace all occurences of
- Keycloak is configured to start in
dev-mode (start-dev), allowing it to be used without SSL certificates (meaning that all traffic is insecure). Configure Keycloak to start in 'normal' mode, see the./BaseInfrastructure/docker-compose.ymlfor some hints on setting this up correctly in combination with a reverse proxy that terminates SSL traffic.
| Service | Port | Environment variable |
|---|---|---|
| MapEditor | 8111 | MAPEDITOR_PORT |
| Grafana | 3000 | GRAFANA_PORT |
| Keycloak | 8080 | KEYCLOAK_PORT |
| PGAdmin | 5050 | PGADMIN_PORT |
| Service | Port | Environment variable |
|---|---|---|
| Mongo | 27017 | MONGO_PORT |
| InfluxDB | 8086 | INFLUXDB_PORT |
| PostgresDB | 5432 | POSTGRES_PORT |
| Service | Port | Environment variable |
|---|---|---|
| ESDL Drive | 9080 | - |
| ESDL Drive | 9443 (SSL-port) | - |
| Service | User | Password | Configured in | Comment |
|---|---|---|---|---|
| Keycloak | admin | admin | docker-compose.yml | |
| Grafana | admin | admin | docker-compose.yml | |
| InfluxDB | admin | admin | docker-compose.yml | |
| PostgresDB | postgres | password | docker-compose.yml | |
| PostgresDB | keycloak | password | init-database.sh | |
| PostgresDB | boundary_service | password | init-database.sh | |
| PostgresDB | drive | password | init-database.sh and ESDLDrive/docker-compose.yml | |
| PGAdmin | admin@admin.org | admin | docker-compose.yml |
...
The file ./BaseInfrastructure/postgres/init-databases.sh contains the initialization script for the postgres database.
ESDL Drive exists of 3 components: ESDL Drive (API), CDO-Server (ESDL -> Relational database mapper) and Postgres (Database).
ESDL Drive uses the Postgres database for storage of ESDL files. CDO-Server must be configured with the correct credentials to connect to Postgres database from the base infrastructure. This can be done in the docker-compose.yml of ESDLDrive. The database and 'drive' account are created in the init-databases.sh script
ESDL Drive (API) is secured by KeyCloak and needs some configuration, among others the URL of the ESDL-Mapeditor realm (both internal as external accessible by the browser). This is configuration is defined in ESDLDrive/docker-compose.yml. Environment variables defined in the YAML file are used in server.xml which configures the Open Liberty server that is packaged in the container in /servers/esdl-drive-server.
=======
As ESDL is being updated frequently to support more use cases, the database schema of the ESDL Drive needs to be updated accordingly. If you set AUTO_UPDATE=1 in the environmental variables of cdo-server, it will try to auto-update the schema, but no guarantees are given. Please backup your postgres database before migration, or use the esdl-drive tool to extract all ESDL-files from the drive before upgrading, to make sure you have a good backup.
See the Data migration explanation for more information.
MapEditor and ESSIM are distributed under the Apache 2.0 License.
