Changelog

Unreleased

Added

Include description of the egon-data workflow in our documentation #23
There's now a wrapper around subprocess.run in egon.data.subprocess.run. This wrapper catches errors better and displays better error messages than Python's built-in function. Use this wrapper wenn calling other programs in Airflow tasks.
You can now override the default database configuration using command line arguments. Look for the switches starting with --database in egon-data --help. See PR #159 for more details.
Docker will not be used if there is already a service listening on the HOST:PORT combination configured for the database.
You can now supply values for the command line arguments for egon-data using a configuration file. If the configuration file doesn't exist, it will be created by egon-data on it's first run. Note that the configuration file is read from and written to the directtory in which egon-data is started, so it's probably best to run egon-data in a dedicated directory. There's also the new function egon.data.config.settings which returns the current configuration settings. See PR #159 for more details.
You can now use tasks which are not part of a Dataset, i.e. which are unversioned, as dependencies of a dataset. See PR #318 for more details.
You can now force the tasks of a Dataset to be always executed by giving the version of the Dataset a ".dev" suffix. See PR #318 for more details.
OSM data import as done in open_ego #1 which was updated to the latest long-term data set of the 2021-01-01 in #223 <#223>`_
Verwaltungsgebiete data import (vg250) more or less done as in open_ego #3
Zensus population data import #2
Zensus data import for households, apartments and buildings #91
DemandRegio data import for annual electricity demands #5
Download cleaned open-MaStR data from Zenodo #14
NEP 2021 input data import #45
Option for running workflow in test mode #112
Abstraction of hvmv and ehv substations #9
Filter zensus being inside Germany and assign population to municipalities #7
RE potential areas data import #124
Heat demand data import #101
Demographic change integration #47
Creation of voronoi polygons for hvmv and ehv substations #9
Add hydro and biomass power plants eGon2035 #127
Creation of the ehv/hv grid model with osmTGmod, see issue #4 and PR #164
Identification of medium-voltage grid districts #10
Distribute electrical demands of households to zensus cells #181
Distribute electrical demands of cts to zensus cells #210
Include industrial sites' download, import and merge #117
Integrate scenario table with parameters for each sector #177
The volume of the docker container for the PostgreSQL database is saved in the project directory under docker/database-data. The current user ($USER) is owner of the volume. Containers created prior to this change will fail when using the changed code. The container needs to be re-created. #228
Extract landuse areas from OSM #214
Integrate weather data and renewable feedin timeseries #19
Create and import district heating areas #162
Integrate electrical load time series for cts sector #109
Assign voltage level and bus_id to power plants #15
Integrate solar rooftop for etrago tables #255
Integrate gas bus and link tables #198
Integrate data bundle #272
Add household electricity demand time series, mapping of demand profiles to census cells and aggregated household electricity demand time series at MV grid district level #256
Integrate power-to-gas installation potential links #293
Integrate distribution of wind onshore and pv ground mounted generation #146
Integrate dynamic line rating potentials #72
Integrate gas voronoi polygons #308
Integrate supply strategies for individual and district heating #232
Integrate gas production #321
Integrate industrial time series creation #237
Merge electrical loads per bus and export to etrago tables #328
Insert industial gas demand #321
Integrate existing CHP and extdended CHP > 10MW_el #266
Add random seed to CLI parameters #351
Extend zensus by a combined table with all cells where there's either building, apartment or population data #359
Add example metadata for OSM, VG250 and Zensus VG250. Add metadata templates for licences, context and some helper functions. Extend docs on how to create metadata for tables. #139
Integrate DSM potentials for CTS and industry #259
Assign weather cell id to weather dependant power plants #330
Distribute wind offshore capacities #329
Add CH4 storages #405

Changed

Adapt structure of the documentation to project specific requirements #20
Switch from Travis to GitHub actions for CI jobs #92
Rename columns to id and zensus_population_id in zensus tables #140
Revise docs CONTRIBUTING section and in particular PR guidelines #88 and #145
Drop support for Python3.6 #148
Improve selection of zensus data in test mode #151
Delete tables before re-creation and data insertation #166
Adjust residential heat demand in unpopulated zenus cells #167
Introduce mapping between VG250 municipalities and census cells #165
Delete tables if they exist before re-creation and data insertation #166
Add gdal to pre-requisites #185
Update task zensus-inside-germany #196
Update installation of demandregio's disaggregator #202
Update etrago tables #243and #285
Migrate VG250 to datasets #283
Allow configuring the airflow port #281
Migrate mastr, mv_grid_districts and re_potential_areas to datasets #297
Migrate industrial sites to datasets #237
Rename etrago tables from e.g. egon_pf_hv_bus to egon_etrago bus etc. #334
Move functions used by multiple datasets #323
Migrate scenario tables to datasets #309
Migrate weather data and power plants to datasets #314
Create and fill table for CTS electricity demand per bus #326
Migrate osmTGmod to datasets #305
Filter osm landuse areas, rename industrial sites tables and update load curve function #378
Remove version columns from eTraGo tables and related code #384
Remove country column from scenario capacities table #391
Update version of zenodo download #397
Rename columns gid to id #169
Remove upper version limit of pandas #383
Use random seed from CLI parameters for CHP and society prognosis functions #351
Changed demand.egon_schmidt_industrial_sites - table and merged table (industrial_sites) #423
Replace 'gas' carrier with 'CH4' and 'H2' carriers #436
Adjust file path for industrial sites import #397
Rename columns subst_id to bus_id #335
Apply black and isort for all python scripts #463

Bug fixes

Heat demand data import #157
Substation sequence #171
Adjust names of demandregios nuts3 regions according to nuts version 2016 #201
Delete zensus buildings, apartments and households in unpopulated cells #202
Fix input table of electrical-demands-zensus #217
Import heat demand raster files successively to fix import for dataset==Everything #204
Replace wrong table name in SQL function used in substation extraction #236
Fix osmtgmod for osm data from 2021 by updating substation in Garenfeld and set srid #241 #258
Adjust format of voltage levels in hvmv substation #248
Change order of osmtgmod tasks #253
Fix missing municipalities #279
Fix import of hydro power plants #270
Fix path to osm-file for osmtgmod_osm_import #258
Fix conflicting docker containers by setting a project name #289
Update task insert-nep-data for pandas version 1.3.0 #322
Fix versioning conflict with mv_grid_districts #340
Set current working directory as java's temp dir when executing osmosis #344
Fix border gas voronoi polygons which had no bus_id #362
Add dependency from WeatherData to Vg250 #387
Fix unnecessary columns in normal mode for inserting the gas production #387
Add xlrd and openpyxl to installation setup #400
Store files of OSM, zensus and VG250 in working dir #341
Remove hard-coded slashes in file paths to ensure Windows compatibility #398
Add missing dependency in pipeline.py #412
Add prefix egon to MV grid district tables #349
Bump MV grid district version no #432
Add curl to prerequisites in the docs #440
Replace NAN by 0 to avoid empty p_set column in DB #414
Exchange bus 0 and bus 1 in Power-to-H2 links #458
Add data_bundle to industrial_sites task dependencies #468