Desaggregate heat pumps #905

nailend · 2022-09-05T09:16:08Z

Fixes #903 .

Necessary branches/ merged to dev yet:

features/#962-final-buildings-mapping
fixes/#985-create-unique-ids-on-generated-tasks
features/#684-distribute-pv-rooftop-buildings-3
fixes/#1031-fix-weather-cells-mapping
fixes/#989-mismatch-cts-bus-id
features/#987-add-distribution-rural-heatpumps

TODO:

module docstring @birgits
is it ok that SH run is not possible to be run on a local computer RAM < 64gb
check DE run results @birgits

Before merging into `dev`-branch, please make sure that

the CHANGELOG.rst was updated.
new and adjusted code is formated using black and isort.
the Dataset-version is updated when existing datasets are adjusted.
the branch was merged into the
continuous-integration/run-everything-over-the-weekend-branch.
the workflow is running successful in test mode.
the workflow is running successful in Everything mode.

…-desaggregate-heat-pumps

…capacity before rerunning the task

…exports minimum heat pump capacity to csv file

…ry usage

nesnoj · 2022-11-30T09:48:21Z

src/egon/data/datasets.yml

@@ -1092,6 +1092,9 @@ emobility_mit:
    export_results_to_csv: True
    parallel_tasks: 10

+demand_timeseries_mvgd:
+  parallel_tasks: 10


We experienced more stable runs with only 5 tasks - change for now until #1034 is solved @birgits ?

Sure, I can change it!

ClaraBuettner · 2022-11-30T15:33:01Z

Hi @nailend,
I need to add a dependency from the new task that you have added here to one of my tasks on another feature-branch. I talked to @birgits and she said that I can push this into this branch.
I tried to do so, but already before my changes I can not start airflow on this branch:

Traceback (most recent call last):
  File "/home/clara/test-egon-data/venv/lib/python3.8/site-packages/airflow/models/dagbag.py", line 256, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/clara/test-egon-data/eGon-data/src/egon/data/airflow/dags/pipeline.py", line 51, in <module>
    from egon.data.datasets.heat_supply import HeatSupply
  File "/home/clara/test-egon-data/eGon-data/src/egon/data/datasets/heat_supply/__init__.py", line 12, in <module>
    from egon.data.datasets.heat_supply.individual_heating import (
  File "/home/clara/test-egon-data/eGon-data/src/egon/data/datasets/heat_supply/individual_heating.py", line 204, in <module>
    from egon.data.datasets.electricity_demand_timeseries.mapping import (
ModuleNotFoundError: No module named 'egon.data.datasets.electricity_demand_timeseries.mapping'

Do you already know about this problem?

birgits · 2022-11-30T16:12:22Z

@ClaraBuettner that is because you also need another branch. I usually merge this branch into the current CI that contains all other branches you need and test it then.

…rago

This reverts commit 3111d68.

nesnoj · 2022-12-12T13:23:49Z

@birgits I've set the no of parallel connections back to 10 in 7b7aeb1 by reverting 3111d68.

PS: A merge of dev into this branch would do it good :)

nesnoj · 2022-12-23T01:27:54Z

[ ] check DE run results @birgits

Have you done this @birgits?

I started to merge dev into this branch but gave up at cts_buildings.py as it was too freaky o_O ....I think you gotta do this

birgits · 2023-01-05T13:51:47Z

Yup, I checked the DE results and they look good!

nailend · 2023-01-10T06:44:30Z

I started to merge dev into this branch but gave up at cts_buildings.py as it was too freaky o_O ....I think you gotta do this

Will take care today!

nailend · 2023-01-10T13:49:22Z

@nesnoj @birgits conflicts are solved. But #986 is still missing in the dev... @gnn are you done with it? from 5e5946f I assume, it might already have passed it final test?!

birgits · 2023-01-23T18:46:13Z

@gnn merged his stuff. @nailend do you need to merge the dev again or can @nesnoj start his review?

nailend · 2023-01-23T18:52:07Z

@gnn merged his stuff.

Are you sure ? The PR #986 is still open...

birgits · 2023-01-23T18:56:05Z

No, sorry, I haven't checked it, it's just what @gnn told me... @gnn can you please check if #986 still needs to be merged/closed whatever?!

nesnoj · 2023-01-26T11:33:55Z

No, sorry, I haven't checked it, it's just what @gnn told me... @gnn can you please check if #986 still needs to be merged/closed whatever?!

That one has been merged, thanks @gnn .

nesnoj · 2023-01-26T11:35:30Z

@gnn merged his stuff. @nailend do you need to merge the dev again or can @nesnoj start his review?

I'm afraid that I do not have the time for a decent review of the HP disaggregation :(

ClaraBuettner · 2023-01-31T12:50:03Z

I have a PR that needs the changes of this branch (#1027) and I would like to merge it. Do you have an idea when this PR will be merged @nesnoj ?

nesnoj · 2023-02-01T14:59:25Z

I have a PR that needs the changes of this branch (#1027) and I would like to merge it. Do you have an idea when this PR will be merged @nesnoj ?

No, but @birgits @nailend ?

birgits added 2 commits September 5, 2022 10:21

Add first draft of heat pump desaggregation

ed55038

Minor changes

42f79ef

nailend added the 🏗️ integration Integrating a data processing step label Sep 5, 2022

nailend self-assigned this Sep 5, 2022

nailend and others added 19 commits September 5, 2022 12:26

Add scenario specific wrapper functions

8b6107b

Black

997dfcc

Add table for EgonEtragoTimeSeriesIndividualHeating

4e17813

Add db export template

42d3513

Merge branch 'features/#671-assign-cts-to-buildings' into features/#903…

c9e46df

…-desaggregate-heat-pumps

Disable pv_rooftop import

67ca91e

Add seperate datasets for each scenario

a2a999e

Add further dependencies

a1804fd

Rename db table

0f61ed9

Add cts heat demand

dc5d39c

Remove sanity check

f575316

Use different peak load for each scenario

5b5304a

Add pass

7203cb4

Devide disaggregation function

8fc36b8

Merge branch 'features/#671-assign-cts-to-buildings' into features/#903…

e712c7e

…-desaggregate-heat-pumps

Update function name

c784bb0

Black

5f28eb5

Use sqla adapter for int

b4ac9ff

Update function name

ec13661

nailend marked this pull request as draft September 12, 2022 08:07

nailend marked this pull request as ready for review September 12, 2022 10:27

nailend marked this pull request as draft September 12, 2022 10:27

nailend and others added 4 commits September 19, 2022 18:40

Move and fix adapter bug

d8319a6

Add timeit decorator

c3ba422

Use timeit decorator

a7de35e

Adapt column name

c5c8a44

birgits added 2 commits November 18, 2022 16:41

Bug fix scale CTS heat demand profile to total CTS heat demand in grid

ab717e6

Bug fix only scale CTS profile if it exists

a13930f

ClaraBuettner mentioned this pull request Nov 22, 2022

Add loads of rural gas boilers for eGon2035 #1027

Merged

6 tasks

birgits added 4 commits November 24, 2022 17:13

Add dataset documentation

2f73d79

Add function to delete pypsa eur sec csv file with minimum heat pump …

f11b5c0

…capacity before rerunning the task

Bug fix remove deletion of pypsa eur sec csv file from function that …

255f5c4

…exports minimum heat pump capacity to csv file

Adapt function creating residential heat load profiles to reduce memo…

eca1a72

…ry usage

nesnoj reviewed Nov 30, 2022

View reviewed changes

Reduce number of parallel tasks to avoid connection error

3111d68

ClaraBuettner and others added 2 commits December 1, 2022 08:51

Add dependency from heat pumps 2035 to heat demand time series for et…

7aa0aef

…rago

Revert "Reduce number of parallel tasks to avoid connection error"

7b7aeb1

This reverts commit 3111d68.

nailend added 2 commits January 10, 2023 14:06

Merge branch 'dev' into features/#903-desaggregate-heat-pumps

b5db836

Remove old adhoc fixes for #989

6a58844

Merge branch 'dev' into features/#903-desaggregate-heat-pumps

8dfaa24

nailend merged commit d021716 into dev Feb 2, 2023

nailend deleted the features/#903-desaggregate-heat-pumps branch February 2, 2023 15:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Desaggregate heat pumps #905

Desaggregate heat pumps #905

nailend commented Sep 5, 2022 •

edited

Loading

nesnoj Nov 30, 2022

birgits Nov 30, 2022

ClaraBuettner commented Nov 30, 2022

birgits commented Nov 30, 2022

nesnoj commented Dec 12, 2022

nesnoj commented Dec 23, 2022

birgits commented Jan 5, 2023 •

edited

Loading

nailend commented Jan 10, 2023

nailend commented Jan 10, 2023 •

edited

Loading

birgits commented Jan 23, 2023

nailend commented Jan 23, 2023 •

edited

Loading

birgits commented Jan 23, 2023

nesnoj commented Jan 26, 2023

nesnoj commented Jan 26, 2023 •

edited

Loading

ClaraBuettner commented Jan 31, 2023 •

edited by nailend

Loading

nesnoj commented Feb 1, 2023

Desaggregate heat pumps #905

Desaggregate heat pumps #905

Conversation

nailend commented Sep 5, 2022 • edited Loading

Before merging into dev-branch, please make sure that

nesnoj Nov 30, 2022

Choose a reason for hiding this comment

birgits Nov 30, 2022

Choose a reason for hiding this comment

ClaraBuettner commented Nov 30, 2022

birgits commented Nov 30, 2022

nesnoj commented Dec 12, 2022

nesnoj commented Dec 23, 2022

birgits commented Jan 5, 2023 • edited Loading

nailend commented Jan 10, 2023

nailend commented Jan 10, 2023 • edited Loading

birgits commented Jan 23, 2023

nailend commented Jan 23, 2023 • edited Loading

birgits commented Jan 23, 2023

nesnoj commented Jan 26, 2023

nesnoj commented Jan 26, 2023 • edited Loading

ClaraBuettner commented Jan 31, 2023 • edited by nailend Loading

nesnoj commented Feb 1, 2023

nailend commented Sep 5, 2022 •

edited

Loading

Before merging into `dev`-branch, please make sure that

birgits commented Jan 5, 2023 •

edited

Loading

nailend commented Jan 10, 2023 •

edited

Loading

nailend commented Jan 23, 2023 •

edited

Loading

nesnoj commented Jan 26, 2023 •

edited

Loading

ClaraBuettner commented Jan 31, 2023 •

edited by nailend

Loading