load shedding and shifting new example #585

AnasAlgarei · 2023-03-24T17:58:59Z

Closes # (if applicable).
#574

Changes proposed in this Pull Request

This PR includes a new PyPSA example that explains how load shedding and load shifting are represented in PyPSA. Load shedding is added as a generator that substitutes the decreased load, whereas load shifting is presented as a storage unit that could be charged and discharged, meaning that the load is decreased in certain periods and increased in others. This example needs further development to include elastic demand.

Checklist

Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
Newly introduced dependencies are added to environment.yaml, environment_docs.yaml and setup.py (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

for more information, see https://pre-commit.ci

codecov · 2023-03-24T18:10:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.38%. Comparing base (467db67) to head (dc56e1b).
Report is 6 commits behind head on master.

❗ Current head dc56e1b differs from pull request most recent head 04bead1. Consider uploading reports for the commit 04bead1 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #585       +/-   ##
===========================================
+ Coverage   66.30%   78.38%   +12.07%     
===========================================
  Files          26       26               
  Lines        7090     6948      -142     
  Branches     1425     1531      +106     
===========================================
+ Hits         4701     5446      +745     
+ Misses       2105     1179      -926     
- Partials      284      323       +39

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pz-max · 2023-03-28T21:57:31Z

I am currently reviewing this and worked on @AnasAlgarei branch by applying this: https://stackoverflow.com/a/37686809

git clone https://github.com/pz-max/PyPSA.git
cd PyPSA
git remote add AnasAlgarei https://github.com/AnasAlgarei/PyPSA
git fetch AnasAlgarei
git checkout -b load-example AnasAlgarei/load_shedding_shifting_example

I opened a PR with suggestion in @AnasAlgarei repo https://github.com/AnasAlgarei/PyPSA/tree/load_shedding_shifting_example

Load example

for more information, see https://pre-commit.ci

AnasAlgarei · 2023-04-06T03:42:07Z

Hi @pz-max, I just comitted a final draft of the load modelling example. However, it seems that it failed one of the checks with error: FAILED test/test_examples.py::test_ac_dc_meshed - urllib.error.HTTPError: HTTP Error 503: first byte timeout. When I looked up this error, it looks like it's related to timeout when doing a HTTP request to an external server. So would re-running the checks resolve it? If yes, explainging how to do it is appreciated.

Load shedding shifting example

Load example

for more information, see https://pre-commit.ci

pz-max · 2023-04-07T17:27:01Z

Ready for review @FabianHofmann. This PR is adding a load modelling example including:

load shedding
load shifting
elastic demand
Let us know a time if you prefer a joint review :)

FabianHofmann · 2023-05-09T10:47:20Z

Hey @AnasAlgarei, thanks for the nice contribution. After reviewing a bit, some points came up how to improve the example.

It would be better to show the effects of load shifting and load shedding separately. I'd propose to rearrange the story to the following outline

Show the network with load shifting only.
Alter the network to make it infeasible.
Introduce load shedding (which represents the value of lost load) to make it feasible again. This demonstrates the real purpose of load shedding
Show elastic demand

or

Show the network without any flexibility, show that it is infeasible.
Introduce load shedding (which represents the value of lost load) to make it feasible.
Show load shifting as a more flexible and realistic alternative.
Show elastic demand

Thinking about it, I tend to prefer the second option, as it adds more and more complexity.

We also discovered some inconsistencies with the load shifting device. It has the carrier 'battery' and a standing loss and efficiencies for charging and discharging. However, load shifting should not be constrained to any of these right? You do not lose energy when you shift load, e.g. you would not need more power if you consume it at a later point.

That's it for the start. Sorry that reviewing takes so long and thanks again for the nice work.

pz-max · 2023-05-09T13:13:03Z

PR is coming on my end. @AnasAlgarei can you merge the PR into your branch? I don't have the rights yet

AnasAlgarei · 2023-10-02T00:02:01Z

@AnasAlgarei Linopy now also has QP programming capabilities https://github.com/PyPSA/linopy/releases I think this PR could get updated in the latest section talking about the QP way of solving this. To clarify, I think we don't need to demonstrate solving this as QP problem as another PyPSA example can be dedicated towards it but just updating the comment might be good. Unless @Irieo feels to show off the QP implementation :)

Also, looking at the example after some time, do you have some other ideas on how the PR can improve Anas (e.g. clarity of examples)? Maybe we can push this for the next release. Feel free to add the release note already with a short description... there is a general interest to get the example included

Hey @pz-max, I made the suggested adjustments to the file, please let me know of any further adjustments. Thanks for following up on this matter.

pz-max · 2023-10-02T07:08:15Z

examples/notebooks/load-modelling.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**Note.** Here we solve the elastic demand as linear programming (LP) problem and several iterations. It is also possible to solve the problem with **one single** optimization run. However, this requires to reformulate the problem as quadratic programming (QP) problem, see [here](https://github.com/PyPSA/PyPSA/issues/574#issuecomment-1451865672). PyPSA can also do that, however, we need to add the QP solving capability to Linopy. This is already tracked as [feature request](https://github.com/PyPSA/linopy/issues/18). Stay tuned!"


Needs to be adjusted. Linopy has now capabilities to solve quadratic problems (QP)

@AnasAlgarei , regarding the smart thermostat example.
I think we should also dismiss the "smart thermostat" example. A thermostat is a measurement device. People find it irritating when a thermostat is modelled as a storage unit without a detailed explanation. Can you find another load-shifting example that is more focused ideally on electricity? We need a short and concise description of how the device behaves and why its modelled like e.g. a storage_unit

Further, I think the input:

carrier could be removed.
-p_nom could be set positive to say that there is a device.
-cycling losses (you know which variable I mean) could be removed. It increases unnecessary the complexity.

Needs to be adjusted. Linopy has now capabilities to solve quadratic problems (QP)

Apologies about this, realised I didn't save the last changes before committing, all sorted now.

@AnasAlgarei , regarding the smart thermostat example. I think we should also dismiss the "smart thermostat" example. A thermostat is a measurement device. People find it irritating when a thermostat is modelled as a storage unit without a detailed explanation. Can you find another load-shifting example that is more focused ideally on electricity? We need a short and concise description of how the device behaves and why its modelled like e.g. a storage_unit

Further, I think the input:

* `carrier` could be removed. -`p_nom` could be set positive to say that there is a device. -`cycling losses` (you know which variable I mean) could be removed. It increases unnecessary the complexity.

Thanks for the comments, will work on that and update 👍

Hi @pz-max, I made changes as per your comments. Please review when possible.

for more information, see https://pre-commit.ci

…nasAlgarei/PyPSA into load_shedding_shifting_example

for more information, see https://pre-commit.ci

ekatef

Hello @AnasAlgarei 🙂 Great example and excellent explanations!

Have added some comments to the code. Apart of them, I'd suggest to keep color coding the same for all the plots: e.g. make load orange everywhere. My feeling is, it would facilitate analysis of the plots. What do you think?

ekatef · 2023-10-17T08:28:09Z

examples/notebooks/load-modelling.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This notebook provides a simple illustration of how load shedding, load shifting, and elastic demand can be modeled in PyPSA. Through the example, we'll simulate a single day of electrical consumption in a network with solar generation, which could represent a household or a small community.\n",


To me, a second phrase hasn't sounded very clear at first reading: " a network with solar generation, which could represent...". Would it be possible to re-formulate a bit? Like "solar-powered network, which could represent..."?

Edited to "we'll simulate a single day of electrical consumption in a solar-powered network, which could represent a household or a small community." ✔️

ekatef · 2023-10-17T08:36:12Z

examples/notebooks/load-modelling.ipynb

+    "\n",
+    "We'll start by constructing a minimal example network to demonstrate the demand modeling cases. The network consists of:\n",
+    "\n",
+    "- a single bus with a time span between 04:00 - 20:00.\n",


Not sure I get the point here. Does it mean that we consider only 04:00-20:00? It would be also great to specify what does the bus represent, assuming that a reader can be not perfectly fluent with PyPSA components. So, a short clarification could be very helpful, like: "a bus which represents a network node, to which generation and loads are connected".

Edited to:
"## Build example network

We'll start by constructing a minimal example network to demonstrate the demand modeling cases. The network consists of:

a single bus which represents a network node, to which generation and loads are connected.

an hourly time span of a single day, from 00:00 to 23:00.

one PV (Photovoltaic) generator, generating only in the daytime.

an electricity load profile (active), with peak load between 19:00 - 23:00." ✔️

Sounds great! 👍🏽

ekatef · 2023-10-17T08:38:24Z

examples/notebooks/load-modelling.ipynb

+    "    carrier=\"electricity\",\n",
+    "    p_nom_extendable=True,\n",
+    "    standing_loss=0.01,\n",
+    "    cyclic_stage_of_charge=True,\n",


A typo should by fixed: cyclic_stage_of_charge -> cyclic_state_of_charge

fixed ✔️

ekatef · 2023-10-17T08:39:44Z

examples/notebooks/load-modelling.ipynb

+    "    \"StorageUnit\",\n",
+    "    \"EV battery\",\n",
+    "    bus=\"My bus\",\n",
+    "    p_nom=1,  # positive to indicate that there is a devise\n",


Shouldn't it be device?

fixed ✔️

ekatef · 2023-10-17T08:50:22Z

examples/notebooks/load-modelling.ipynb

+    "    )\n",
+    "\n",
+    "# Fit a linear curve to the data\n",
+    "slope, intercept = np.polyfit(updated_average_loads, updated_prices, 1)\n",


Could it probably make sense to add overlay results of the simulation with the original elasticity curve? @AnasAlgarei what is your feeling about that?

The original line is already overlayed, but it is zoomed in the graph for the points to be clear to see.

When zooming out the points will be very close to each other that it is not clear:

@AnasAlgarei absolutely agree regarding zooming :) An attempt to keep the original scale doesn't look nice.

I have been thinking about making more evident that the elasticity curve above and here are the same thing, like continuing the elasticity line beyond the modeled points and probably trying to keep the same angle with the axises. But that would require additions to the code, which would be distractive. So, let's keep as is.

I think, the legend will be enough to make the connection between both plots.

AnasAlgarei · 2023-10-19T23:41:02Z

Hello @AnasAlgarei 🙂 Great example and excellent explanations!

Have added some comments to the code. Apart of them, I'd suggest to keep color coding the same for all the plots: e.g. make load orange everywhere. My feeling is, it would facilitate analysis of the plots. What do you think?

Thank you so much @ekatef for the thorough review of the example. Your comments are really appreciated and were taken into account where possible. I agree that setting a consistant colors for all plots makes it look better, I have included that as well. I will commit all changes soon 😃

ekatef · 2023-10-21T11:09:58Z

Hello @AnasAlgarei 🙂 Great example and excellent explanations!
Have added some comments to the code. Apart of them, I'd suggest to keep color coding the same for all the plots: e.g. make load orange everywhere. My feeling is, it would facilitate analysis of the plots. What do you think?

Thank you so much @ekatef for the thorough review of the example. Your comments are really appreciated and were taken into account where possible. I agree that setting a consistant colors for all plots makes it look better, I have included that as well. I will commit all changes soon 😃

Hello @AnasAlgarei. Awesome!
Thanks a lot for making the changes. Absolutely agree with your approach. Looking forward for the final version committed 🙂

for more information, see https://pre-commit.ci

AnasAlgarei · 2023-10-21T19:16:33Z

Hello @AnasAlgarei 🙂 Great example and excellent explanations!
Have added some comments to the code. Apart of them, I'd suggest to keep color coding the same for all the plots: e.g. make load orange everywhere. My feeling is, it would facilitate analysis of the plots. What do you think?

Thank you so much @ekatef for the thorough review of the example. Your comments are really appreciated and were taken into account where possible. I agree that setting a consistant colors for all plots makes it look better, I have included that as well. I will commit all changes soon 😃

Hello @AnasAlgarei. Awesome! Thanks a lot for making the changes. Absolutely agree with your approach. Looking forward for the final version committed 🙂

Hi @ekatef, I just committed all changes. Kindly have a look at it when possilbe, and let me know what you think. Thanks again for the review 😃

ekatef · 2023-10-23T12:25:06Z

Hey @AnasAlgarei! Looks great 😄
Amazing work!

Irieo · 2023-10-25T15:21:13Z

Hi @AnasAlgarei and @pz-max @ekatef thanks for this example! Fabian and I went through it. We think Load shedding and Load shifting are ready with small adjustments, and we have some concerns re. Elastic demand. I don't see a way to comment lines in Notebook files in a PR, so the comments are above:

Load shedding

# Add load shedding as generator
network.add(
    "Generator",
    "load shedding",
    bus="My bus",
    p_nom=50,
    p_max_pu=1.0,
    marginal_cost=1e2,  # 100 €/MWh
    p_nom_extendable=True,
)

One should either set p_nom to some value OR set p_nom_extendable=True.
If you do both, after optimizing the network, you will see that glpk reduces the size of the load shedding virtual generator to the "nominal capacity" required to do the job.

network.generators.p_nom_opt
Generator
solar 0.4
load shedding 0.2
Name: p_nom_opt, dtype: float64

Note that the default value of the capital cost is zero, check network.component_attrs["Generator"]

Load shifting

First, we thought that description of the mechanism should be rather generic, i.e., pointing that it could be EVs, or delay of execution of flexible workloads (and associated power loads) in data centers, and many more.

Second, it might be worth mentioning that you use here PyPSA StorageUnit component as a tool to model load shifting, but it is also possible to model load shifting via other components -- so that the exact formulation of the load shifting is tailored to the modeller needs. Eventually, the same result of your example can be achieved with StorageUnit, Store and Generator as a mean to model load shifting. Here is an example.

Third, the same comment applies as in load shedding, since you set p_nom_extendable=True, the glpk reduces the capacity of the load shedding mechanism to 0.21 MW. It is not a problem, but rather a modelling artefact.

Elastic demand

First, the main thing where we were perplexed, is the fact that the concept of the demand elasticity to price is illustrated via system_cost_per_MWh. As such, the average system price is the parameter that you update via iterations. The concept one could expect in this illustration is the hourly reaction of the demand to price deviations. Thus, when price elasticity of demand is implemented, demand in hours of high price would reduce, and in hours of low price would increase. The price here refers to an hourly price for a specific commodity in a specific bus.

p.s. It is interesting that if one checks the marginal price time-series in your bus, the prices in most hours are 1e2 (i.e. load shedding costs), even when load shedding does not take place, since you have a single generator and a single bus.

Second, you start using .lopf() instead of .optimise(). If user has a new version of PyPSA and linopy, .lopf() gives an error. So I run n.optimise() instead to reproduce your examples. More generally, is there a reason you change optimization framework across these examples?

Other

We spotted that n = n.copy() does not copy some network attributes, like objective, so be careful. Fabian created an issue already #767

pz-max · 2023-10-28T09:00:37Z

@AnasAlgarei

Load shedding example

Let's kick out p_nom_extendable=True and also p_max_pu=1. The former will lead to the use of 50 MW load-shedding generator size, and the latter is already the default and won't change anything.

network.add(
"Generator",
"load shedding",
bus="My bus",
p_nom=50,
marginal_cost=100, # 100 €/MWh
)

Load shifting example

Can you make the adjustments as @Irieo is pointing out? (Mostly about changing the description.)

Elastic demand

@Irieo , I think we could keep the annual electricity price with the argument that most households don't see hourly prices (in UK the partially do with e.g. Octopus Energy). But @AnasAlgarei let's change it as Fabian and Iegor suggest. Might confuse less people.

P.S. I hope the marginal price time-series will change. It should be 0 when there is no load shedding.

fneum · 2023-10-28T17:34:24Z

The elastic demand should not be done with iterative optimisation.

Since PyPSA v0.24.0, you can model elastic demand with a quadratic marginal cost term (https://pypsa.readthedocs.io/en/latest/release_notes.html#pypsa-0-24-0-27th-june-2023)

Here's how this can be implemented:

elastic_intercept = 2000 # change here to alter level of price elasticity
load = 100 # MW
n.add(
    "Generator",
    "load-shedding",
    bus="electricity",
    carrier="load",
    marginal_cost_quadratic=elastic_intercept / ( 2 * load),
    p_nom=load,
)
n.add("Load", "load", bus="electricity", carrier="load", p_set=load)

This turns the model into a QP, but still convex, so performance is not significantly impacted.

AnasAlgarei · 2023-10-30T20:41:47Z

Thank you @Irieo and @FabianHofmann for the informative review and comments, and thank you @pz-max and @ekatef for helping me make this example look better. I will implement the changes suggested in the comments and update here when completed.

Load example fix

for more information, see https://pre-commit.ci

AnasAlgarei and others added 2 commits March 24, 2023 17:41

load shedding and shifting new example

04464c1

[pre-commit.ci] auto fixes from pre-commit.com hooks

84f2b16

for more information, see https://pre-commit.ci

add suggestion

f36a84c

AnasAlgarei and others added 11 commits March 31, 2023 18:56

Merge pull request #1 from pz-max/load-example

7d8e417

Load example

[pre-commit.ci] auto fixes from pre-commit.com hooks

a8420b7

for more information, see https://pre-commit.ci

small_markdowns_corrections

450dd00

[pre-commit.ci] auto fixes from pre-commit.com hooks

0a110cb

for more information, see https://pre-commit.ci

Merge branch 'PyPSA:master' into load_shedding_shifting_example

3110f5f

Merge branch 'PyPSA:master' into load_shedding_shifting_example

968d2a8

simulating_elastic_demand

93008d1

[pre-commit.ci] auto fixes from pre-commit.com hooks

06fa747

for more information, see https://pre-commit.ci

Merge branch 'PyPSA:master' into load_shedding_shifting_example

47730a9

file correction and adding doc link

455dbfc

[pre-commit.ci] auto fixes from pre-commit.com hooks

ccffb17

for more information, see https://pre-commit.ci

pz-max and others added 4 commits April 7, 2023 16:43

adjust text and make more compact

ff97e30

Merge pull request #2 from pz-max/load_shedding_shifting_example

26b03ef

Load shedding shifting example

Merge pull request #2 from pz-max/load-example

9eada72

Load example

[pre-commit.ci] auto fixes from pre-commit.com hooks

e4cd7ad

for more information, see https://pre-commit.ci

fneum requested a review from FabianHofmann April 10, 2023 19:14

Merge branch 'master' into load_shedding_shifting_example

3f1bbf4

Irieo self-requested a review May 2, 2023 14:47

Merge branch 'master' into load_shedding_shifting_example

0cdaae5

Merge branch 'master' into load_shedding_shifting_example

2a32b3e

add fixes

c1efc66

small clarifications and editing the QP comment

97e9946

pz-max reviewed Oct 2, 2023

View reviewed changes

AnasAlgarei and others added 6 commits October 3, 2023 13:35

editing the QP comment

1385e10

[pre-commit.ci] auto fixes from pre-commit.com hooks

3c9f437

for more information, see https://pre-commit.ci

added full day snapshots and varying demand

cdb0f2f

Merge branch 'PyPSA:master' into load_shedding_shifting_example

a6e773d

Merge branch 'load_shedding_shifting_example' of https://github.com/A…

b5c1627

…nasAlgarei/PyPSA into load_shedding_shifting_example

[pre-commit.ci] auto fixes from pre-commit.com hooks

f5608bf

for more information, see https://pre-commit.ci

ekatef reviewed Oct 17, 2023

View reviewed changes

Merge branch 'PyPSA:master' into load_shedding_shifting_example

1c1201b

AnasAlgarei and others added 3 commits October 21, 2023 19:50

clear explanations, fixing typos, consistant colors

0b9aef0

merging changes

5c198df

[pre-commit.ci] auto fixes from pre-commit.com hooks

9c70247

for more information, see https://pre-commit.ci

Merge branch 'master' into load_shedding_shifting_example

dc56e1b

Merge branch 'master' into load_shedding_shifting_example

6650f54

pz-max and others added 6 commits April 10, 2024 14:13

Merge branch 'master' into load-example

ee512dd

add QP formulation for elastic demand

f61f91e

Merge branch 'load_shedding_shifting_example' into load-example

71a1ea3

Merge pull request #4 from pz-max/load-example

14316a1

Load example fix

[pre-commit.ci] auto fixes from pre-commit.com hooks

d4fcfdc

for more information, see https://pre-commit.ci

Merge branch 'master' into load_shedding_shifting_example

04bead1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load shedding and shifting new example #585

load shedding and shifting new example #585

AnasAlgarei commented Mar 24, 2023 •

edited

codecov bot commented Mar 24, 2023 •

edited

pz-max commented Mar 28, 2023

AnasAlgarei commented Apr 6, 2023

pz-max commented Apr 7, 2023 •

edited

FabianHofmann commented May 9, 2023

pz-max commented May 9, 2023 •

edited

AnasAlgarei commented Oct 2, 2023

pz-max Oct 2, 2023

pz-max Oct 2, 2023

AnasAlgarei Oct 3, 2023

AnasAlgarei Oct 3, 2023

AnasAlgarei Oct 8, 2023

ekatef left a comment

ekatef Oct 17, 2023

AnasAlgarei Oct 19, 2023

ekatef Oct 21, 2023

ekatef Oct 17, 2023

AnasAlgarei Oct 19, 2023

ekatef Oct 21, 2023

ekatef Oct 17, 2023

AnasAlgarei Oct 19, 2023

ekatef Oct 17, 2023

AnasAlgarei Oct 19, 2023

ekatef Oct 17, 2023

AnasAlgarei Oct 19, 2023

ekatef Oct 21, 2023

AnasAlgarei commented Oct 19, 2023

ekatef commented Oct 21, 2023

AnasAlgarei commented Oct 21, 2023 •

edited

ekatef commented Oct 23, 2023

Irieo commented Oct 25, 2023

pz-max commented Oct 28, 2023

fneum commented Oct 28, 2023 •

edited

AnasAlgarei commented Oct 30, 2023 •

edited

load shedding and shifting new example #585

Are you sure you want to change the base?

load shedding and shifting new example #585

Conversation

AnasAlgarei commented Mar 24, 2023 • edited

Changes proposed in this Pull Request

Checklist

codecov bot commented Mar 24, 2023 • edited

Codecov Report

pz-max commented Mar 28, 2023

AnasAlgarei commented Apr 6, 2023

pz-max commented Apr 7, 2023 • edited

FabianHofmann commented May 9, 2023

pz-max commented May 9, 2023 • edited

AnasAlgarei commented Oct 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekatef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnasAlgarei commented Oct 19, 2023

ekatef commented Oct 21, 2023

AnasAlgarei commented Oct 21, 2023 • edited

ekatef commented Oct 23, 2023

Irieo commented Oct 25, 2023

Load shedding

Load shifting

Elastic demand

Other

pz-max commented Oct 28, 2023

Load shedding example

Load shifting example

Elastic demand

fneum commented Oct 28, 2023 • edited

AnasAlgarei commented Oct 30, 2023 • edited

AnasAlgarei commented Mar 24, 2023 •

edited

codecov bot commented Mar 24, 2023 •

edited

pz-max commented Apr 7, 2023 •

edited

pz-max commented May 9, 2023 •

edited

AnasAlgarei commented Oct 21, 2023 •

edited

fneum commented Oct 28, 2023 •

edited

AnasAlgarei commented Oct 30, 2023 •

edited