added binder links

AyrtonB · Mar 28, 2021 · 085e9ba · 085e9ba
1 parent 3b83672
commit 085e9ba
Show file tree

Hide file tree

Showing 35 changed files with 275 additions and 220 deletions.
diff --git a/README.md b/README.md
@@ -1,9 +1,25 @@
-[![DOI](https://zenodo.org/badge/326810654.svg)](https://zenodo.org/badge/latestdoi/326810654) [![PyPI version](https://badge.fury.io/py/moepy.svg)](https://badge.fury.io/py/moepy)
+[![DOI](https://zenodo.org/badge/326810654.svg)](https://zenodo.org/badge/latestdoi/326810654) [![PyPI version](https://badge.fury.io/py/moepy.svg)](https://badge.fury.io/py/moepy) [![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?urlpath=lab)
 
 # Merit-Order-Effect
 
 This repository/site outlines the development and usage of code and analysis used in calculating the Merit-Order-Effect (MOE) of renewables on price and carbon intensity of electricity markets. Beyond MOE analysis the `moepy` library can be used more generally for standard, quantile, and bootstrapped LOWESS estimation. The particular implementation of LOWESS in this software has been extended to significantly reduce the computational resource required.
 
+You can install the library using:
+
+```bash
+pip install moepy
+```
+
+`moepy` makes it simple to fit a LOWESS curve, a quick start example to generate the plot below can be found [here](https://ayrtonb.github.io/Merit-Order-Effect/ug-04-gb-mcc/).
+
+![](../img/latest_gb_mcc.png)
+
+<br>
+
+The library also includes the option to ensemble LOWESS models together and smooth them over time, an example is shown below for the marginal cost curve of dispatchable generation in Great Britain.
+
+![](../img/UK_price_MOE_heatmap.png)
+
 <br>
 
 ### The Paper
@@ -16,31 +32,6 @@ The `moepy` library was developed to enable new research into the Merit-Order-Ef
 > 
 > Our results indicate that renewables delivered reductions equal to 318M tonnes of CO$_{2}$ between 2010 and 2020, and 56B EUR between 2015 and 2020 in Germany. In GB the reductions amounted to a 442M tonnes of CO$_{2}$and £17B saving between 2010 and 2020.
 
-<br>
+We identified a strong relationship between increasing renewable penetration and the Merit-Order-Effect:
 
-### Repo Publishing - To Do
-
-Notebook Polishing Changes:
-- [x] Add docstrings (can be one-liners unless shown in the user-guides or likely to be used often)
-- [x] Add a mini sentence or two at the top of each nb explaining what it's about
-- [x] Ensure there is a short explanation above each code block
-- [x] Move input data to a raw dir
-- [ ] Check all module imports are included in settings.ini
-- [x] Re-run all of the notebooks at the end to check that everything works sequentially
-
-Additional Code:
-- [x] Re-attempt LIGO fitting example as part of a user-guide
-- [ ] Add in the prediction and confidence interval plots
-- [ ] Add a lot more to the EDA examples
-- [ ] Every week re-run a single analysis (could be based on the example in the user-guide) and show the generated fit at the top of the ReadMe
-- [ ] Try to speed things up, e.g. with Numba ([one person has already started doing this](https://gist.github.com/agramfort/850437#gistcomment-3437320))
-- [ ] Get the models saved on S3 or figshare and pulled into binder via a postBuild script
-
-External/ReadMe
-- [ ] Separate the binder and development `environment.yml` files (have the dev one inside the batch scripts folder)
-- [x] Add the GH action for version assignment triggering pypi push and zenodo update
-- [ ] Just before the paper is published set the version to 1.0.0 and have a specific Binder link that builds from that version as stored in the Zenodo archive ([example guide here](https://blog.jupyter.org/binder-with-zenodo-af68ed6648a6))
-- [ ] Link the zotero collection
-- [ ] Add citations for both the external data I use and the resulting time-series I generate
-- [ ] Add bibtex citation examples for both the paper and the code (could use [this](https://citation-file-format.github.io/cff-initializer-javascript/))
-- [ ] Mention the new module in the [gist](https://gist.github.com/agramfort/850437) that some of the basic regression code was inspired by 
+![](../img/GB_MOE_RES_relationship_95_CI.png)
diff --git a/docs/00-documentation.ipynb b/docs/00-documentation.ipynb
@@ -127,11 +127,11 @@
        "<progress style=\"width:60ex\" max=\"14\" value=\"14\" class=\"Progress-main\"/></progress>\n",
        "<span class=\"Progress-label\"><strong>100%</strong></span>\n",
        "<span class=\"Iteration-label\">14/14</span>\n",
-       "<span class=\"Time-label\">[00:15<00:00, 1.05s/it]</span></div>"
+       "<span class=\"Time-label\">[00:24<00:01, 1.69s/it]</span></div>"
       ],
       "text/plain": [
        "\u001b[A\u001b[2K\r",
-       " [████████████████████████████████████████████████████████████] 14/14 [00:15<00:00, 1.05s/it]"
+       " [████████████████████████████████████████████████████████████] 14/14 [00:24<00:01, 1.69s/it]"
       ]
      },
      "metadata": {},

diff --git a/docs/dev-01-retrieval.md b/docs/dev-01-retrieval.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-01-retrieval.ipynb)
+
 This notebook outlines the retrieval of the fuel generation and price data required for the merit-order-effect analyses.
 
 <br>

diff --git a/docs/dev-02-eda.md b/docs/dev-02-eda.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-02-eda.ipynb)
+
 This notebook includes some visualisation and exploration of the price and fuel data for Germany and Great Britain
 
 <br>

diff --git a/docs/dev-03-lowess.md b/docs/dev-03-lowess.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-03-lowess.ipynb)
+
 Outlines the development of the Scikit-Learn compatible `Lowess` model, as well as its extension `LowessDates` used for time-adaptive LOWESS regression. Included are functions for extending both models to generate prediction and confidence intervals. 
 
 <br>
@@ -30,7 +32,7 @@ from moepy import eda
 
 <br>
 
-### LOESS Development
+### LOWESS Development
 
 Before we go ahead any further we'll create some sample data for fitting and also define the fraction of the data over which we'll do the localised regression.
 
@@ -52,7 +54,7 @@ plt.plot(x, y)
 
 
 
-    [<matplotlib.lines.Line2D at 0x1a4b2c4ae80>]
+    [<matplotlib.lines.Line2D at 0x1fd650f86d0>]
 
 
 
@@ -202,7 +204,7 @@ timeit(lambda: get_weights(x, x[5]), number=10000)
 
 
 
-    0.39384629999999987
+    0.6126709999999989
 
 
 
@@ -244,7 +246,7 @@ all_weights[:5, :5]
 
 <br>
 
-Not too bad at all, we could now use this to weight the fitting of the polynomials in the LOESS. However, we've carried out most of these operations as part of for loops over vectors, what if we could store our data in matrices and do single operations over them?
+Not too bad at all, we could now use this to weight the fitting of the polynomials in the LOWESS. However, we've carried out most of these operations as part of for loops over vectors, what if we could store our data in matrices and do single operations over them?
 
 Thankfully Numpy has lots of helpful functions to aid us in this. We'll start by creating a matrix with the distances, to do this we can reshape the vectors into matrices of shape (25, 1) and (1, 25), then deduct the matrix with only one row from the matrix with only one column.
 
@@ -286,7 +288,7 @@ This approach brings an order of magnitude speed-up to the operation
 timeit(lambda: [get_dist(x, x[x_idx]) for x_idx in range(len(x))], number=10000)
 ```
 
-    672 ms ± 10.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
+    825 ms ± 163 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
 
 
 ```python
@@ -295,7 +297,7 @@ timeit(lambda: [get_dist(x, x[x_idx]) for x_idx in range(len(x))], number=10000)
 timeit(lambda: vector_to_dist_matrix(x), number=10000)
 ```
 
-    43.4 ms ± 2.63 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
+    55.5 ms ± 3.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
 
 
 <br>
@@ -320,7 +322,7 @@ timeit(lambda: gramfort_get_dist_thresholds(x, frac_idx), number=10000)
 
 
 
-    1.3339513000000007
+    1.7785535999999986
 
 
 
@@ -346,7 +348,7 @@ timeit(lambda: get_dist_thresholds(x, frac_idx, dist_matrix), number=10000)
 
 
 
-    0.06741010000000003
+    0.10906620000000089
 
 
 
@@ -382,7 +384,7 @@ timeit(lambda: (1 - inv_linear_weights ** 3) ** 3, number=10000)
 
 
 
-    0.2501098000000006
+    0.4310475999999994
 
 
 
@@ -393,7 +395,7 @@ timeit(lambda: np.power(1 - np.power(inv_linear_weights, 3), 3), number=10000)
 
 
 
-    0.24560150000000114
+    0.44873660000000015
 
 
 
@@ -408,7 +410,7 @@ timeit(lambda: (1 - inv_linear_weights ** 50000) ** 50000, number=10000)
 
 
 
-    0.28330940000000027
+    0.5387637000000005
 
 
 
@@ -419,7 +421,7 @@ timeit(lambda: np.power(1 - np.power(inv_linear_weights, 50000), 50000), number=
 
 
 
-    0.2943747000000023
+    0.526861499999999
 
 
 
@@ -515,7 +517,7 @@ timeit(lambda: get_full_dataset_weights_matrix(x, frac=frac), number=10000)
 
 
 
-    1.0831957999999986
+    1.4529582999999988
 
 
 
@@ -700,7 +702,7 @@ ax.set_ylabel('Regression Nodes')
 
 <br>
 
-We'll check this still works when we want to carry out a LOESS fit over all points
+We'll check this still works when we want to carry out a LOWESS fit over all points
 
 ```python
 weights = get_weights_matrix(x, frac=frac)
@@ -826,7 +828,7 @@ for i in range(n):
     y_pred[i] = betas[0] + betas[1] * x[i]
 
 plt.plot(x, y, label='Original')
-plt.plot(x, y_pred, label='LOESS')
+plt.plot(x, y_pred, label='LOWESS')
 plt.legend(frameon=False)
 ```
 
@@ -860,7 +862,7 @@ for i in range(len(x_pred)):
     y_pred[i] = betas[0] + betas[1] * x_pred[i]
 
 plt.plot(x, y, label='Original')
-plt.plot(x_pred, y_pred, label='LOESS')
+plt.plot(x_pred, y_pred, label='LOWESS')
 plt.legend(frameon=False)
 ```
 
@@ -954,7 +956,7 @@ y = np.sin(x)
 y_pred = lowess_fit_and_predict(x, y)
 
 plt.plot(x, y, label='Original')
-plt.plot(x, y_pred, '--', label='LOESS')
+plt.plot(x, y_pred, '--', label='LOWESS')
 plt.legend(frameon=False)
 ```
 
@@ -1048,7 +1050,7 @@ print(f'MAPE: {round(100*np.abs((y_true-y_pred)/y_true).mean(), 3)}%')
 
 <br>
 
-### LOWESS on Real-World Data
+#### LOWESS on Real-World Data
 
 We'll now evaluate the LOWESS fit on some real data, we'll start by loading the electric insights dataset
 
@@ -1206,7 +1208,7 @@ We can use our `lowess_fit_and_predict` function to make a lowess model of this
 y_pred = lowess_fit_and_predict(x, y_noisy, frac=0.2, num_fits=25) 
 
 plt.plot(x, y, label='Original', zorder=2)
-plt.plot(x, y_pred, '--', label='LOESS', color='k', zorder=3)
+plt.plot(x, y_pred, '--', label='LOWESS', color='k', zorder=3)
 plt.scatter(x, y_noisy, label='With Noise', color='C1', s=5, zorder=1)
 plt.legend(frameon=False)
 ```
@@ -1358,7 +1360,7 @@ def robust_lowess_fit_and_predict(x, y, frac=0.4, reg_anchors=None, num_fits=Non
 y_pred = robust_lowess_fit_and_predict(x, y_noisy, frac=0.2, num_fits=25) 
 
 plt.plot(x, y, label='Original', zorder=2)
-plt.plot(x, y_pred, '--', label='Robust LOESS', color='k', zorder=3)
+plt.plot(x, y_pred, '--', label='Robust LOWESS', color='k', zorder=3)
 plt.scatter(x, y_noisy, label='With Noise', color='C1', s=5, zorder=1)
 plt.legend(frameon=False)
 ```
@@ -1531,7 +1533,7 @@ y_pred = lowess.predict(x_pred)
 
 # Plotting
 plt.plot(x, y, label='Original', zorder=2)
-plt.plot(x_pred, y_pred, '--', label='Robust LOESS', color='k', zorder=3)
+plt.plot(x_pred, y_pred, '--', label='Robust LOWESS', color='k', zorder=3)
 plt.scatter(x, y_noisy, label='With Noise', color='C1', s=5, zorder=1)
 plt.legend(frameon=False)
 ```

diff --git a/docs/dev-04-price-surface-estimation.md b/docs/dev-04-price-surface-estimation.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-04-price-surface-estimation.ipynb)
+
 This notebook outlines how to specify different variants the model, then proceeds to fit them.
 
 <br>

diff --git a/docs/dev-05-price-moe.md b/docs/dev-05-price-moe.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-05-price-moe.ipynb)
+
 This notebook outlines the analysis required to determine the price merit-order-effect of variable renewable generation in the GB and DE power markets.
 
 <br>

diff --git a/docs/dev-06-carbon-surface-estimation-and-moe.md b/docs/dev-06-carbon-surface-estimation-and-moe.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-06-carbon-surface-estimation-and-moe.ipynb)
+
 This notebook outlines the analysis required to determine the carbon merit-order-effect of variable renewable generation in the GB and DE power markets.
 
 <br>

diff --git a/docs/dev-07-prediction-confidence-and-intervals.md b/docs/dev-07-prediction-confidence-and-intervals.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-07-prediction-confidence-and-intervals.ipynb)
+
 This notebook outlines the calculation of the prediction and confidence intervals for the GB and DE price MOE models
 
 <br>

diff --git a/docs/dev-08-hyper-parameter-tuning.md b/docs/dev-08-hyper-parameter-tuning.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-08-hyper-parameter-tuning.ipynb)
+
 This notebook outlines the hyper-parameter optimisation procedure used to tune the models
 
 <br>

diff --git a/docs/dev-09-tables-and-figures.md b/docs/dev-09-tables-and-figures.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-09-tables-and-figures.ipynb)
+
 This notebook provides a programmatic workflow for generating the tables used in the MOE paper, as well as the diagram to show the time-adaptive smoothing weights.
 
 <br>

diff --git a/docs/dev-10-ci-cd.md b/docs/dev-10-ci-cd.md
@@ -2,6 +2,14 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fdev-10-ci-cd.ipynb)
+
+This notebook includes helper functions and processes used in the continuous integration and deployment of the `moepy` library.
+
+<br>
+
+### Imports
+
 ```python
 #exports
 import os

diff --git a/docs/img/nbs/dev-03-lowess_cell_73_output_0.png b/docs/img/nbs/dev-03-lowess_cell_73_output_0.png
diff --git a/docs/img/nbs/output_71_0.png b/docs/img/nbs/output_71_0.png
diff --git a/docs/index.md b/docs/index.md
@@ -1,4 +1,4 @@
-[![DOI](https://zenodo.org/badge/326810654.svg)](https://zenodo.org/badge/latestdoi/326810654) [![PyPI version](https://badge.fury.io/py/moepy.svg)](https://badge.fury.io/py/moepy)
+[![DOI](https://zenodo.org/badge/326810654.svg)](https://zenodo.org/badge/latestdoi/326810654) [![PyPI version](https://badge.fury.io/py/moepy.svg)](https://badge.fury.io/py/moepy) [![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?urlpath=lab)
 
 # Merit-Order-Effect
 
@@ -10,6 +10,10 @@ You can install the library using:
 pip install moepy
 ```
 
+The library also includes the option to ensemble LOWESS models together and smooth them over time, an example is shown below for the marginal cost curve of dispatchable generation in Great Britain.
+
+![](../img/UK_price_MOE_heatmap.png)
+
 <br>
 
 ### The Paper
@@ -22,4 +26,8 @@ The `moepy` library was developed to enable new research into the Merit-Order-Ef
 > 
 > Existing research focuses on linear methods for modelling the merit order stack, masking the larger MOE seen in the steeper top/bottom regions. In this work a blended LOWESS model is used to capture the non-linear relationship between electricity price and dispatchable generation, with historical renewable output data then used to simulate the MOE. The stationary nature of many existing methodologies means they struggle to adapt to changes in the system such as the effect of the Covid-19 pandemic, we use a time-adaptive model to effectively address this limitation. Alongside an extension to the standard LOWESS implementation the use of a time-adaptive model significantly reduces the computational resource required. 
 > 
-> Our results indicate that renewables delivered reductions equal to 318M tonnes of CO$_{2}$ between 2010 and 2020, and 56B EUR between 2015 and 2020 in Germany. In GB the reductions amounted to a 442M tonnes of CO$_{2}$and £17B saving between 2010 and 2020.
+> Our results indicate that renewables delivered reductions equal to 318M tonnes of CO$_{2}$ between 2010 and 2020, and 56B EUR between 2015 and 2020 in Germany. In GB the reductions amounted to a 442M tonnes of CO$_{2}$and £17B saving between 2010 and 2020.
+
+We identified a strong relationship between increasing renewable penetration and the Merit-Order-Effect:
+
+![](../img/GB_MOE_RES_relationship_95_CI.png)
diff --git a/docs/ug-01-quantile.md b/docs/ug-01-quantile.md
@@ -2,6 +2,8 @@
 
 
 
+[![Binder](https://notebooks.gesis.org/binder/badge_logo.svg)](https://notebooks.gesis.org/binder/v2/gh/AyrtonB/Merit-Order-Effect/main?filepath=nbs%2Fug-01-quantile.ipynb)
+
 In this example we'll use power output data from Portugese hydro-plants to demonstrate how the quantile LOWESS model can be used.
 
 <br>