Merge pull request #484 from rl-institut/release_v0.3.1

Release v0.3.1

.gitignore

@@ -81,6 +81,9 @@ target/
 # pyenv
 .python-version
 
+# folium map
+report/assets/proj_map
+
 # celery beat schedule file
 celerybeat-schedule
 
@@ -111,4 +114,17 @@ venv.bak/
 
 
 # autogenerated json
-mvs_csv_config.json
+mvs_csv_config.json
+
+# ERA5 nc file
+*.nc
+
+# report files
+/report/assets/proj_map*
+/report/*.pdf
+
+# Local input folders, eg. for pilots
+/example_1/
+/UVTgv_input_files/
+/inputs/
+

.readthedocs.yml

@@ -0,0 +1,21 @@
+# .readthedocs.yml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+# Required
+version: 2
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: docs/conf.py
+# Build documentation with MkDocs
+#mkdocs:
+#  configuration: mkdocs.yml
+
+# Optionally build your docs in additional formats such as PDF
+#formats:
+#  - pdf
+# Optionally set the version of Python and requirements required to build your docs
+python:
+   version: 3.7
+   install:
+      - requirements: docs/requirements.txt
+      - requirements: requirements.txt

.travis.yml

@@ -14,9 +14,9 @@ before_install:
 
 # command to install dependencies
 install:
-  - pip install -e .
   - pip install -r requirements.txt
   - pip install -r tests/test_requirements.txt
+  - pip install -e .
 
 # commands to run tests 
 script:

CHANGELOG.md

@@ -17,13 +17,96 @@ Here is a template for new release sections
 -
 ```
 
-## [Unreleased]
+## [0.3.1] - 2020-07-30
 
 ### Added
--
+- Release protocol in `CONTRIBUTING.md` file (#353)
+- Custom heat demand profile generation (#371)
+- Add custom solar thermal collector generation profile (#370)
+- Input template folder for easy generation of new simulations (#374), later also for tests of the input folder
+- Tests for ABE usecase (grid, PV, battery) (#385)
+- Test to verify that input folders have all required parameters (#398)
+- New `dict` `REQUIRED_MVS_PARAMETERS` to gather the required parameters from the csv or json
+ input type (#398)
+- `utils.py` module in `src` to gather the functions `find_input_folders` and `compare_input_parameters_with_reference` which can be used to find and validate input folders (#398)
+- Code and test for checking for new parameters in csv and raising warning message if not defined (`A1.check_for_newly_added_parameters`). This then also adds a default value to the new parameter  (#384)
+- Exception if an energyVector does not have internal generation or consumption from a DSO, and is only supplied by energy conversion from another sector: renewable share = 0. (#384)
+- Tests for source components in D1 (#391)
+- Option `-i` for `python mvs_report.py`, `python mvs_report.py -h` for help (#407)
+- Pyppeteer package for OS X users in troubleshooting (#414)
+- Add an enhancement to the auto-report by printing the log messages such as warnings and errors (#417)
+- New `dict` `REQUIRED_JSON_PARAMETERS` to gather the required parameters from the json input files (#432)
+- `.readthedocs.yml` configuration file (#435, #436)
+- Calculation of levelized cost of energy (`LCOE_ASSET`) of each asset in E2 (#438)
+- Tests for LCOE function in `test_E2_economics` (#438)
+- Output of `scalars.xlsx`now also includes `INSTALLED_CAP` and `LCOE_ASSET`(#438)
+- File `constants_output.py` to contain all keys included in `scalars.xlsx` (#453)
+- Installation help for `pygraphviz` on Win10/64bit systems in `troubleshooting.rst` (#379)
+- Add Plotly-based blots (line diagrams for energy flows and bar charts) to `F2_autoreport.py` (#439)
+- LCOE_ASSET (Levelized Cost of Energy of Asset) explaination in KPI documentation (#458)
+- Heat demand profiles with option of using monitored weather data (ambient temperature) at the use case UVtgV. note: file not provided so far (#474)
+- Benchmark test for simple case grid and diesel without test for fuel consumption (#386)
+
 ### Changed
--
+- Use selenium to print the automatic project report, `python mvs_report.py -h` for help (#356)
+- Sort parameters in csv´s within the input folder (#374)
+- Change relative folder path to absolute in tests files (#396)
+- Replace all variables wacc, discount_factor and project_lifetime in the project (#383)
+- Improve styling of the pdf report (#369)
+- `LIST_OF_NEW_PARAMETERS` renamed `EXTRA_CSV_PARAMETERS` and moved from `A1` to `constants.py
+` (#384)
+- Order of parameters in `tests/inputs`, fixed missing parameters  (#384)
+- Only a single output flow for sources (instead of multiple possible) as discussed in #149  (#391)
+- Move `existing` parameter into Investment objects of D1 components (was before added to output flow) (#391)
+- Use pyppeteers instead of selenium to emulate the webbrowser and print the pdf report
+ automatically (#407)
+- Update flowchart again (#409)
+- Label of storage components (storage capacity, input power, output power) will by default be redefined to the name of the storage and this component (#415)
+- Version number and date is only to be edited in one file (#419)
+- Add `ìnputs` folder to `.gitignore` (#401)
+- Change the calculation of the residual value for specific capex in C2 and test_C2 (#289, #247, PR #431): Now the present value of the residual value is considered
+- Explicitly return the dataframe with parameters value in function
+ `check_for_newly_added_parameter` (#428)
+- Rename function `check_for_newly_added_parameter` in `check_for_official_extra_parameters` (#428)
+- Add `ìnputs` folder to `.gitignore` (#401)
+- Readthedocs links to simple scenario `tests/inputs` (#420)
+- Adapt and add logging messages for components added to the model in D1 (#429)
+- Moved list of keys to be printed in `scalars.xlsx` to `constants_output.py` (#453)
+- Renamed `"peak_flow"` to `PEAK_FLOW` and `"average_flow"` to `AVERAGE_FLOW` (#453)
+- Changed function `E2.lcoe_asset()` and its tests, now processes one asset at a time (#453)
+- Added arguments ``-f`, `-log`, `warning`` to all `parse_args` and `main()` in `tests` (#456)
+- File `Developing.rst` with new description of tests and conventions (#456)
+- Added a `setup_class` (remove dir) to `test_B0.TestTemporaryJsonFileDisposal` (#379)
+- Created function to read version number and date from file instead of importing it from module
+ (#463)
+- Fixed `E0.store_results_matrix()`, now available types: `str`, `bool`, `None`, dict (with key VALUE), else (`int`/`float`). If KPI not in asset, no value is attributed. Added test for function (#468, #470)
+- Fixed `main()` calls in `test_F1_plotting.py` (#468)
+- Added `pyppdf==0.0.12` to `requirements.txt` (#473)
+- Tests for A0: Now new dirs are only created if not existant
+- Function `A0.check_output_folder()`, now after `shutil.rmtree` we still `try-except os.mkdirs`, this fixes local issues with `FileExistsError`.  (#474)
+
 ### Removed
+- Selenium to print the automatic project report for help (#407)
+- `MaximumCap` from list of required parameters for `energyStorage` assets (#415)
+- `inputs` folder (#401)
+- `tests/test_benchmark.py` module (#401)
+- Outdated table of tests of MVS `docs/tables/table_tests.csv` (#456)
+- Removed function `C0.complete_missing_cost_data()` as this should be covered by A1 for csv files (#379)
+- Old plots in `F2_autoreport.py` generated with matplotlib (#439)
+- Parameter `restore_from_oemof_file` from all files (inputs, tests) (#483)
+- Deleted columns from `fixcost.csv` as this is currently not used (#362)
+
+### Fixed
+- Bug connected to global variables (#356)
+- Duplicate of timeseries files (#388)
+- Warnings from local readthedocs compilation (#426)
+- Bug on local install (#437)
+- Input folder `tests/inputs` with simple example scenario (#420)
+- Description of storage efficiency in readthedocs (#457)
+- MVS can now be run with argument `-pdf` (fix pyppeteer issue) (#473)
+- Adapted benchmark tests input folders to template (#386)
+- Local failing pytests (`FileExistsError`) on Ubuntu and Win10 (#474, #483)
+- 9 Warnings due to excess parameter `restore_from_oemof_file` (#483)
 
 ## [0.3.0] - 2020-06-08
 
@@ -33,18 +116,15 @@ Here is a template for new release sections
 ### Changed
 - Test input files (#343)
 - All parameters of the json/csv input files are now defined by constant variables (i.e, `CRATE="crate"` instead of string `"crate"`) (#346)
-- Use "is" instead of "==" in if clauses for True, False and None (#346)
-- Categorize constants in 'constants_json_strings.py' (#347)
+- Use `is` instead of `==` in if clauses for True, False and None (#346)
+- Categorize constants in `constants_json_strings.py` (#347)
 - Renaming CAPEX_FIX = "capex_fix" into COST_DEVELOPMENT = "development_costs" (#347, #350)
 - Renaming CAPEX_VAR = "capex_var" into SPECIFIC_COST = "specific_costs" (#347, #350)
 - Renaming OPEX_FIX = "opex_fix" into SPECIFIC_COST_OM = "specific_costs_om" (#347, #350)
 - Renaming OPEX_VAR = "opex_var" into PRICE_DISPATCH = "dispatch_price" (#347, #350)
 - Change last strings into global constants in "constants_json_strings.py" (#349)
 - Autoreport now refers to actual project and scenario name + ID (#349)
 
-### Removed
-
-
 
 ## [0.2.1] - 2020-05-28
 
@@ -89,6 +169,7 @@ tipps for module building, and hint that units in the MVS are not checked (#229)
 - Possibility to deploy the report of the results in a browser (#323)
 - A main() function to be used by a server which only accepts json variable and returns json
  variable (not saving to a file) (#327)
+- Add information about the feature to view the web app and generate PDF of the automatic report to readthedocs (#283)
 
 ### Changed
 - Default input files from "inputs": Changed some parameters (#143)
@@ -115,9 +196,10 @@ tipps for module building, and hint that units in the MVS are not checked (#229)
 - Rename mvs_eland_tool/mvs_eland_tool.py --> mvs_eland_tool/local_deploy.py (#327)
 - Now main (local use) and run_simulation (server use) are available in mvs_eland_tool package
   (#327)
+
 
 ### Removed
-- Removed parameter ´oemof_file_name´ from ´simulation_settings.csv´, as well as from all input
+- Removed parameter `oemof_file_name` from `simulation_settings.csv`, as well as from all input
  files etc. The name is hardcoded now (#150)
 
 ### Fixed

CONTRIBUTING.md

@@ -10,7 +10,7 @@
 Development of a feature for this repository should follow the workflow described 
 by [Vincent Driessen](https://nvie.com/posts/a-successful-git-branching-model/).
 
-Here are the minimal procedure you should follow : 
+Here is the minimal procedure you should follow : 
 
 #### Step 1: Create an issue.
 
@@ -72,7 +72,7 @@ If a test fails, it is only due to what you changed (the test must passed before
   test names and error messages are there to help you find the error, please read them to try to
    debug yourself before seeking assistance :)
 
-Some test take more time to run (run a simulation) and were therefore disabled by default. It is
+Some tests take more time to run (run a simulation) and were therefore disabled by default. It is
  nevertheless important that you run a simulation test before you ask a review, or whenever you
   are ready to merge, to make sure the
   code still works.
@@ -104,6 +104,79 @@ Once you are satisfied with your PR you should ask someone to review it. Before
   described succinctly what you have done in the [CHANGELOG](https://github.com/rl-institut/mvs_eland/blob/dev/CHANGELOG.md) file (indicating the number of the PR in parenthesis after
   the description, not the number of the issue).
 
+## Release protocol
+
+Once you are ready to publish a release, branch off from `dev`
+    ```bash
+    git checkout -b release/vX.Y.Z dev
+    ```
+For meaning of X, Y and Z version numbers, please refer to this [semantic versioning guidelines](https://semver.org/spec/v2.0.0.html).
+
+In this branch, you should normally only update the version number in the `CHANGELOG.md` and `setup.py` files.
+
+Your `CHANGELOG.md` file could look like this before the release
+```
+## [unreleased]
+
+### Added
+- feature 1
+- feature 2
+### Changed 
+- thing 1
+- thing 2
+### Removed
+- some stuff
+```
+
+Simply replace `unreleased` by `X.Y.Z` and add the date of release in [ISO format](https://xkcd.com/1179/), then add the structure for a new `unreleased` version
+
+```
+## [unreleased]
+
+### Added
+-
+### Changed 
+-
+### Removed
+-
+
+## [X.Y.Z] - 20**-**-**
+### Added
+- feature 1
+- feature 2
+### Changed 
+- thing 1
+- thing 2
+### Removed
+- some stuff
+```
+
+After pushing these changes, create a pull request from `release/vX.Y.Z` towards `master` and merge it in `master`.
+
+Locally, merge `release/vX.Y.Z` into `dev`
+```
+git checkout release/vX.Y.Z
+```
+
+```
+git pull
+```
+
+```
+git checkout dev
+```
+
+```
+git merge release/vX.Y.Z
+```
+And push your these updates to the remote
+```
+git push
+```
+
+The idea behind this procedure is to avoid creating a merge commit in `dev` (because `master` would otherwise have two merge commit for this release once you merge the next release).
+
+Finally, [create a release](https://help.github.com/en/github/administering-a-repository/creating-releases) on github. Please choose master as the target for the tag and format the tag as `vX.Y.Z`. In the description field simply copy-paste the content of the `CHANGELOG`descriptions for this release and you're done!
 
 ## Contributing to Readthedocs
 

README.md

@@ -44,7 +44,7 @@ To set up the MVS, follow the steps below:
 
 * Open Anaconda prompt (or other software as Pycharm) to create and activate a virtual environment
 
-    `conda create -n [your_env_name] python=3.5`
+    `conda create -n [your_env_name] python=3.6`
     `activate [your env_name]`
 
 * Install required packages from requirements.txt file using pip
@@ -118,14 +118,31 @@ Edit the csv files (or, for devs, the json file) and run the `main()` function.
 - `display_output` (str): Sets the level of displayed logging messages. Options: "debug", "info", "warning", "error". Default: "info".
 - `lp_file_output` (bool): Specifies whether linear equation system generated is saved as lp file. Default: False.
 
-## Generate report
+## Generate pdf report
 
-To generate a report (only in browser for now), run the following command **after** a simulation
- ran:
+### directly after running a simulation
+
+Use the option `-pdf` in the command line  `python mvs_tool.py` to generate a pdf report in the
+ simulation's output folder (default in `MVS_outputs/simulation_report.pdf`):
+
+ `python mvs_tool.py -pdf`
+
+### post-processing
+To generate a report of the simulation's results, run the following command **after** a simulation
+ generated an output folder:
 
 `python mvs_report.py`
 
-the report should appear in your browser (at http://127.0.0.1:8050) as a Plotly Dash app.
+the report should appear in your browser (at http://127.0.0.1:8050) as an interactive Plotly Dash app.
+
+You can then print the report via your browser print functionality (ctrl+p), however the layout of
+ the pdf report is only well optimized for chrome or chromimum browser.
+
+It is also possible to automatically print the report as pdf by using the option `-pdf`. By default
+, it will
+save the report in the `report` folder. See`python mvs_report.py -h` for more information about
+possible options. The css and images used to make the report pretty should be located under
+`report/assets`.
 
 ## Contributing
 

docs/Developing.rst

@@ -4,7 +4,18 @@ Contributing to MVS
 
 Proposed workflow
 -----------------
-The workflow is described in the CONTRIBUTING.md file in the repository
+The workflow is described in the  `CONTRIBUTING.md file <https://github.com/rl-institut/mvs_eland/blob/dev/CONTRIBUTING.md>`_ in the repository.
+
+
+Unit tests (pytests)
+--------------------
+
+When developing code for the MVS please make sure that you always also develop test in `tests`. We integrate those unit tests with `pytest`. 
+Make sure that your tests are as leightweight as possible - this means that you do not always have to run the whole code to test for one feature, but can test a function with a standalone tests. Please refer to the other tests that have already been introduced.
+
+Always aim for the test coverage button on `the main page of the github reprository <https://github.com/rl-institut/mvs_eland/>`_ to reach 100%!
+
+When you do have to run the MVS itself for a test, eg. for benchmark tests, please always use the agruments `-f -log warning` to make the test results better readable.
 
 Build documentation
 -------------------
@@ -25,12 +36,3 @@ Here is how to set that in pycharm
   :width: 600
   :alt: pycharm docstring's format setting
 
-Tests
------
-
-Some tests are integrated into the MVS reprository. They should execute correctly on each developer's computer after adding a new feature and will be tested when attempting to merge into the developer/master branch of the github project.
-
-.. csv-table:: Implemented tests
-   :file: ./tables/table_tests.csv
-   :widths: 30, 50, 70, 70
-   :header-rows: 1