Integrate autoreporting mvs #284

Bachibouzouk · 2020-04-27T21:50:20Z

Fix #282
Fix #310

Changes proposed in this pull request:
Fixed:

Logging message for maximumCap value ([Bug] Logging error connected to maximumCap #310)

The following steps were realized, as well (if applies):

Use in-line comments to explain your code
Write docstrings to your code
For new functionalities: Explain in readthedocs
Write test(s) for your new patch of code
Update the CHANGELOG.md
Apply black (black . --exclude docs/)
Check if benchmark tests pass locally (pytest tests/test_benchmark.py)

_{For more information on how to contribute check the CONTRIBUTING.md.}

Bachibouzouk · 2020-04-27T23:26:46Z

@mahendrark - check the changes I made here to successfully run the code :)

smartie2076 · 2020-04-28T08:01:59Z

Doesn`t this solely implement parsing the argument that initiates the pdf report? I can not see that F2 is actually integrated ie. executed by the mvs jet. Can you change the title of this PR in that case?

Bachibouzouk · 2020-04-28T08:33:00Z

Doesn`t this solely implement parsing the argument that initiates the pdf report? I can not see that F2 is actually integrated ie. executed by the mvs jet. Can you change the title of this PR in that case?

I wrote the function print_pdf() which can be called for that, I haven't got the time to integrate it yesterday because it was not easy to figure out how to pack a server in a function and kill it after generating a pdf. I also don't know if the function works on windows. I should have created a draft PR not a PR I agree

smartie2076 · 2020-04-28T09:14:15Z

Doesn`t this solely implement parsing the argument that initiates the pdf report? I can not see that F2 is actually integrated ie. executed by the mvs jet. Can you change the title of this PR in that case?

I wrote the function print_pdf() which can be called for that, I haven't got the time to integrate it yesterday because it was not easy to figure out how to pack a server in a function and kill it after generating a pdf. I also don't know if the function works on windows. I should have created a draft PR not a PR I agree

Ah, okay! I did not find your comment about the server anymore, so I was not sure if you wanted to tackle this here, too.

Bachibouzouk · 2020-05-03T22:19:00Z

@mahendrark, please make your changes to the F2 reporting from this branch and as PR towards this branch

Bachibouzouk · 2020-05-10T15:07:26Z

@mahendrark - could you push your changes so we can merge them in this branch and merge this branch in dev. Right now what prevent me to do so are the non-general calls to specific values of the output. I mean non-general in the sense that they only apply to a specific simulation. We should write the code so that it can autogenerate a report for any kind of output folder. In the present situation the call to access directly "input_timeseries_Habor_kW.png" which prevent us to autogenerate reports for any simulation which would not have generated this file (which is specific to only one of the pilots).

I can make the code more general, however I understood you were tackling this issue so I don't want to step on your toes, let me know :)

mahendrark · 2020-05-11T08:44:03Z

@Bachibouzouk
I will make the code changes required to make the code general and then push the changes once I am done with that.

smartie2076 · 2020-05-11T11:51:09Z

Nice @Bachibouzouk , that you have figured out how to store the report as pdf!

I am exited to see the the report being more integrated to the mvs @mahendrark :) Let me know if you do not know how to acess specific values in the PR request that you create for that.

mahendrark · 2020-05-13T11:58:29Z

@Bachibouzouk, @smartie2076

The solution to that issue is basically plotting the graphs in F2, instead of looking for the images of the plots in Outputs folder.

Shall I plot all the necessary graphs with Plotly then?

Bachibouzouk · 2020-05-13T12:02:39Z

The quick and dirty solution is to add .png image one after the other at the end of the report, I would go for this one at first so that it unlocks #284. I think plots with plotly make sense then :)

@smartie2076 has the last word on that though

smartie2076 · 2020-05-13T13:22:52Z

The quick and dirty solution is to add .png image one after the other at the end of the report, I would go for this one at first so that it unlocks #284. I think plots with plotly make sense then :)

@smartie2076 has the last word on that though

I agree, lets keep the png for now. For that, I could also adapt the resulting json so that it includes the paths to all generated plots. We can talk about that in our call later.

smartie2076 · 2020-05-13T15:03:08Z

Hi @Bachibouzouk ! There are actually issues before I or mahendra can continue working on this PR.

First issue, when executing ´python mvs_tool.py -i inputs -f -ext csv`. It does not terminate the simulation or anything:

Traceback (most recent call last):
  File "mvs_tool.py", line 4, in <module>
    main()
  File "D:\PycharmProjects\mvs_eland\mvs_eland_tool\mvs_eland_tool.py", line 154, in main
    output_processing.evaluate_dict(dict_values)
  File "D:\PycharmProjects\mvs_eland\src\F0_output.py", line 108, in evaluate_dict
    dict_values["simulation_settings"]["path_output_folder"],
  File "D:\PycharmProjects\mvs_eland\src\F2_autoreport.py", line 87, in create_app
    dfprojectData = pd.read_csv(os.path.join(path_input_folder, "project_data.csv"))
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pandas\io\parsers.py", line 702, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pandas\io\parsers.py", line 429, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pandas\io\parsers.py", line 895, in __init__
    self._make_engine(self.engine)
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pandas\io\parsers.py", line 1122, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pandas\io\parsers.py", line 1853, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas\_libs\parsers.pyx", line 387, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas\_libs\parsers.pyx", line 705, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File b'inputs\\project_data.csv' does not exist: b'inputs\\project_data.csv'

Second issue, when executing ´python src/F2_autoreport.py´ (even after installing ´wkhtmltopdf ´, which has to be added to the requirements.txt), the pdf is not generated:

Loading formatted geocoded file...
 * Serving Flask app "F2_autoreport" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:8050/ (Press CTRL+C to quit)
INFORMATION: Es konnten keine Dateien mit dem angegebenen
Muster gefunden werden.
Traceback (most recent call last):
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pdfkit\configuration.py", line 21, in __init__
    with open(self.wkhtmltopdf) as f:
FileNotFoundError: [Errno 2] No such file or directory: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/F2_autoreport.py", line 810, in <module>
    print_pdf(app)
  File "src/F2_autoreport.py", line 32, in print_pdf
    pdfkit.from_url("http://127.0.0.1:8050", path_report_pdf)
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pdfkit\api.py", line 24, in from_url
    configuration=configuration, cover_first=cover_first)
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pdfkit\pdfkit.py", line 42, in __init__
    self.configuration = (Configuration() if configuration is None
  File "D:\PycharmProjects\mvs_eland\venv_windows\lib\site-packages\pdfkit\configuration.py", line 27, in __init__
    'https://github.com/JazzCore/python-pdfkit/wiki/Installing-wkhtmltopdf' % self.wkhtmltopdf)
OSError: No wkhtmltopdf executable found: "b''"
If this file exists please check that this process can read it. Otherwise please install wkhtmltopdf - https://github.com/JazzCore/python-pdfkit/wiki/Installing-wkhtmltopdf

I will probably do some coding on Friday, in case we need to have a call on this.

Bachibouzouk · 2020-05-13T15:27:23Z

First issue, when executing ´python mvs_tool.py -i inputs -f -ext csv`. It does not terminate the simulation or anything:

I know the code does not work smoothly, that is what @mahendrark should solve :) (i did comment lines so that the code works on my end)

smartie2076 · 2020-05-13T15:35:43Z

First issue, when executing ´python mvs_tool.py -i inputs -f -ext csv`. It does not terminate the simulation or anything:

I know the code does not work smoothly, that is what @mahendrark should solve :) (i did comment lines so that the code works on my end)

Do you know if the solution to issue 1 and issue 2 is one and the same for both? Because the link to ´inputs/project_data.csv´ seems to be something with relative paths (issue 1), while for issue 2 basically might be wrong because of any of the png. Is there a way to print the name of the file that it can not find?

Bachibouzouk · 2020-05-13T15:38:53Z

Issue for 2) might be Windows ... I didn't have it...

smartie2076 · 2020-05-13T16:10:02Z

Issue 1) will be solved by @Bachibouzouk. Issue 2 is due to issues installing wkhtmltopdf, and the output was generated now. I will create a dictionary of paths to the created png files so that they can easily be called from F2. For this PR, we do not need the plotlydash plots but will only use pngs.

@mahendrark : Please refer to this on how to get the pdf to compile!

smartie2076 · 2020-05-13T17:45:17Z

With the last commits, I created a dictionary of paths to plots, that can be accessed for the pdf report. This is now included in ´json_with_results.json´:

    "paths_to_plots": {
        "costs": [
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\annuity.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\annuity_minor.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\operation_and_maintenance_costs.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\operation_and_maintenance_costs_minor.png"
        ],
        "performance": [
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\optimal_additional_capacities.png"
        ],
        "flows_on_busses": [
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\Electricity_flows_365_days.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\Electricity bus_flows_365_days.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\PV plant (mono) bus_flows_365_days.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\ESS Li-Ion bus_flows_365_days.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\Electricity (DSO) bus_flows_365_days.png"
        ],
        "nx": [],
        "demands": [
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\input_timeseries_Habor_kW.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\input_timeseries_Remaining demand_kW.png",
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\input_timeseries_Shore power (docked ships)_kW.png"
        ],
        "resources": [
            "D:\\PycharmProjects\\mvs_eland\\MVS_outputs\\input_timeseries_PV plant (mono)_kW.png"
        ]
    },

The simulation does not run through due to issue 1. Also, I have not added tests for the generation of this dict yet (not sure if I have to, as they will be removed with plotlydash plots anyway). I need issue 1 to be fixed first to fix the failing tests.

I have indicated in following pdf (from standalone execution of F2) where the different plots should go. @Bachibouzouk - please look out for PF, those messages are for you, @mahendrark please look out for MR, those remarks are for you.

out_commented.pdf

@Bachibouzouk : Do you want to take over here and reference the paths in F2? This would take a long time for me to understand, and I would have no clue if simple loops would work.

You can find the constants with which to access the json dict of plots here.

@mahendrark please wait for @Bachibouzouk to be done with integrating the png, before you start finetuning the rest.

Bachibouzouk · 2020-05-14T06:08:48Z

@smartie2076, @mahendrark - it does run through now, however pdfkit has a problem to embed the images in the outputed pdf. In my last commit I indicated where you can increase the time to be able to vizualise the report in your browser before the "print_to_pdf" command is executed :)

smartie2076 · 2020-05-14T08:18:33Z

Thank you @Bachibouzouk !

The problem seems to have to do with the time allowed to store the pdf or something.
If I use

time.sleep(0) / time.sleep(10)

The images are not stored: out.pdf

time.sleep(60)

Nothing is stored at all: out.pdf

smartie2076 · 2020-05-14T08:42:38Z

Okay, there still are 14 failing tests, but that doesn´t stop us from working further on this. @mahendrark we can have a call on what you can do today or tomorrow.

Bachibouzouk · 2020-05-14T11:51:15Z

Thank you @Bachibouzouk !

The problem seems to have to do with the time allowed to store the pdf or something.
If I use

time.sleep(0) / time.sleep(10)

The images are not stored: out.pdf

time.sleep(60)

Nothing is stored at all: out.pdf

This time is the time one waits before executing the command. I think the problem lies within the html to pdf converter, because if you set 60 seconds, then you can visit 127.0.0.1:8050 and you see the whole report. Another problem might be that the dash renderer is not properly called by the pdf converter... (The renderer turns tthe phyrhon code into html code which is then rendered by. Browser)

smartie2076 · 2020-05-15T08:40:08Z

This time is the time one waits before executing the command. I think the problem lies within the html to pdf converter, because if you set 60 seconds, then you can visit 127.0.0.1:8050 and you see the whole report. Another problem might be that the dash renderer is not properly called by the pdf converter... (The renderer turns tthe phyrhon code into html code which is then rendered by. Browser)

I know that the .sleep() is giving me time to open the browser - I just thought that it might be a hint to whats going wrong, if this .sleep() actually influences the pdf that is being generated...? It seemed very weird to me.

Another question: Would it easily be possible to store the generated html? That could help us to finetune the formatting while the pdf is not generated. Also, later when we include dashplots, it may retain the interactive nature of the plots...?

smartie2076 · 2020-05-15T09:01:14Z

Sorry for the multiple pushes, somehow the pytests locally result in something different then the travis tests :/

Bachibouzouk · 2020-05-15T09:24:24Z

tests/test_F0_output.py

+    PLOTS_COSTS,
+)
+
+dict_plots = {


I would tend to create such a dict_plot in constants.py

Ah... yeah, makes sense. But don´t I need to use .deepcopy() then to make sure that the changes in eg. dict_values are not applied to the dict_plots in the tests as well?

d1 = dict(a=1, b=2) d2 = dict(c=3) d2.update(d1) print(d2) d1['d']=4 print(d2)

d2 is not modified, so probably updates takes care of that for you :)

Before it was fetching simulation settings from csv files, however, by only json should by used.

Adn import test constants from tests/constants.py instead of src.constants

mahendrark · 2020-05-18T10:20:43Z

Hey all,

I am getting this error when I try to run f2 script:

Traceback (most recent call last):
File "/home/mr/Projects/mvs_eland/src/F2_autoreport.py", line 736, in
test_app = create_app(CSV_FOLDER, OUTPUT_FOLDER)
TypeError: create_app() takes 1 positional argument but 2 were given

smartie2076 · 2020-05-18T10:27:24Z

Hey all,

I am getting this error when I try to run f2 script:

Traceback (most recent call last):
File "/home/mr/Projects/mvs_eland/src/F2_autoreport.py", line 736, in
test_app = create_app(CSV_FOLDER, OUTPUT_FOLDER)
TypeError: create_app() takes 1 positional argument but 2 were given

Hi @mahendrark ! As this PR is closed, can you please create an issue and report a bug with the bug report template? This will also help us to keep track of existing bugs and whether they are solved.

Also, if you include me/PF with @+github handle, we automatically receive a mail :)

mahendrark · 2020-05-18T10:40:14Z

@smartie2076 thank you

Will do that right away :)

smartie2076 marked this pull request as draft April 30, 2020 20:56

Bachibouzouk force-pushed the feature/integrate-autoreporting-mvs branch 3 times, most recently from d35ca58 to 1debd24 Compare May 3, 2020 22:17

Bachibouzouk marked this pull request as ready for review May 3, 2020 22:18

smartie2076 mentioned this pull request May 15, 2020

Removing complete logging messages when executing pytest #312

Closed

Bachibouzouk commented May 15, 2020

View reviewed changes

Bachibouzouk and others added 23 commits May 15, 2020 22:46

Lint with black

9a6d14a

Modify F2.create_app argument to results_json

d272403

Before it was fetching simulation settings from csv files, however, by only json should by used.

Include images automatically

06367d8

Unify use of result_json

cfc6ff4

Add css class

31e8c32

Fix typo

a482b14

Add comment

b89c49a

Replaced strings of plot dict with constant variables

ea19de0

Changed one string back

b49a5ce

Fixed plot calls

9596877

Fixed tests A1

94b897f

Fix pytests F0

96051b6

Fix pytests F1

be424f6

Update constant call in A0

ed53e4c

Fix test A0

d67372e

Fix test A0

7339f5c

Fix test A1

a90878b

Move DICT_plots to constants

f5bd684

Adn import test constants from tests/constants.py instead of src.constants

Simplify str addition

7bcc5f8

Change F2.print_pdf argument name and lint with black

e85dabb

Add tests for generation of pdf report

af93b6f

Fix tests for pdf report

1bad7e1

Update changelog

34db9d8

Bachibouzouk force-pushed the feature/integrate-autoreporting-mvs branch from d69aadc to 34db9d8 Compare May 15, 2020 20:46

Bachibouzouk merged commit 10dac5f into dev May 15, 2020

Bachibouzouk deleted the feature/integrate-autoreporting-mvs branch May 15, 2020 21:24

Bachibouzouk mentioned this pull request May 28, 2020

Release/v0.2.1 #341

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate autoreporting mvs #284

Integrate autoreporting mvs #284

Bachibouzouk commented Apr 27, 2020 •

edited by smartie2076

Bachibouzouk commented Apr 27, 2020

smartie2076 commented Apr 28, 2020

Bachibouzouk commented Apr 28, 2020

smartie2076 commented Apr 28, 2020

Bachibouzouk commented May 3, 2020

Bachibouzouk commented May 10, 2020

mahendrark commented May 11, 2020

smartie2076 commented May 11, 2020

mahendrark commented May 13, 2020 •

edited

Bachibouzouk commented May 13, 2020 •

edited

smartie2076 commented May 13, 2020

smartie2076 commented May 13, 2020

Bachibouzouk commented May 13, 2020

smartie2076 commented May 13, 2020

Bachibouzouk commented May 13, 2020

smartie2076 commented May 13, 2020

smartie2076 commented May 13, 2020

Bachibouzouk commented May 14, 2020

smartie2076 commented May 14, 2020

smartie2076 commented May 14, 2020

Bachibouzouk commented May 14, 2020

smartie2076 commented May 15, 2020

smartie2076 commented May 15, 2020

Bachibouzouk May 15, 2020

smartie2076 May 15, 2020

Bachibouzouk May 15, 2020 •

edited

mahendrark commented May 18, 2020

smartie2076 commented May 18, 2020

mahendrark commented May 18, 2020

Integrate autoreporting mvs #284

Integrate autoreporting mvs #284

Conversation

Bachibouzouk commented Apr 27, 2020 • edited by smartie2076

Bachibouzouk commented Apr 27, 2020

smartie2076 commented Apr 28, 2020

Bachibouzouk commented Apr 28, 2020

smartie2076 commented Apr 28, 2020

Bachibouzouk commented May 3, 2020

Bachibouzouk commented May 10, 2020

mahendrark commented May 11, 2020

smartie2076 commented May 11, 2020

mahendrark commented May 13, 2020 • edited

Bachibouzouk commented May 13, 2020 • edited

smartie2076 commented May 13, 2020

smartie2076 commented May 13, 2020

Bachibouzouk commented May 13, 2020

smartie2076 commented May 13, 2020

Bachibouzouk commented May 13, 2020

smartie2076 commented May 13, 2020

smartie2076 commented May 13, 2020

Bachibouzouk commented May 14, 2020

smartie2076 commented May 14, 2020

smartie2076 commented May 14, 2020

Bachibouzouk commented May 14, 2020

smartie2076 commented May 15, 2020

smartie2076 commented May 15, 2020

Bachibouzouk May 15, 2020

Choose a reason for hiding this comment

smartie2076 May 15, 2020

Choose a reason for hiding this comment

Bachibouzouk May 15, 2020 • edited

Choose a reason for hiding this comment

mahendrark commented May 18, 2020

smartie2076 commented May 18, 2020

mahendrark commented May 18, 2020

Bachibouzouk commented Apr 27, 2020 •

edited by smartie2076

mahendrark commented May 13, 2020 •

edited

Bachibouzouk commented May 13, 2020 •

edited

Bachibouzouk May 15, 2020 •

edited