Move data and modules around by rouille · Pull Request #245 · Breakthrough-Energy/PowerSimData

rouille · 2020-08-03T03:48:46Z

Purpose

Rename/move modules and move data as listed in #244.

What is the code doing?

There are only two modules that I have tweaked:

powersimdata.data_access.scenario_list module where I added an argument to the _execute_and_check_err method to raise an explicit error message if the command fails.
powersimdata.network.usa_tamu.usa_tamu_model module where I modified the path to the data in the _set_data_loc method.
Otherwise I created some __init__ module along with modifying various import statements and existing __init__ modules.

Where to look

The only module that is not covered by the tests is powersimdata.data_access.scenario_list. Make sure everything looks good there.

Time estimate

15 minutes.

jenhagg · 2020-08-03T21:54:40Z

-            raise IOError(
-                "Failed to delete entry in %s on server" % const.SCENARIO_LIST
-            )
+            raise IOError(err_message)


Thanks for fixing this!

jenhagg

We want to synchronize corresponding changes to PostREISE, etc?

rouille · 2020-08-03T22:29:53Z

We want to synchronize corresponding changes to PostREISE, etc?

In fact, there is nothing in PostREISE that needs to be changed. I thought the constants module was used in PostREISE but it is never imported. The ScenarioInfo class that uses the constants module is used in PostREISE. All tests pass in PostREISE with this branch of PowerSimData checked out.:

[~/CEM/PostREISE] (develop) brdo$ pipenv sync
Creating a virtualenv for this project…
Pipfile: /Users/brdo/CEM/PostREISE/Pipfile
Using /usr/local/bin/python3 (3.8.3) to create virtualenv…
⠸ Creating virtual environment...created virtual environment CPython3.8.3.final.0-64 in 3213ms
  creator CPython3Posix(dest=/Users/brdo/.local/share/virtualenvs/PostREISE-2SBrWDwC, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/brdo/Library/Application Support/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/brdo/.local/share/virtualenvs/PostREISE-2SBrWDwC
Installing dependencies from Pipfile.lock (60d44d)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 39/39 — 00:00:49
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
All dependencies are now up-to-date!
[~/CEM/PostREISE] (develop) brdo$ pipenv shell
Launching subshell in virtual environment…
[~/CEM/PostREISE] (develop) brdo$  . /Users/brdo/.local/share/virtualenvs/PostREISE-2SBrWDwC/bin/activate
(PostREISE) [~/CEM/PostREISE] (develop) brdo$ pytest .
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /Users/brdo/CEM/PostREISE
collected 129 items                                                                                                                                            

postreise/analyze/generation/tests/test_binding.py ....................                                                                                  [ 15%]
postreise/analyze/generation/tests/test_capacity_value.py ......................                                                                         [ 32%]
postreise/analyze/generation/tests/test_carbon.py ....                                                                                                   [ 35%]
postreise/analyze/generation/tests/test_curtailment.py .................                                                                                 [ 48%]
postreise/analyze/generation/tests/test_summarize.py ...                                                                                                 [ 51%]
postreise/analyze/tests/test_helpers.py ....                                                                                                             [ 54%]
postreise/analyze/transmission/tests/test_congestion_surplus.py ...                                                                                      [ 56%]
postreise/analyze/transmission/tests/test_mwmiles.py ....                                                                                                [ 59%]
postreise/analyze/transmission/tests/test_statelines.py .....                                                                                            [ 63%]
postreise/plot/multi/tests/test_plot_bar.py ..........                                                                                                   [ 71%]
postreise/plot/multi/tests/test_plot_helpers.py ...............                                                                                          [ 82%]
postreise/plot/multi/tests/test_plot_pie.py ....                                                                                                         [ 86%]
postreise/plot/multi/tests/test_plot_shortfall.py .........                                                                                              [ 93%]
postreise/plot/tests/test_plot_shadowprice.py .........                                                                                                  [100%]

===================================================================== 129 passed in 18.69s =====================================================================
(PostREISE) [~/CEM/PostREISE] (develop) brdo$

jenhagg · 2020-08-03T22:34:13Z

Ha, awesome. I got an error with local branch in PostREISE but seems that's the only thing affected. All good from my side.

rouille · 2020-08-04T00:38:20Z

Just realize I forgot to update the path to the data in the setup.py file.

danielolsen · 2020-08-04T19:49:32Z

        self.ct = ct

-        self.TMP_DIR = "%s/scenario_%s" % (const.EXECUTE_DIR, scenario_info["id"])
+        self.TMP_DIR = "%s/scenario_%s" % (


Do you think it would be cleaner to do this with posixpath.join?

danielolsen

Looks good.

BainanXia · 2020-08-04T21:30:29Z

-            raise IOError(err_message[0].strip())
-
+        err_message = "Failed to generate id for new scenario"
+        stdout = self._execute_and_check_err(command, err_message)


Where is 'command' defined? Should it be 'script' instead?

BainanXia · 2020-08-04T21:37:27Z

+    :raises ValueError: if model or engine does not exist.
    """

    def __init__(self, interconnect, source="usa_tamu", engine="REISE"):


Should we change the default engine into 'REISE.jl' some time.

We changed the default when creating a scenario in #229. I am not sure there is a point in having a default here, since all entries in the ScenarioList should be appropriately tagged.

A Grid object does not have access to the ScenarioList file. And we need to know the engine to appropriately read the *_grid.mat files. This optional argument is kind of transparent because it is handled in the _set_ct_and_grid method of the Analyze class

BainanXia

Thanks for taking care of this tedious work. I think the reorganization makes the package structure and naming more reasonable.

refactor: rename module const as server_setup

e6c193e

rouille added this to the A New Hope milestone Aug 3, 2020

rouille requested review from BainanXia, danielolsen and jenhagg August 3, 2020 03:48

rouille assigned danielolsen, rouille, BainanXia and jenhagg Aug 3, 2020

jenhagg reviewed Aug 3, 2020

View reviewed changes

jenhagg approved these changes Aug 3, 2020

View reviewed changes

rouille linked an issue Aug 3, 2020 that may be closed by this pull request

Reorganize package #244

Closed

9 tasks

rouille modified the milestones: A New Hope, Oppressive Sun Aug 3, 2020

danielolsen reviewed Aug 4, 2020

View reviewed changes

rouille force-pushed the ben/organize branch from 4db0fbf to 7f92f37 Compare August 4, 2020 21:02

danielolsen approved these changes Aug 4, 2020

View reviewed changes

BainanXia reviewed Aug 4, 2020

View reviewed changes

Comment thread powersimdata/design/scenario_info.py

BainanXia reviewed Aug 4, 2020

View reviewed changes

BainanXia approved these changes Aug 4, 2020

View reviewed changes

rouille added 6 commits August 4, 2020 14:53

refactor: customize error messages

346562d

refactor: move PrintManager class

4f02480

test: write test for PrintManager class

cc5695a

refactor: rename module profiles as output_data

588a97c

refactor: rename module profiles as input_data

380bb20

refactor: move TAMU data and related modules into new folder network

2d02146

rouille added 2 commits August 4, 2020 14:53

refactor: move constants module

5790fa5

chore: modify path to data in setup

4fce95b

rouille force-pushed the ben/organize branch from 7f92f37 to 4fce95b Compare August 4, 2020 21:53

rouille merged commit 00848fd into develop Aug 4, 2020

rouille deleted the ben/organize branch August 4, 2020 22:14

danielolsen mentioned this pull request Oct 29, 2020

fix: typo in delete_entry #325

Merged

ahurli mentioned this pull request Mar 11, 2021

Develop into Master #410

Merged

Conversation

rouille commented Aug 3, 2020

Purpose

What is the code doing?

Where to look

Time estimate

Uh oh!

jenhagg Aug 3, 2020

Choose a reason for hiding this comment

Uh oh!

jenhagg left a comment

Choose a reason for hiding this comment

Uh oh!

rouille commented Aug 3, 2020

Uh oh!

jenhagg commented Aug 3, 2020

Uh oh!

rouille commented Aug 4, 2020

Uh oh!

danielolsen Aug 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rouille Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

danielolsen left a comment

Choose a reason for hiding this comment

Uh oh!

BainanXia Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

rouille Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BainanXia Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

danielolsen Aug 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rouille Aug 4, 2020

Choose a reason for hiding this comment

Uh oh!

BainanXia left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danielolsen Aug 4, 2020 •

edited

Loading

danielolsen Aug 4, 2020 •

edited

Loading