Reeds #18

MRossol · 2019-11-29T22:00:31Z

reV to ReEDS pipeline w/ CLI

MRossol · 2019-11-29T22:01:01Z

Main functionality implemented along with a CLI
TODO: Create tests, will do so before chat on Thursday

MRossol · 2019-11-30T22:24:19Z

Test complete.

@mmowers see /reVX/tests/data/reeds/ReEDS_* for the current outputs

mmowers · 2019-12-02T19:27:10Z

Hi @MRossol,
Just confirming, are these the two files for me to review?:

MRossol · 2019-12-02T19:29:09Z

@mmowers, No, the files starting with ReEDS:
https://github.com/NREL/reVX/blob/reeds/tests/data/reeds/ReEDS_classifications.csv
https://github.com/NREL/reVX/blob/reeds/tests/data/reeds/ReEDS_Profiles.h5
https://github.com/NREL/reVX/blob/reeds/tests/data/reeds/ReEDS_Timeslice_means.csv
https://github.com/NREL/reVX/blob/reeds/tests/data/reeds/ReEDS_Timeslice_stdevs.csv

mmowers · 2019-12-02T20:36:52Z

@MRossol, awesome thanks, I was on the master branch instead of reeds, so those files weren't there.

I see the supply curve in region/class/bin designations in ReEDS_classifications.csv, capacity factor means and standard deviations by timeslice in ReEDS_Timeslice_means.csv and ReEDS_Timeslice_stdevs.csv, and representative profiles in ReEDS_Profiles.h5.

Some thoughts/questions (we can discuss on the phone if easier):

Are you still ultimately shooting for the input/output structure of the doc? If not we should discuss and align.
Are all these outputs all for PV or onshore wind?
ReEDS_classifications.csv:
1. is site_lcoe called "mean_lcoe"?
2. Do you have a run of the full supply curve somewhere as well that i could see?
3. What is the "res_class" column referring to?
4. I see two region columns, "reeds_region" and "region". They look identical, are they?
5. "bin" is the supply curve bin, correct?
6. In this case, is "class" is based on wind speed or LCOE?
ReEDS_Timeslice_means.csv and ReEDS_Timeslice_stdevs.csv:
1. What do the column headers mean? region/class/bin? If so, we don't need the bin designation- we only need this data by timeslice, region, and class (see "performance_[tech].csv" in the doc)
ReEDS_Profiles.h5:
1. What are the separate tables, rep_profiles_0, rep_profiles_1, and rep_profiles_2?
2. It looks like there may be a separate profile for reach region/bin/class. If so, we only need a profile by region and class (see "hourly_cf_[tech].pkl" in the doc)
3. It looks like this data is for every 30 mins, could we get this for hourly instead?

MRossol · 2019-12-02T22:01:23Z

@mmowers See inline below:

Are you still ultimately shooting for the input/output structure of the [doc]
(https://docs.google.com/document/d/1SEOafxhZphXw7nFARVpQ4L1J22GZNh45wKkyCBqBdEU)? If not we should discuss and align.

We are happy to try and create the outputs you need, this was the first attempt, I would like to discuss the input formats

Are all these outputs all for PV or onshore wind?

These are all for PV, it was just for testing purposes, i.e. its what we had

ReEDS_classifications.csv:
1. is site_lcoe called "mean_lcoe"?
- yes
1. Do you have a run of the full supply curve somewhere as well that i could see?
- not yet
1. What is the "res_class" column referring to?
- It is the resource class defined by reV to select technology
1. I see two region columns, "reeds_region" and "region". They look identical, are they?
- In this case they are the same, I can walk through how it works tomorrow
1. "bin" is the supply curve bin, correct?
- We need to talk about this tomorrow as I am very confused by your nomenclasture between resource classes (TRGs), supply curve bins, and "clusters"
1. In this case, is "class" is based on wind speed or LCOE?
- I know this is not "accurate" but these are TRG classes even though its for solar, I wanted to test the code
ReEDS_Timeslice_means.csv and ReEDS_Timeslice_stdevs.csv:
1. What do the column headers mean? region/class/bin? If so, we don't need the bin designation- we only need this data by timeslice, region, and class (see "performance_[tech].csv" in the doc)
- Good to know can update after we confirm what classes are vs bins...
ReEDS_Profiles.h5:
1. What are the separate tables, rep_profiles_0, rep_profiles_1, and rep_profiles_2?
- This is an example of pulling 3 representative profiles for each "region", 0 is the most representative, followed by 1, then 2.
1. It looks like there may be a separate profile for reach region/bin/class. If so, we only need a profile by region and class (see "hourly_cf_[tech].pkl" in the doc)
2. It looks like this data is for every 30 mins, could we get this for hourly instead?
- the NSRDB is natively at 30minutes and my preference would be to NOT downscale to hourly. you can instead just pull every other datapoint if you want hourly. It seems short-sighted to remove data...

grantbuster

Looks good overall but will definitely benefit from a walk through. I assume we'll have other thoughts after tomorrow's regroup but here are some initial thoughts.

grantbuster · 2019-12-02T22:03:35Z

reVX/reeds/reeds_classification.py

+        if isinstance(cluster_on, str):
+            cluster_on = [cluster_on, ]
+
+        data = RPMClusters._normalize_values(rev_table[cluster_on].values,


Might as well move this method to a common utility repo with the clustering algorithms?

Good call, will do

Added it to the ClusterMethods class

grantbuster · 2019-12-02T22:07:44Z

reVX/reeds/reeds_profiles.py

+logger = logging.getLogger(__name__)
+
+
+class ReedsProfiles(RepProfiles):


This is awesome!

You did 90% of the work!

grantbuster · 2019-12-02T22:08:15Z

reVX/reeds/reeds_profiles.py

+            profile.
+        n_profiles : int
+            Number of representative profiles to save to fout.
+        bins : None | str | pandas.DataFrame | pandas.Series | dict


Might want more description on what None does since its default. Also maybe corresponding description of the reg_cols defaults.

grantbuster · 2019-12-02T22:09:16Z

reVX/reeds/reeds_timeslices.py

+            raise ReedsValueError(msg)
+
+        index_col = [c for c in timeslice_map.columns
+                     if 'time' in c.lower()]


I think the search string should be "datetime" or even "datetimeindex". I'm picturing two columns, one with "datetimeindex" and the other with "timeslice_id" (has "time" in it!). Plus all of the error messages say "datetime".

I'll admit I got lazy here, but I agree, will update

grantbuster · 2019-12-02T22:12:53Z

reVX/reeds/reeds_timeslices.py

+        """
+        means = []
+        stdevs = []
+        for s, slice_map in timeslices.groupby('slice'):


I see that your slice ID column is just called "slice". Can you add a check for this in _parse_timeslices()?

Whoops, meant to run the groupby on all remaining columns, will fix

grantbuster · 2019-12-02T22:17:47Z

reVX/reeds/reeds_timeslices.py

+    Create ReEDS timeslices from region-bin-class groups and representative
+    profiles
+    """
+    def __init__(self, rep_profiles, timeslice_map, meta=None,


We need to verify that the timeslice statistics are calculated from representative profiles and not profiles from all sites in a region/timeslice. I'm totally not sure, but that would be a high level misunderstanding.

Looking at @mmowers' doc, it would appear this is an open question.

grantbuster · 2019-12-02T22:18:52Z

reVX/reeds/reeds_timeslices.py

+        stdevs = pd.concat(stdevs, axis=1).T
+        stdevs.index.name = 'timeslice'
+
+        return means, stdevs


Clean method! Nice!

grantbuster · 2019-12-02T22:21:54Z

reVX/reeds/reeds_classification.py

+logger = logging.getLogger(__name__)
+
+
+class ReedsClassifier:


I think this looks good but I definitely would benefit from a walk through :)

grantbuster · 2019-12-03T16:35:34Z

Michael and Grant code review notes:

Classification items:

Expose "groups" (region, bin, class) as properties: groups, keys. (Done)
Raise key error in getitem (Done)
Agg table - specify mean/sum for different vars (Done)
Region input might be a shape file? Should be a global utility with cli. (I can do this)

Timeslices:

Might want to run with stats on the full CF profiles for all sites in each unique region/bin/class/timeslice. Might want to pass in gen output handler and gid list to parallel workers.

MRossol · 2019-12-07T21:06:07Z

@grantbuster
Reeds should be done, I've implemented legacy formats for profiles and timeslices.

I would love your eyes on these two formatting methods in ReedsTimeslices as they are really slow but I'm not sure how else to speed them up without a for loop...
def _flatten_timeslices(table, value_name, reg_cols):
def _create_correlation_table(corr_coeffs, reg_cols):

The timeslice CLI entry needs to be updated on your branch, hopefully we can do that and merge it monday...

grantbuster · 2019-12-12T18:36:48Z

Call with Matt:
• Regions - need to allow a subset of regions
• Timeslices - option to use hour number (to "match" end of hour data)
• Classes - Make sure classes can be in any sorted order (min/max)
• Classes need to start at 1
• Bins need to start at 1

remove redundant logging functions

…rest

protect hour against "overflow"

…nteger, and process pool with spawn option

…eds specifications

…tting

…default.

grantbuster · 2020-01-09T15:43:05Z

Merging this pull request since we've tested thoroughly.

MRossol added the feature New feature or request label Nov 29, 2019

MRossol requested review from mmowers and grantbuster November 29, 2019 22:00

MRossol assigned grantbuster and MRossol Nov 29, 2019

MRossol force-pushed the reeds branch from 15f92dd to 76a7900 Compare December 2, 2019 19:19

grantbuster requested changes Dec 2, 2019

View reviewed changes

MRossol force-pushed the reeds branch from c2ce44e to 1f533dd Compare December 3, 2019 02:47

MRossol force-pushed the reeds branch from 022ca43 to 8b94693 Compare December 5, 2019 02:57

grantbuster force-pushed the reeds branch 2 times, most recently from 7e935ed to 3608af5 Compare December 11, 2019 23:48

MRossol force-pushed the reeds branch 2 times, most recently from f68d5a9 to 387807f Compare December 16, 2019 16:08

grantbuster force-pushed the reeds branch 2 times, most recently from 83d3e2c to 4a7a6c3 Compare December 24, 2019 00:59

MRossol added 5 commits January 3, 2020 10:09

add relative sub-package imports

ed08ada

remove redundant logging functions

start reV to ReEDS framework

efd9f88

start ReEDS classes

7e717ad

init for ReEDS classes finished

2905330

add ReEDSProfiles

6ad8c50

MRossol and others added 12 commits January 3, 2020 10:11

start working on timeslice

2338344

allow for region_map to be a subset of the values in a column of inte…

bd696e1

…rest

fix docs config and update docs

02c276e

update timeslice mapping to take datetime or hour

3b53396

protect hour against "overflow"

add minute to get_hour_of_year calculation

1f77671

fix to rep_profiles_stats

20b93c1

reprocess test files with bins starting at 1

8ee5cde

pass through kwargs as much as possible

fe83cbd

various renames and better rep profile output formatting

9e91dff

timezone fix to profiles

0d6f1a8

bug fixes on reeds timeslices including timezone issues, roll using i…

680254c

…nteger, and process pool with spawn option

minor bug fix to profile timezone export

e5fb08b

grantbuster force-pushed the reeds branch from db85c95 to e5fb08b Compare January 3, 2020 17:13

grantbuster added 7 commits January 3, 2020 10:28

added slimed down table output for reeds classification as per the re…

96bf4b1

…eds specifications

Lots of bug fixes to get tests to pass again.

5ad6abe

seperated out timeslice stats formatting and coeff table legacy forma…

85ae293

…tting

Added a correlation dictionary to h5 output method and made this the …

0ffc4bd

…default.

minor bug fix on correlation output extension csv -> h5

91adc33

added integer scaling for timeslice correlations

01bffd6

added compression

925b3f9

grantbuster force-pushed the reeds branch from 0244bfd to 925b3f9 Compare January 3, 2020 21:43

grantbuster added 3 commits January 3, 2020 15:39

added matrix sparsification for data size reduction

36c70f7

changed reeds output tables as per request by Matt

cfbb8c1

disabled matrix sparsification by default

b2446da

grantbuster approved these changes Jan 9, 2020

View reviewed changes

grantbuster merged commit e646323 into master Jan 9, 2020

grantbuster deleted the reeds branch January 9, 2020 15:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reeds #18

Reeds #18

MRossol commented Nov 29, 2019

MRossol commented Nov 29, 2019

MRossol commented Nov 30, 2019

mmowers commented Dec 2, 2019

MRossol commented Dec 2, 2019

mmowers commented Dec 2, 2019

MRossol commented Dec 2, 2019 •

edited

Loading

grantbuster left a comment

grantbuster Dec 2, 2019

MRossol Dec 3, 2019

MRossol Dec 3, 2019

grantbuster Dec 2, 2019

MRossol Dec 3, 2019

grantbuster Dec 2, 2019

grantbuster Dec 2, 2019

MRossol Dec 3, 2019

grantbuster Dec 2, 2019

MRossol Dec 3, 2019

grantbuster Dec 2, 2019

grantbuster Dec 2, 2019

grantbuster Dec 2, 2019

grantbuster commented Dec 3, 2019

MRossol commented Dec 7, 2019 •

edited

Loading

grantbuster commented Dec 12, 2019

grantbuster commented Jan 9, 2020

		logger = logging.getLogger(__name__)


		class ReedsProfiles(RepProfiles):

Reeds #18

Reeds #18

Conversation

MRossol commented Nov 29, 2019

MRossol commented Nov 29, 2019

MRossol commented Nov 30, 2019

mmowers commented Dec 2, 2019

MRossol commented Dec 2, 2019

mmowers commented Dec 2, 2019

MRossol commented Dec 2, 2019 • edited Loading

grantbuster left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grantbuster commented Dec 3, 2019

MRossol commented Dec 7, 2019 • edited Loading

grantbuster commented Dec 12, 2019

grantbuster commented Jan 9, 2020

MRossol commented Dec 2, 2019 •

edited

Loading

MRossol commented Dec 7, 2019 •

edited

Loading