Gb/plexos agg #8
Conversation
…ob config input csv with explicit groups
…on a different node
…with duplicate lat/lon
… aggregation pipeline
- plexos single node agg - data cleaner tests - high level aggregation buildout integrated test
grantbuster
force-pushed
the
gb/plexos_agg
branch
from
2f51d2e
to
ee9ea3f
Compare
| @@ -0,0 +1,102 @@ | |||
| # -*- coding: utf-8 -*- | |||
| """reX SQL Database handler. | |||
There was a problem hiding this comment.
Ug database. Is there any way we can deprecate this and have the GDS team give us flat files or is this the crux of the reV -> ReEDS -> reV -> PLEXOS pipeline?
There was a problem hiding this comment.
After discussion, our preferred method would be to have the DB handler (both retrieval and push) be in reVX, then have an optional DB push step in the reV pipeline that imports from reVX within a try except statement, so that it's not a hard dependency.
reVX/plexos/dpv_plexos.py
Outdated
| """ | ||
|
|
||
| self._root_dir = root_dir | ||
| self._sub_dirs = sub_dirs |
There was a problem hiding this comment.
Would it be useful to have this be optional? I.e., if it's None than we would search the root_dir for all sub_dirs and ignore a pre-defined set (e.g., stdout or stderr)
There was a problem hiding this comment.
Yes thats a good idea, actions:
- add parse subdirs method
- remove subdirs from classmethod entry point
- input subdirs kwarg from job frac map
| self._year = year | ||
| self._fpaths = self._get_all_fpaths() | ||
|
|
||
| def _get_sub_dir_fpath(self, sub_dir): |
There was a problem hiding this comment.
What happens if there are multiple .h5 files in the sub directory?
There was a problem hiding this comment.
Better idea: make list of fpaths that fit the criteria, if more than one, raise.
reVX/plexos/dpv_plexos.py
Outdated
| """ | ||
|
|
||
| if not sub_dir.endswith('/'): | ||
| sub_dir += '/' |
There was a problem hiding this comment.
why do you need the backsplash?
| fpaths.append(self._get_sub_dir_fpath(sub_dir)) | ||
| return fpaths | ||
|
|
||
| def _merge_data(self, dset, job_frac_map): |
There was a problem hiding this comment.
From different tilt/azimuth combinations. Added details in docstring.
|
|
||
| return rev_sc, reeds_build | ||
|
|
||
| def _make_forecast_map(self): |
There was a problem hiding this comment.
We shouldn't need this anymore
reVX/plexos/rev_reeds_plexos.py
Outdated
| plexos_coord_labels = self._get_coord_labels(self._plexos_nodes) | ||
| sc_coord_labels = self._get_coord_labels(self._sc_build) | ||
| tree = cKDTree(self._plexos_nodes[plexos_coord_labels]) | ||
| _, i = tree.query(self._sc_build[sc_coord_labels], k=k) |
There was a problem hiding this comment.
Any concerns about plexos nodes and resource not being near each other?
There was a problem hiding this comment.
Definitely copy in the logging info statement from dpv run.
| self._ammend_plexos_meta(i, res_gids, gen_gids, res_built) | ||
|
|
||
| else: | ||
| for i, inode in enumerate(np.unique(self._node_map)): |
There was a problem hiding this comment.
Add completion logging here too
There was a problem hiding this comment.
Will add % complete logging into both parallel and serial.
| profiles[:, i] = profile | ||
| self._ammend_plexos_meta(i, res_gids, gen_gids, res_built) | ||
|
|
||
| logger.info('Finished plexos node aggregation.') |
There was a problem hiding this comment.
Remove after adding % complete to both parallel and serial profiles making
|
|
||
| return meta, profiles | ||
|
|
||
| def merge_extent(self, new_meta, new_profiles): |
There was a problem hiding this comment.
Merges different extents and technologies - pv urban/rural, can/con/mex.
|
@MRossol, review comments resolved. |
Plexos aggregation complete. Just finished adding test files. No rush on this as it is not a FY2019 goal.