adjustments to be able to run workflows in dev branch #218
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AnnaKwa
changed the title
AnnaKwa
changed the title
added 18 commits
April 8, 2020 18:23
added 4 commits
April 8, 2020 23:54
…ng/fv3net into dev-feature/train-ml
brianhenn
approved these changes
Apr 9, 2020
| rest = [dim for dim in ds[[var_name]].dims if dim not in first_dims] | ||
| xpose_dims = first_dims + rest | ||
| new_ds = ds[[var_name]].copy().transpose(*xpose_dims) | ||
|
|
||
| for grid_var in _COORD_VARS: | ||
| for grid_var in coord_vars: | ||
| new_ds = new_ds.assign_coords(coords={grid_var: ds[grid_var]}) | ||
|
|
||
| return new_ds.drop( |
There was a problem hiding this comment.
OK I checked and this is not actually necessary for plotting open_restarts datasets, but it is nice to remove the extraneous coords in that case since the point of mappable_var is to reset the coords to the grid vars. So I would suggest the following:
Suggested change
| return new_ds.drop( | |
| for coord in [coord_y_center, coord_x_center, coord_y_outer, coord_x_outer]: | |
| if coord in new_ds.coords: | |
| new_ds = new_ds.drop(coord) | |
| return new_ds |
Co-Authored-By: brianhenn <brianhenn@gmail.com>
nbren12
added a commit
that referenced
this pull request
Apr 13, 2020
* Feature/one step save baseline (#193) This adds several features to the one-step pipeline - big zarr. Everything is stored as one zarr file - saves physics outputs - some refactoring of the job submission. Sample output: https://gist.github.com/nbren12/84536018dafef01ba5eac0354869fb67 * save lat/lon grid variables from sfc_dt_atmos (#204) * save lat/lon grid variables from sfc_dt_atmos * Feature/use onestep zarr train data (#207) Use the big zarr from the one step workflow as input to the create training data pipeline * One-step sfc variables time alignment (#214) This makes the diagnostics variables appended to the big zarr have the appropriate step and forecast_time dimensions, just as the variables extracted by the wrapper do. * One step zarr fill values (#223) This accomplishes two things: 1) preventing true model 0s from being cast to NaNs in the one-step big zarr output, and 2) initializing big zarr arrays with NaNs via full so that if they are not filled in due to a failed timestep or other reason, it is more apparent than using empty which produces arbitrary output. * adjustments to be able to run workflows in dev branch (#218) Remove references to hard coded dims and data variables or imports from vcm.cubedsphere.constants, replace with arguments. Can provide coords and dims as args for mappable var * One steps start index (#231) Allows for starting the one-step jobs at the specified index in the timestep list to allow for testing/avoiding spinup timesteps * Dev fix/integration tests (#234) * change order of required args so output is last * fix arg for onestep input to be dir containing big zarr * update end to end integration test ymls * prognostic run adjustments * Improved fv3 logging (#225) This PR introduces several improvements to the logging capability of our prognostic run image - include upstream changes to disable output capturing in `fv3config.fv3run` - Add `capture_fv3gfs_func` function. When called this capture the raw fv3gfs outputs and re-emit it as DEBUG level logging statements that can more easily be filtered. - Refactor `runtime` to `external/runtime/runtime`. This was easy since it did not depend on any other module in fv3net. (except implicitly the code in `fv3net.regression` which is imported when loading the sklearn model with pickle). - updates fv3config to master * manually merge in the refactor from master while keeping new names from develop (#237) * lint * remove logging from testing * Dev fix/arg order (#238) * update history * fix positional args * fix function args * update history * linting Co-authored-by: Anna Kwa <annak@vulcan.com> Co-authored-by: brianhenn <brianhenn@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The land sea mask variable now has NaN values instead of 0's. This PR converts those to 0 in the training data pipeline.This is now fixed in the zarr write step: #223Fix references to old variable names in training data, model training, offline diagnostics, and prognostic run workflows.