-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Train/test model with blurred data #74
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The pre-processing pipeline was too confusing, and was scatter accross several snakemake rules. This commit combines these into one script called uwnet/data/preprocess.py.
There is no longer a `step` dimension in the training data.
The data blurred with a radius of xxx will be stored at data/processed/training/sigmaxxx.nc The unblurred data will be stored at data/processed/training/noBlur.nc
This commit makes it easier to identify rules related to pre-processing
training doesn't work in this commit
Previously it was hard debugging errors with the input data.
The new pre-processed data has a time varying layer_mass dimension, which broke the metrics calculation.
It now runs on olympus using the `sam_path` specified in the configuration file.
one is for fast debugging purpose one is for the blurred data
model_run_path needs to be set
nbren12
added a commit
that referenced
this pull request
Nov 13, 2020
To simulate the effect of coarse-resolution data, we can test/train the NN on blurred training data. Changes needed: * Add script for blurring the data * Refactor and improve SAM-based pre-processing scripts. The pre-processing pipeline was too confusing, and was scatter across several snakemake rules. Now these are combined into one script: `uwnet/data/preprocess.py.` * Automate NN training and SAM simulation with snakemake These steps had to be executed manually because Sacred generated the folder names automatically. Now the model training and SAM runs are named based on the filename of the json file used to train them. * Improve training messages Previously it was hard debugging errors with the input data. Also use the agg backend for plots, so that the training does not die on olympus.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.