IM quality control #36
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…termine change in gradient with depth)
|
Closes #25 |
Closed
Closed
durbank
added a commit
that referenced
this pull request
Jun 5, 2020
* Hotfix for AGU2019 (#34) * Adds new function to load depth-density data from stats model output * Updates PAIPR functions to integrate with the new 'load_rho.m' function * Allows UI selection of rho data file for each echogram * Stops the clearing of logistic parameters prior to use * Adds new function to load depth-density data from stats model output * Updates PAIPR functions to integrate with the new 'load_rho.m' function * Allows UI selection of rho data file for each echogram * Subsets the raw modeled density and moves outside parfor loop for increased processing speed/reduced memory * Removes tic/toc lines (mismatch causing script error) * New functionality to use integrated stream brightness rather than peak finding and grouping * Increases window length and spacing of window centers * Updates layer tracing to use full unclipped size of echograms * Adds r,k parameters as input options to `opt_param.m` * Optimiziation script (for logistic parameters) for new integrated stream method * Changes RT horizontal window size to 500 m * Clips annual results to years with 2/3 simulation coverage, rather than 100% * Saves the `IM_gradients` matrix to the output `radar` structure * Updates optimization comparisons to v1.3 data * Fixes radar depth cutoff based on bottom streamline positions * Updates optimizations based on smaller horizontal `radar_gradient.m` RT window * Improves variable struct structure of `echo_format` output; Saves the output of echo_format to tmp file in 'data_path' * Addresses issue where streamliness had multiple rows in some cols in high gradient regions, leading to adjacent high likelihood years of 0 accum * Modififications to run PAIPR on SEAT Ku-band echograms with less optimal data cleaning * Removes traces with invalid timestamps * Removes NaN results due to bad trace data (inf or nan gradients generated) * Fixes issues with age assignments: errors in defining surface age; enforces montonicity on ages with each trace * Re-implementation of tracing algorithm (#35) * Modifies radar_trace.m to iteratively recalculate integrated streamline brightness to select starting layer trace position * Re-implements `calc_age.m` as the default (using `radar_trace` for layer generation). REQUIRES new optimization * WIP: Updates regression optimization for new layer tracing * Updates optimization of logistic params for new tracing functions; deprecates certain old scripts * WIP: Updates PAIPR functions to use the new logistic parameters * Changes Windows data directory to reflect changes * IM quality control (#36) * Simplifies stationarizing using differencing * WIP: Calculates a QC ratio based on differences in max IB and angles * WIP: Modifies pk_ratio; performs robust OLS are grouped traces (to determine change in gradient with depth) * Simplifies stationarization using spline differencing and columnwise z-scores * Generates echogram QC image based on consistency of radon transforms * Tracing switch case (#40) * Updates repo structure for better standardization * Adds function to determine layers based on two methods: layer tracing and RT streams * Updates PAIPR to utilize the new option of layer methods * Adds function to chunk raw OIB echograms into 30 km subdomains * Adds script to generalize and update optimization of logistic regression coefficients * Improved function for drawing manual layers on echograms * Adds tmp script for trying new PAIPR methods * Streamlines optimization of log parameters for greater general applicablility * QC-output (#42) * Changes depth overlap; adds QC interpolation (commented out) * Adds function to flag echograms with sufficiently poor data * Updates to save QC summary statistics/flags to outputs * CHPC (#43) * WIP: Adds functionality to run on CHPC resources * Adds ignore file for generated figures/docs * Adds PAIPR dependencies directly to src/ directory * Adds required scripts for processing PAIPR on CHPC * WIP:updates inputs to match chpc system * Working test example of slurm script * Adds bash script to download rclone data in DMZ * Adds if statement for clean MATLAB execution * Removes redundant script * Adds execution timer * Imports and formats long-format .csv files of density profiles * Pulls rho file from gcloud; switch to lonepeak processing * Adds printing statements to track progress * Corrects typo * Adds error message handling to batch scripts * Fixes bugs for depth_idx good data cutoff * Improves readability of caught error messages Co-authored-by: Durban Keeler <u1046484@frisco1.wasatch.peaks> Co-authored-by: Durban Keeler <u1046484@dtn01-dmz.chpc.utah.edu> Co-authored-by: Durban Keeler <u1046484@lonepeak1.wasatch.peaks> * Qc mod (#45) * Adds switch case for whether to overlap echograms; adds script for processing PAIPR locally * Adds name-value pair to inputs for whether to save .mat results to disk * Modifies the QC check to give good, good to certain depth, and poor QC flags * Renames scripts/functions for improved readability; updates calls accordingly * Renames parsave_all to parsave; updates functions accordingly * Adds --progress flag to rclone transfers * Updates in-function names to reflect file name changes * Improves error handling to include error stack * Addresses issue with depth_idx cutoff; expands warning limits under favorable circumstances * Handles multiple rho files; suppresses robust stats warnings; fixes stack error handling * Modifies DMZ download to reflect subdirectories in data * Improves error handling at various points in process Co-authored-by: Durban Keeler <u1046484@frisco1.wasatch.peaks> Co-authored-by: Durban Keeler <u1046484@frisco1.wasatch.peaks> Co-authored-by: Durban Keeler <u1046484@dtn01-dmz.chpc.utah.edu> Co-authored-by: Durban Keeler <u1046484@lonepeak1.wasatch.peaks>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements automated measurements of quality of image for processing during the RT transform steps. Additional work is still required to determine and implement the best way to utilize these measurements.