Decouple noise floor argo workflow from fme#1083
Merged
Merged
Conversation
spencerkclark
changed the title
fme and update configurations to include frozen precipitationfme and update configurations to include frozen precipitation
Broadcast area weights so that we can compute means of fields that are not already 2D over "grid_yt" and "grid_xt". Xarray's weighted will automatically broadcast weights to match each data variable's dimensions, but scalar coordinates like ak and bk do not have such dimensions to broadcast to and therefore taking a mean over "grid_yt" and "grid_xt" raises an error. Another option would be to drop these variables when computing the noise floor, but we will keep things this way for backwards compatibility.
spencerkclark
force-pushed
the
scripts/update-noise-floor
branch
from
34e76e9 to
a31de0e
Compare
spencerkclark
changed the title
fme and update configurations to include frozen precipitationfme
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR decouples the argo portion of the noise floor workflow from
fmeso that we can run it using the standard data processing image instead of an oldfmeimage. Oldfmeimages cannot open zarr v3 stores, but newfmeimages do not have dask installed, which is necessary for this workflow. The only coupling was in computing the area weights, which is straightforward to do without a special function.The main motivation for this is to enable computing noise floor statistics for the
total_frozen_precipitation_rate, which is only present in our newer intermediate zarr v3 stores.Note I have not touched
generate_train_report.py, which still depends onfme, but is typically run in a local environment, and does not require dask.Changes:
scripts/noise_floor/generate_stats.get_output_datasetsso that we do not need to import a function fromfme.fme.core.distributed.distributed.Distributedcontext, since we are no longer calling anything infme.total_frozen_precipitation_rateto thenames, and updates the outputstats_pathto new locations for output from the re-run workflow for each.Successful argo jobs (the longer one is the AMIP case):