Reduce verbosity of output during inference and evaluation #2006
Merged
Reduce verbosity of output during inference and evaluation #2006
Conversation
github-actions
bot
added
the
eval
Collaborator
Author
|
@iluise : can you have a look |
iluise
approved these changes
Mar 9, 2026
Jubeku
added a commit
that referenced
this pull request
Mar 10, 2026
* Improve support for latent losses (#1963) * Revert 2D rope to false by default (#1967) Set to True by accident * Implementation of DataReaderMesh (#1840) * First implementation of DataReaderMesh * Move to datareaders extra * ruff * ruff2 * Undo ruff * undo auto-linting * correct typo in eval config (#1971) * Added all-physical-streams option and x/y axis limits (#1972) * Added all-physical-streams option and x/y axis limits * Fix * Changed flag for all streams * Removed old code * moved metric parsing to eval_from_config (#1977) Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> * Fixed integration test (#1980) * [1974][model] Add fallback to config loading (#1985) * Add fallback to config loading * Adjust error message to be not misleading * Homegenize naming convention * Introduce bias/diff maps and animations (#1912) * Introduce bias/diff maps and animations * minor correction * Changes based on review * Introduce "plot_bias" in evaluation configuration (#1986) * Fixed index ordering to not have shuffled output (#1982) * Fixed idxs_inv to revert data point shuffeling * Fixed output handling * Handling of empty data case, addressing reviewer comment * [1893][eval] csvreader cleanup (#1906) * refactor csvreader * check if dataarray size is 0 * fix and use original logic for empty data * linting fixes * revert assertions back * [1890][eval] Move MergeReader to own module (#1892) * move mergereader * use assertions only * implement scoring for the sub-steps within the forecast window (#1896) * work in progress * working for forecast_step * working version * restore no valid times option * lint * Rename scale_z_channels to _scale_z_channels * fix 1 sample bug * Remove points_per_sample from ReaderOutput Remove points_per_sample from ReaderOutput return. * remove n_point_per_sample * fix lead time coord in compute_scores * lint * fix integration test * Fix integration test single stream (#1996) * fix test single * change yml extension and minor fixes --------- Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> * [1907][eval] clean up wegen_reader.py (#1911) * clean up wegen_reader.py * remove exception * consistent reader naming * add blank line * use assertions only * make names consistent * Merge branch 'develop' into 1907-wegenreader-cleanup * revert is_regular --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1888][eval] Refactor Reader class (#1889) * refactor Reader * use assertion only * fix npp atms --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1975][model] Load model path from private repo instead of json (#1998) * Load model path from private repo instead of json * Lint * Script to compute spatial autocorrelation of structured/unstructured datasets (#1955) * standalone script to compute spatial autocorrelation of variables in a structured or unstructured dataset * remove commits that should be in pr 1951 * lint * addressed comments * removed last failure returning 500km default, and moved to packages science * updated a note * rename autocorrelation script * update example usage * Correct EMA halflife_steps calculation with rampup_ratio (#2001) Corrected rampup calculation: https://github.com/NVlabs/edm2/blob/4bf8162f601bcc09472ce8a32dd0cbe8889dc8fc/training/phema.py#L145 Co-authored-by: Wael <wael.almikaeel.95@gmail.com> * Reduce verbosity of output during inference and evaluation (#2006) * Fix incorrect length in validation progress bar * Removing too verbose output * [1766][1743][1332] lint and unit-test fix (#1802) * [1766][1742] fix lint and unit-test * [1766] fix linter * [1766] lint local and global consistent * [1332] add script to detect bad functions (getattr) * code quality: lint and bad functions * [1766] disable some checks * [1877] Script to populate PR labels from linked issues (#1878) * script * branch * more dirs * typo * enable * Fixed bug in linear embedding (#2012) * Adding forecast_steps feature to plot_train (#2010) * Adding forecast_steps feature to plot_train * Renamed arguement to conform to hyphen convention * Added forecast step to filename --------- Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> --------- Co-authored-by: Christian Lessig <christian.lessig@ecmwf.int> Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> Co-authored-by: Kacper Nowak <kacper.nowak@awi.de> Co-authored-by: Till Hauer <till@web-hauer.de> Co-authored-by: s6sebusc <49226935+s6sebusc@users.noreply.github.com> Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> Co-authored-by: Matthias Karlbauer <matthias.karlbauer@ecmwf.int> Co-authored-by: Savvas Melidonis <79579567+SavvasMel@users.noreply.github.com> Co-authored-by: Michael Tarnawa <18899420+mtar@users.noreply.github.com> Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: pierluigicosi <91318382+pierluigicosi@users.noreply.github.com> Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> Co-authored-by: Wael <wael.almikaeel.95@gmail.com> Co-authored-by: Simone Norberti <63310821+simone99n@users.noreply.github.com> Co-authored-by: Timothy Hunter <tim.hunter@ecmwf.int>
Jubeku
added a commit
that referenced
this pull request
Mar 12, 2026
* Improve support for latent losses (#1963) * Revert 2D rope to false by default (#1967) Set to True by accident * Implementation of DataReaderMesh (#1840) * First implementation of DataReaderMesh * Move to datareaders extra * ruff * ruff2 * Undo ruff * undo auto-linting * correct typo in eval config (#1971) * Added all-physical-streams option and x/y axis limits (#1972) * Added all-physical-streams option and x/y axis limits * Fix * Changed flag for all streams * Removed old code * moved metric parsing to eval_from_config (#1977) Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> * Fixed integration test (#1980) * [1974][model] Add fallback to config loading (#1985) * Add fallback to config loading * Adjust error message to be not misleading * Homegenize naming convention * Introduce bias/diff maps and animations (#1912) * Introduce bias/diff maps and animations * minor correction * Changes based on review * Introduce "plot_bias" in evaluation configuration (#1986) * Fixed index ordering to not have shuffled output (#1982) * Fixed idxs_inv to revert data point shuffeling * Fixed output handling * Handling of empty data case, addressing reviewer comment * [1893][eval] csvreader cleanup (#1906) * refactor csvreader * check if dataarray size is 0 * fix and use original logic for empty data * linting fixes * revert assertions back * [1890][eval] Move MergeReader to own module (#1892) * move mergereader * use assertions only * implement scoring for the sub-steps within the forecast window (#1896) * work in progress * working for forecast_step * working version * restore no valid times option * lint * Rename scale_z_channels to _scale_z_channels * fix 1 sample bug * Remove points_per_sample from ReaderOutput Remove points_per_sample from ReaderOutput return. * remove n_point_per_sample * fix lead time coord in compute_scores * lint * fix integration test * Fix integration test single stream (#1996) * fix test single * change yml extension and minor fixes --------- Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> * [1907][eval] clean up wegen_reader.py (#1911) * clean up wegen_reader.py * remove exception * consistent reader naming * add blank line * use assertions only * make names consistent * Merge branch 'develop' into 1907-wegenreader-cleanup * revert is_regular --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1888][eval] Refactor Reader class (#1889) * refactor Reader * use assertion only * fix npp atms --------- Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> * [1975][model] Load model path from private repo instead of json (#1998) * Load model path from private repo instead of json * Lint * Script to compute spatial autocorrelation of structured/unstructured datasets (#1955) * standalone script to compute spatial autocorrelation of variables in a structured or unstructured dataset * remove commits that should be in pr 1951 * lint * addressed comments * removed last failure returning 500km default, and moved to packages science * updated a note * rename autocorrelation script * update example usage * Correct EMA halflife_steps calculation with rampup_ratio (#2001) Corrected rampup calculation: https://github.com/NVlabs/edm2/blob/4bf8162f601bcc09472ce8a32dd0cbe8889dc8fc/training/phema.py#L145 Co-authored-by: Wael <wael.almikaeel.95@gmail.com> * Reduce verbosity of output during inference and evaluation (#2006) * Fix incorrect length in validation progress bar * Removing too verbose output * [1766][1743][1332] lint and unit-test fix (#1802) * [1766][1742] fix lint and unit-test * [1766] fix linter * [1766] lint local and global consistent * [1332] add script to detect bad functions (getattr) * code quality: lint and bad functions * [1766] disable some checks * [1877] Script to populate PR labels from linked issues (#1878) * script * branch * more dirs * typo * enable * Fixed bug in linear embedding (#2012) * Adding forecast_steps feature to plot_train (#2010) * Adding forecast_steps feature to plot_train * Renamed arguement to conform to hyphen convention * Added forecast step to filename --------- Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> * add noise distribution plotting * plot noise distribution and decoded noised tokens * fix noise level in validation to p_mean * rm noise and token distribution plotting --------- Co-authored-by: Christian Lessig <christian.lessig@ecmwf.int> Co-authored-by: Seb Hickman <56727418+shmh40@users.noreply.github.com> Co-authored-by: Kacper Nowak <kacper.nowak@awi.de> Co-authored-by: Till Hauer <till@web-hauer.de> Co-authored-by: s6sebusc <49226935+s6sebusc@users.noreply.github.com> Co-authored-by: buschow1 <buschow1@jwlogin04.juwels> Co-authored-by: Matthias Karlbauer <matthias.karlbauer@ecmwf.int> Co-authored-by: Savvas Melidonis <79579567+SavvasMel@users.noreply.github.com> Co-authored-by: Michael Tarnawa <18899420+mtar@users.noreply.github.com> Co-authored-by: iluise <72020169+iluise@users.noreply.github.com> Co-authored-by: pierluigicosi <91318382+pierluigicosi@users.noreply.github.com> Co-authored-by: cosi1 <cosi1@jwlogin21.juwels> Co-authored-by: cosi1 <cosi1@jwb0001.juwels> Co-authored-by: Ilaria Luise <luise.ilaria@gmail.com> Co-authored-by: Wael <wael.almikaeel.95@gmail.com> Co-authored-by: Simone Norberti <63310821+simone99n@users.noreply.github.com> Co-authored-by: Timothy Hunter <tim.hunter@ecmwf.int>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Reduce verbosity of output during inference and evaluation to make it more reasonable.
Issue Number
Closes #2004
Checklist before asking for review
./scripts/actions.sh lint./scripts/actions.sh unit-test./scripts/actions.sh integration-testlaunch-slurm.py --time 60