-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG/ISSUE] Benchmark plots omit comparisons unless species/emissions are found in both Ref and Dev #19
Comments
Let's set the values to NaN so it doesn't appear that the version outputs the variables with all zeros. The diff and frac diff plots can be grayed out to be clear that the comparison is not applicable. |
I have pushed commit d872148, which will now print emissions totals for all variables that are present in at least one of the Ref and Dev datasets. Missing values will now show up as "nan" in the emissions totals. We will also make similar edits to the plotting routines so that missing values will be shown with NaNs. That will take a bit longer. |
I have pushed commit 7df6df1 to master. This will now plot variables that are in Ref but not Dev (or vice-versa) as all grey in surface or zonal mean plots. This commit should resolve this issue (but I'll leave the issue open in case further tweaks are needed). |
1 similar comment
I have pushed commit 7df6df1 to master. This will now plot variables that are in Ref but not Dev (or vice-versa) as all grey in surface or zonal mean plots. This commit should resolve this issue (but I'll leave the issue open in case further tweaks are needed). |
I have pushed commit 6900abe to master. This now fixes some issues in the benchmark plots caused incorrect tests for arrays that are all zero and arrays that are all NaN. We now use more robust tests. This now produces plots that look like this for GC-Classic: (1) Emissions plot for CH4 anthro, which is zero in both Ref and Dev: (2) Concentration plot (single level) for a species in "Ref" that is not present in "Dev": (3) Zonal mean plot for a species in Ref that is not in Dev: |
I also discovered that some plots that have a missing species in one dataset can cause the color range for the non-missing species to be incorrectly plotted: In this plot, EmisDST1_Anthro is missing in Ref, but present in Dev. The Ref panel, as well as all of the absolute difference panels should be plotted as all gray, but are not: Upon further review, we had to rewrite the logic in how the min and max values for the data ranges (i.e. vmin0, vmax0, vmin1, vmax1, vmin, vmax) were computed in order to better account for NaN values. This was added to commit 8d8282c This commit now fixes the issue: |
Added 90fed1a to fix an typo that caused single-level plots with a missing species in Ref to be plotted incorrectly when using the RdBu color scale. This commit now produces the correct behavior: |
When creating the benchmark plots, species and/or emissions are only plotted if they found in both Ref and Dev datasets.
For example if a species is present in Ref, but removed from Dev, then the benchmark code will totally skip that comparison. This has caused some confusion among GCSC members who are examining the benchmark results.
What we should do is to set the missing species to zero and then force the comparison.
The text was updated successfully, but these errors were encountered: