Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG/ISSUE] Benchmark plots omit comparisons unless species/emissions are found in both Ref and Dev #19

Closed
yantosca opened this issue Aug 22, 2019 · 7 comments
Assignees
Labels
category: Bug Something isn't working

Comments

@yantosca
Copy link
Contributor

When creating the benchmark plots, species and/or emissions are only plotted if they found in both Ref and Dev datasets.

For example if a species is present in Ref, but removed from Dev, then the benchmark code will totally skip that comparison. This has caused some confusion among GCSC members who are examining the benchmark results.

What we should do is to set the missing species to zero and then force the comparison.

@yantosca yantosca self-assigned this Aug 22, 2019
@lizziel
Copy link
Contributor

lizziel commented Aug 22, 2019

Let's set the values to NaN so it doesn't appear that the version outputs the variables with all zeros. The diff and frac diff plots can be grayed out to be clear that the comparison is not applicable.

@yantosca
Copy link
Contributor Author

I have pushed commit d872148, which will now print emissions totals for all variables that are present in at least one of the Ref and Dev datasets. Missing values will now show up as "nan" in the emissions totals.

We will also make similar edits to the plotting routines so that missing values will be shown with NaNs. That will take a bit longer.

@yantosca
Copy link
Contributor Author

I have pushed commit 7df6df1 to master. This will now plot variables that are in Ref but not Dev (or vice-versa) as all grey in surface or zonal mean plots. This commit should resolve this issue (but I'll leave the issue open in case further tweaks are needed).

1 similar comment
@yantosca
Copy link
Contributor Author

I have pushed commit 7df6df1 to master. This will now plot variables that are in Ref but not Dev (or vice-versa) as all grey in surface or zonal mean plots. This commit should resolve this issue (but I'll leave the issue open in case further tweaks are needed).

@yantosca
Copy link
Contributor Author

I have pushed commit 6900abe to master. This now fixes some issues in the benchmark plots caused incorrect tests for arrays that are all zero and arrays that are all NaN. We now use more robust tests.

This now produces plots that look like this for GC-Classic:

(1) Emissions plot for CH4 anthro, which is zero in both Ref and Dev:

gcpy_fix_sfc

(2) Concentration plot (single level) for a species in "Ref" that is not present in "Dev":

gcpy_fix_1

(3) Zonal mean plot for a species in Ref that is not in Dev:

gcpy_fix_zm

@yantosca
Copy link
Contributor Author

I also discovered that some plots that have a missing species in one dataset can cause the color range for the non-missing species to be incorrectly plotted:

In this plot, EmisDST1_Anthro is missing in Ref, but present in Dev. The Ref panel, as well as all of the absolute difference panels should be plotted as all gray, but are not:

nan1

Upon further review, we had to rewrite the logic in how the min and max values for the data ranges (i.e. vmin0, vmax0, vmin1, vmax1, vmin, vmax) were computed in order to better account for NaN values. This was added to commit 8d8282c

This commit now fixes the issue:

nan2

@yantosca
Copy link
Contributor Author

Added 90fed1a to fix an typo that caused single-level plots with a missing species in Ref to be plotted incorrectly when using the RdBu color scale. This commit now produces the correct behavior:

nan3

@yantosca yantosca added category: Bug Something isn't working resolved labels Oct 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants