CDAT Migration: Refactor `polar` set #749

forsyth2 · 2023-10-25T20:43:29Z

Description

Refactor the polar set. Following the directions at https://github.com/E3SM-Project/e3sm_diags/wiki/CDAT-Migration-FY24-%E2%80%90-General-Guide#getting-started:

git fetch upstream
git checkout cdat-migration-fy24
git checkout -b refactor/659-polar
git branch -m refactor/659-polar-set # Include "set" in name
mamba env create -f conda-env/dev.yml -n e3sm_diags_dev_659
mamba activate e3sm_diags_dev_659 # Says to use conda
conda activate e3sm_diags_dev_659
python -m pip install .

Closes CDAT Migration Phase 2: Refactor polar set #659

Summary of Changes:

polar_driver.py and polar_plot.py

Refactor to use Xarray/xCDAT based modules

driver/utils/regrid.py

Update subset_and_align_datasets() to make copies of dataset objects before performing computations for cleaner code and ease of debugging
Fix _subset_on_region() not subsetting specific regions correctly due to order of subsetting tuple. This function makes sure that the axes/axes being subsetted on is in ascending order. If it is not in ascending order, some slice specifications might have two negative values (e.g., (-90, -55) for 'polar_S') which can result in the subset incorrectly excluding of values that seem like they don't fit within the slice slice spec range when they actually are (circular coordinates).

plot/utils.py

Add optional rect arg to _add_colorbar() for adjustment of the rect used for creating colorbar, along with _get_rect(), which can vary based on the set
Add optional left_text_pos and right_text_pos args to _add_min_mean_max_test() and add_rmse_and_corr_text() for adjustment of the x and y position of each respective text, which can vary based on the set

cosp_histogram_py

Update variable and function names for consistency with other refactored plotters.
Replace block of code for writing out plots with _save_plot()

zonal_mean_2d_plot.py

Remove extraneous "#" character

auxiliary_tools/cdat_regression_testing/template_run_script.py

Update incorrect chown command with chmod.

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

If applicable:

New and existing unit tests pass with my changes (locally and CI/CD build)
I have added tests that prove my fix is effective or that my feature works
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

forsyth2 · 2023-11-03T20:56:53Z

Using this script:

diags_set=polar

cd e3sm_diags
# https://anaconda.org/cdat/repo?sort=_name&sort_order=asc
for cdat_package in cd77 cdat cdat_compute_graph cdat_info cdat_jupyter_vcdat cdms2 cdp cdtime cdutil cibots compute_graph django-allow-cidr dv3d e3sm_nex esgf-compute-api esgf-compute-cert esgf-search ezget genutil image-compare jupyter-vcdat jupyterlab-git lats libcdms lmoments mesalib myproxyclient output_viewer selenium_testlib testsrunner thermo toolchain vcdat vcs vcs-js vcsaddons vtk vtk-cdat wk xmgrace
do
    git grep -n ${cdat_package} e3sm_diags/driver/${diags_set}_driver.py > /dev/null
    if [ $? == 0 ]; then
	echo ""
	echo "git grep -n ${cdat_package} e3sm_diags/driver/${diags_set}_driver.py"
	git grep -n ${cdat_package} e3sm_diags/driver/${diags_set}_driver.py
    fi
    git grep -n ${cdat_package} e3sm_diags/plot/cartopy/${diags_set}_plot.py > /dev/null
    if [ $? == 0 ]; then
	echo ""
	echo "git grep -n ${cdat_package} e3sm_diags/plot/cartopy/${diags_set}_plot.py"
	git grep -n ${cdat_package} e3sm_diags/plot/cartopy/${diags_set}_plot.py
    fi
done

I get:

git grep -n cdms2 e3sm_diags/driver/polar_driver.py
e3sm_diags/driver/polar_driver.py:6:import cdms2
e3sm_diags/driver/polar_driver.py:74:            with cdms2.open(mask_path) as f:
e3sm_diags/driver/polar_driver.py:94:                # This is cdms2 fix for bad mask, Denis' fix should fix this.
e3sm_diags/driver/polar_driver.py:98:                # This is cdms2 fix for bad mask, Denis' fix should fix this.
e3sm_diags/driver/polar_driver.py:101:                # This is cdms2 fix for bad mask, Denis' fix should fix this.

forsyth2 · 2023-11-15T20:31:42Z

@chengzhuzhang @tomvothecoder Are the testing directions at https://e3sm-project.github.io/e3sm_diags/_build/html/main/dev_guide/testing.html up to date? I tried running ./tests/test.sh on the main branch and got:

================================================================================================== short test summary info ==================================================================================================
FAILED tests/integration/test_diags.py::TestAllSets::test_lat_lon - AssertionError: Lists differ: ['ERA-Interim-T-850-ANN-global.png'] != []
FAILED tests/integration/test_diags.py::TestAllSets::test_lat_lon_regional - AssertionError: Lists differ: ['ERA-Interim-T-850-ANN-CONUS_RRM.png'] != []
FAILED tests/integration/test_diags.py::TestAllSets::test_polar - AssertionError: Lists differ: ['ERA-Interim-T-850-ANN-polar_S.png'] != []
FAILED tests/integration/test_diags.py::TestAllSets::test_zonal_mean_2d - AssertionError: Lists differ: ['ERA-Interim-T-ANN-global.png'] != []
=================================================================================== 4 failed, 23 passed, 22 warnings in 351.33s (0:05:51) ===================================================================================

tomvothecoder · 2023-11-27T17:37:14Z

@chengzhuzhang @tomvothecoder Are the testing directions at e3sm-project.github.io/e3sm_diags/_build/html/main/dev_guide/testing.html up to date? I tried running ./tests/test.sh on the main branch and got:

================================================================================================== short test summary info ==================================================================================================
FAILED tests/integration/test_diags.py::TestAllSets::test_lat_lon - AssertionError: Lists differ: ['ERA-Interim-T-850-ANN-global.png'] != []
FAILED tests/integration/test_diags.py::TestAllSets::test_lat_lon_regional - AssertionError: Lists differ: ['ERA-Interim-T-850-ANN-CONUS_RRM.png'] != []
FAILED tests/integration/test_diags.py::TestAllSets::test_polar - AssertionError: Lists differ: ['ERA-Interim-T-850-ANN-polar_S.png'] != []
FAILED tests/integration/test_diags.py::TestAllSets::test_zonal_mean_2d - AssertionError: Lists differ: ['ERA-Interim-T-ANN-global.png'] != []
=================================================================================== 4 failed, 23 passed, 22 warnings in 351.33s (0:05:51) ===================================================================================

I don't recall any changes to how tests are run on the main or the cdat-migration-fy24 branches. I'm not sure why these tests are failing on main.

tomvothecoder · 2023-12-04T20:20:08Z

I recently squashed the cdat-migration-fy24 branch to a few commits and just rebased this branch on top of it.

Can you make sure to checkout the latest version of his branch? I also recommend stashing and reapplying any changes you might have.

forsyth2 · 2023-12-18T18:41:39Z

Note to self: see forsyth2#4 (#752) for an example.

forsyth2 · 2023-12-18T19:43:30Z

@tomvothecoder What am I supposed to rebase to remove that initial commit for #677 (d1aec82)? If I run git rebase upstream/cdat-migration-fy24, I get merge conflicts, specifically on e3sm_diags/run.py. But that doesn't make sense because the commit history for that branch (https://github.com/E3SM-Project/e3sm_diags/commits/cdat-migration-fy24/) does in fact include that pull request's merge, and my later commits don't touch that file.

forsyth2 · 2023-12-18T19:57:24Z

Re: testing this branch: I basically took the tests from #752 and updated them to be more like the lat_lon tests in https://github.com/E3SM-Project/e3sm_diags/blob/main/examples/ex5-model-vs-obs/diags.cfg, since polar is pretty similar in purpose to lat_lon. I then ran them using the directions at https://e3sm-project.github.io/e3sm_diags/_build/html/master/examples.html#running-the-diagnostics. However, I'm getting the following errors.

$ python run_polar.py -d diags.cfg
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/e3sm_diags/examples/run_polar.py", line 42, in <module>
    runner.run_diags([param])
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/run.py", line 28, in run_diags
    final_params = self.get_final_parameters(parameters)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/run.py", line 65, in get_final_parameters
    other_params = self._get_other_diags(parameters[0].run_type)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/run.py", line 249, in _get_other_diags
    params = self.parser.get_cfg_parameters(argparse_vals_only=False)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/parser/core_parser.py", line 798, in get_cfg_parameters
    params = self._get_cfg_parameters(
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/parser/core_parser.py", line 817, in _get_cfg_parameters
    cfg_file_obj = self._create_cfg_hash_titles(cfg_file)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/parser/core_parser.py", line 851, in _create_cfg_hash_titles
    with open(cfg_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'examples/test_refactor/diags.cfg'
[WARNING] yaksa: 10 leaked handle pool objects

and

$ python run_polar.py -d diags.cfg --multiprocessing --num_workers=32
Traceback (most recent call last):
  File "/gpfs/fs1/home/ac.forsyth2/e3sm_diags/examples/run_polar.py", line 42, in <module>
    runner.run_diags([param])
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/run.py", line 28, in run_diags
    final_params = self.get_final_parameters(parameters)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/run.py", line 65, in get_final_parameters
    other_params = self._get_other_diags(parameters[0].run_type)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/run.py", line 249, in _get_other_diags
    params = self.parser.get_cfg_parameters(argparse_vals_only=False)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/parser/core_parser.py", line 798, in get_cfg_parameters
    params = self._get_cfg_parameters(
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/parser/core_parser.py", line 817, in _get_cfg_parameters
    cfg_file_obj = self._create_cfg_hash_titles(cfg_file)
  File "/home/ac.forsyth2/miniconda3/envs/e3sm_diags_dev_659/lib/python3.10/site-packages/e3sm_diags/parser/core_parser.py", line 851, in _create_cfg_hash_titles
    with open(cfg_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'examples/test_refactor/diags.cfg'
[WARNING] yaksa: 10 leaked handle pool objects

forsyth2 · 2023-12-18T20:06:34Z

Oh I just realized the directory information is wrong. Rerunning

forsyth2 · 2023-12-18T22:53:11Z

Using the allocation command from #767. A lot of UnboundLocalError: local variable 'proj' referenced before assignment. The viewers is generated but none of the table entries link to plots.

$ python run_polar.py -d diags.cfg
2023-12-18 14:42:02,046 [INFO]: e3sm_diags_driver.py(_save_env_yml:57) >> Saved environment yml file to: /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/prov/environment.yml
2023-12-18 14:42:02,054 [INFO]: e3sm_diags_driver.py(_save_parameter_files:68) >> Saved command used to: /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/prov/cmd_used.txt
2023-12-18 14:42:02,071 [INFO]: e3sm_diags_driver.py(_save_parameter_files:98) >> Saved cfg file to: /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/prov/diags.cfg
2023-12-18 14:42:02,081 [INFO]: e3sm_diags_driver.py(_save_python_script:132) >> Saved Python script to: /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/prov/run_polar.py
2023-12-18 14:42:02,741 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:03,008 [INFO]: polar_driver.py(run_diag:205) >> Selected region: land

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:11,760 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:12,214 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:12,312 [INFO]: polar_driver.py(run_diag:205) >> Selected region: global

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:15,538 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:16,186 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:16,450 [INFO]: polar_driver.py(run_diag:205) >> Selected region: land

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:24,646 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:24,944 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:25,037 [INFO]: polar_driver.py(run_diag:205) >> Selected region: global

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:28,250 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:29,014 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:29,282 [INFO]: polar_driver.py(run_diag:205) >> Selected region: land

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:37,520 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:37,869 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:37,987 [INFO]: polar_driver.py(run_diag:205) >> Selected region: global

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:41,183 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:41,757 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:42,006 [INFO]: polar_driver.py(run_diag:205) >> Selected region: land

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:50,180 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:50,520 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:50,604 [INFO]: polar_driver.py(run_diag:205) >> Selected region: global

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:42:53,774 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:42:54,354 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:42:54,598 [INFO]: polar_driver.py(run_diag:205) >> Selected region: land

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:43:02,808 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:43:03,146 [INFO]: polar_driver.py(run_diag:79) >> Variable: TREFHT
2023-12-18 14:43:03,262 [INFO]: polar_driver.py(run_diag:205) >> Selected region: global

WARNING: Edge bounds are the same. The results of conservative regridding will not conserve.
coordMin =  -90.00, boundMin =  -90.00, coordMax =   90.00, boundMax =   90.00
              
2023-12-18 14:43:06,475 [ERROR]: __init__.py(plot:58) >> Error while plotting polar with backend cartopy
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
Traceback (most recent call last):
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/__init__.py", line 56, in plot
    plot_fcn(ref, test, diff, metrics_dict, parameter)
  File "/global/homes/f/forsyth/miniconda3/envs/e3sm_diags_659/lib/python3.10/site-packages/e3sm_diags/plot/cartopy/polar_plot.py", line 179, in plot
    proj,
UnboundLocalError: local variable 'proj' referenced before assignment
2023-12-18 14:43:06,642 [INFO]: main.py(create_viewer:130) >> polar /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/viewer
2023-12-18 14:43:06,989 [INFO]: main.py(create_viewer:133) >> ('Polar contour maps', 'polar/index.html')
2023-12-18 14:43:07,003 [INFO]: e3sm_diags_driver.py(main:390) >> Viewer HTML generated at /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/viewer/index.html
2023-12-18 14:43:07,016 [INFO]: logger.py(move_log_to_prov_dir:106) >> Log file saved in /global/cfs/cdirs/e3sm/www/forsyth/test_e3sm_refactor/ex5_model_to_obs/prov/e3sm_diags_run.log
[WARNING] yaksa: 10 leaked handle pool objects

forsyth2 · 2023-12-18T23:18:38Z

I tried changing regions to be regions = ["N", "S"] but that just causes KeyErrors

forsyth2 · 2023-12-27T23:53:59Z

Ran into this permissions issue while testing on Perlmutter: #772

forsyth2 · 2023-12-28T00:20:49Z

@tomvothecoder Based on #749 (comment), I think I have actually removed all the CDAT dependencies from polar. I was able to run the example/test on Perlmutter: https://portal.nersc.gov/cfs/e3sm/forsyth/test_e3sm_refactor/test_659/viewer/polar/index.html

Re: these lines:

e3sm_diags/driver/polar_driver.py:94:                # This is cdms2 fix for bad mask, Denis' fix should fix this.
e3sm_diags/driver/polar_driver.py:98:                # This is cdms2 fix for bad mask, Denis' fix should fix this.
e3sm_diags/driver/polar_driver.py:101:                # This is cdms2 fix for bad mask, Denis' fix should fix this.

These refer to MV2, which I seem to recall being CDAT-related, but is not listed at https://anaconda.org/cdat/repo?sort=_name&sort_order=asc.

forsyth2 · 2024-02-07T20:04:26Z

The "How to Use" section includes instructions on setting up a conda environment with the appropriate Python kernel. You can skip the third step for activating the env, and instead select the kernel in Jupyter.

@tomvothecoder Yes, I followed those steps but when I clicked the "Execute Cell" play-button, it prompted "Type to choose a kernel source", with no kernels suggested.

It had a dropdown suggestion of installing a Jupyter extension. I did that and tried again, clicking on cdat_regression_test in the dropdown, as in #749 (comment). (So kernel source is the same as the conda environment then??)

However, that first cell now gives:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[4], [line 5](vscode-notebook-cell:?execution_count=4&line=5)
      [2](vscode-notebook-cell:?execution_count=4&line=2) import glob
      [4](vscode-notebook-cell:?execution_count=4&line=4) import numpy as np
----> [5](vscode-notebook-cell:?execution_count=4&line=5) import xarray as xr
      [7](vscode-notebook-cell:?execution_count=4&line=7) SET_NAME = "polar"
      [8](vscode-notebook-cell:?execution_count=4&line=8) SET_DIR = "659-polar"

ModuleNotFoundError: No module named 'xarray'

tomvothecoder · 2024-02-07T23:12:59Z

The "How to Use" section includes instructions on setting up a conda environment with the appropriate Python kernel. You can skip the third step for activating the env, and instead select the kernel in Jupyter.

@tomvothecoder Yes, I followed those steps but when I clicked the "Execute Cell" play-button, it prompted "Type to choose a kernel source", with no kernels suggested.

It had a dropdown suggestion of installing a Jupyter extension. I did that and tried again, clicking on cdat_regression_test in the dropdown, as in #749 (comment). (So kernel source is the same as the conda environment then??)

However, that first cell now gives:
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[4], [line 5](vscode-notebook-cell:?execution_count=4&line=5)
      [2](vscode-notebook-cell:?execution_count=4&line=2) import glob
      [4](vscode-notebook-cell:?execution_count=4&line=4) import numpy as np
----> [5](vscode-notebook-cell:?execution_count=4&line=5) import xarray as xr
      [7](vscode-notebook-cell:?execution_count=4&line=7) SET_NAME = "polar"
      [8](vscode-notebook-cell:?execution_count=4&line=8) SET_DIR = "659-polar"

ModuleNotFoundError: No module named 'xarray'

When you see ModuleNotFoundError in Python that means the module is not found in the current environment. If you trace back to the install command you ran, xarray is not included. The reason being is that you're using the env created with template_cdat_regression_test_json.ipynb with template_cdat_regression_test_nc.ipynb (which requires xarray)

Each notebook has a different environment in the instructions. It can probably be simplified to be a single environment, but I didn't write them with that in mind.

- Fix comments in `lat_lon_plot.py` and `zonal_mean_2d_plot.py`

tomvothecoder · 2024-03-14T19:14:44Z

e3sm_diags/driver/utils/regrid.py

+    This function makes sure that the axes/axes being subsetted on is in
+    ascending order. For example, if the latitude axis is in descending order,
+    [90, 0, -90], no matches will be made on the subset if the region has a lat
+    slice spec of (-90, -55) (e.g., 'polar_S'). This is because Xarray subsets
+    in ascending order. Sorting the axis beforehand will avoid this issue.
+


@chengzhuzhang Just an FYI about this fix with regional subsetting with Xarray using the region specs dictionary.

A few ref variables were not being subsetted correctly because the lat spec for polar_S is (-90, -55) but the lat coords were in descending order ([90, 0,... -90]).

e3sm_diags/plot/utils.py

e3sm_diags/driver/polar_driver.py

tomvothecoder · 2024-03-14T19:27:27Z

I completed refactoring with successful regression test. All variables are within the relative diff tolerance of 1e-5. I will be merging this PR once the CI/CD build passes.

Polar viewers:

Co-authored-by: Tom Vo <tomvothecoder@gmail.com>

forsyth2 added the cdat-migration-fy24 CDAT Migration FY24 Task label Oct 25, 2023

forsyth2 self-assigned this Oct 25, 2023

forsyth2 linked an issue Oct 26, 2023 that may be closed by this pull request

CDAT Migration Phase 2: Refactor polar set #659

Closed

tomvothecoder added this to the FY24 Q1 (10/1/23 to 12/31/23) milestone Nov 27, 2023

tomvothecoder force-pushed the cdat-migration-fy24 branch 3 times, most recently from 0dfc01d to d1aec82 Compare November 29, 2023 00:58

tomvothecoder mentioned this pull request Dec 1, 2023

[Bug]: tests/integration/test_diags.py is failing on local main branch #758

Closed

tomvothecoder force-pushed the refactor/659-polar-set branch from 7833bda to 8cde51d Compare December 4, 2023 20:19

tomvothecoder force-pushed the cdat-migration-fy24 branch from 340d3b3 to 56c7ffe Compare December 13, 2023 21:25

forsyth2 mentioned this pull request Dec 18, 2023

Diff of polar set refactoring forsyth2/e3sm_diags#3

Closed

forsyth2 force-pushed the refactor/659-polar-set branch from 75b3ef3 to f9b19de Compare December 18, 2023 20:44

tomvothecoder force-pushed the refactor/659-polar-set branch 2 times, most recently from 3a85f49 to b4cc7de Compare December 18, 2023 23:13

forsyth2 mentioned this pull request Dec 19, 2023

Diff after polar rebasing forsyth2/e3sm_diags#5

Closed

forsyth2 marked this pull request as ready for review December 28, 2023 00:21

forsyth2 requested a review from tomvothecoder December 28, 2023 00:21

tomvothecoder force-pushed the cdat-migration-fy24 branch from 56c7ffe to 59781c8 Compare January 3, 2024 21:28

tomvothecoder mentioned this pull request Feb 8, 2024

CDAT Migration Phase 2: Refactor zonal_mean_2d() and zonal_mean_2d_stratosphere() sets #774

Merged

9 tasks

tomvothecoder added 2 commits March 13, 2024 08:49

Refactor polar set

4f0082c

Remove duplicate notebook

dcbd32f

tomvothecoder force-pushed the refactor/659-polar-set branch from 6cdaaf0 to dcbd32f Compare March 13, 2024 15:51

tomvothecoder added 5 commits March 13, 2024 09:09

Add refactored driver

8f7c5f4

Add initial refactored plot

f64ac62

Update function calls in cosp_histogram_plot.py

e26d931

- Fix comments in `lat_lon_plot.py` and `zonal_mean_2d_plot.py`

Remove accidental character

3a5bf95

Update polar plot and add testing results

e4dee33

tomvothecoder changed the title ~~Refactor polar set~~ CDAT Migration: Refactor polar set Mar 14, 2024

tomvothecoder changed the title ~~CDAT Migration: Refactor polar set~~ CDAT Migration: Refactor polar set Mar 14, 2024

tomvothecoder added 2 commits March 14, 2024 12:10

Fix mypy type annotation

96f92a0

Move polar_plot.py to root of plot dir

47500ec

tomvothecoder reviewed Mar 14, 2024

View reviewed changes

e3sm_diags/plot/utils.py Outdated Show resolved Hide resolved

Update e3sm_diags/plot/utils.py

1e34743

tomvothecoder reviewed Mar 14, 2024

View reviewed changes

e3sm_diags/driver/polar_driver.py Outdated Show resolved Hide resolved

tomvothecoder added 2 commits March 14, 2024 12:19

Update e3sm_diags/driver/polar_driver.py

0e74db8

Remove test_polar.py

06061f8

tomvothecoder self-assigned this Mar 14, 2024

tomvothecoder merged commit 0247f05 into E3SM-Project:cdat-migration-fy24 Mar 14, 2024
4 checks passed

tomvothecoder mentioned this pull request Mar 14, 2024

CDAT Migration Phase 2: Refactor polar set #659

Closed

forsyth2 mentioned this pull request Mar 29, 2024

Refactor area_mean_time_series #750

Merged

21 tasks

tomvothecoder added a commit that referenced this pull request May 1, 2024

CDAT Migration: Refactor polar set (#749)

dd98b6c

Co-authored-by: Tom Vo <tomvothecoder@gmail.com>

tomvothecoder added a commit that referenced this pull request Jul 15, 2024

CDAT Migration: Refactor polar set (#749)

de170a4

Co-authored-by: Tom Vo <tomvothecoder@gmail.com>

tomvothecoder added a commit that referenced this pull request Jul 15, 2024

CDAT Migration: Refactor polar set (#749)

3ac8b92

Co-authored-by: Tom Vo <tomvothecoder@gmail.com>

tomvothecoder added a commit that referenced this pull request Aug 21, 2024

CDAT Migration: Refactor polar set (#749)

83a4a9e

Co-authored-by: Tom Vo <tomvothecoder@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CDAT Migration: Refactor `polar` set #749

CDAT Migration: Refactor `polar` set #749

forsyth2 commented Oct 25, 2023 •

edited by tomvothecoder

Loading

forsyth2 commented Nov 3, 2023

forsyth2 commented Nov 15, 2023

tomvothecoder commented Nov 27, 2023

tomvothecoder commented Dec 4, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 27, 2023

forsyth2 commented Dec 28, 2023

forsyth2 commented Feb 7, 2024 •

edited

Loading

tomvothecoder commented Feb 7, 2024 •

edited

Loading

tomvothecoder Mar 14, 2024

tomvothecoder Mar 14, 2024

tomvothecoder commented Mar 14, 2024 •

edited

Loading

CDAT Migration: Refactor polar set #749

CDAT Migration: Refactor polar set #749

Conversation

forsyth2 commented Oct 25, 2023 • edited by tomvothecoder Loading

Description

Checklist

forsyth2 commented Nov 3, 2023

forsyth2 commented Nov 15, 2023

tomvothecoder commented Nov 27, 2023

tomvothecoder commented Dec 4, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 18, 2023

forsyth2 commented Dec 27, 2023

forsyth2 commented Dec 28, 2023

forsyth2 commented Feb 7, 2024 • edited Loading

tomvothecoder commented Feb 7, 2024 • edited Loading

tomvothecoder Mar 14, 2024

Choose a reason for hiding this comment

tomvothecoder Mar 14, 2024

Choose a reason for hiding this comment

tomvothecoder commented Mar 14, 2024 • edited Loading

CDAT Migration: Refactor `polar` set #749

CDAT Migration: Refactor `polar` set #749

forsyth2 commented Oct 25, 2023 •

edited by tomvothecoder

Loading

forsyth2 commented Feb 7, 2024 •

edited

Loading

tomvothecoder commented Feb 7, 2024 •

edited

Loading

tomvothecoder commented Mar 14, 2024 •

edited

Loading