doc/jupyter/Demo/Demo_1b_mean_climate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mean Climate Driver\n",
    "\n",
    "This notebook introduces new users to the PCDMI mean climate driver.\n",
    "\n",
    "This notebook should be run in an environment with python, jupyterlab, pcmdi metrics package, and cdat installed. It is expected that you have downloaded the sample data as demonstrated in [the download notebook](Demo_0_download_data.ipynb).  \n",
    "\n",
    "The following cell reads in the choices you made during the download data step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from user_choices import demo_data_directory, demo_output_directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PCMDI Median Driver is driven via parameter files reflecting your study and environment. The parameter file for this demo is shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import os\n",
      "\n",
      "#\n",
      "#  OPTIONS ARE SET BY USER IN THIS FILE AS INDICATED BELOW BY:\n",
      "#\n",
      "#\n",
      "\n",
      "# RUN IDENTIFICATION\n",
      "# DEFINES A SUBDIRECTORY TO METRICS OUTPUT RESULTS SO MULTIPLE CASES CAN\n",
      "# BE COMPARED\n",
      "case_id = 'basicTest'\n",
      "\n",
      "# LIST OF MODEL VERSIONS TO BE TESTED - WHICH ARE EXPECTED TO BE PART OF\n",
      "# CLIMATOLOGY FILENAME\n",
      "test_data_set = ['ACCESS1-0', 'CanCM4']\n",
      "\n",
      "\n",
      "# VARIABLES TO USE\n",
      "vars = ['rlut']\n",
      "\n",
      "\n",
      "# Observations to use at the moment \"default\" or \"alternate\"\n",
      "reference_data_set = ['all']\n",
      "#ext = '.nc'\n",
      "\n",
      "# INTERPOLATION OPTIONS\n",
      "target_grid = '2.5x2.5'  # OPTIONS: '2.5x2.5' or an actual cdms2 grid object\n",
      "regrid_tool = 'regrid2'  # 'regrid2' # OPTIONS: 'regrid2','esmf'\n",
      "# OPTIONS: 'linear','conservative', only if tool is esmf\n",
      "regrid_method = 'linear'\n",
      "regrid_tool_ocn = 'esmf'    # OPTIONS: \"regrid2\",\"esmf\"\n",
      "# OPTIONS: 'linear','conservative', only if tool is esmf\n",
      "regrid_method_ocn = 'linear'\n",
      "\n",
      "# Templates for climatology files\n",
      "# %(param) will subsitute param with values in this file\n",
      "filename_template = \"cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc\"\n",
      "\n",
      "# filename template for landsea masks ('sftlf')\n",
      "sftlf_filename_template = \"sftlf_%(model_version).nc\"\n",
      "generate_sftlf = True # if land surface type mask cannot be found, generate one\n",
      "\n",
      "# Region\n",
      "regions = {\"rlut\": [\"Global\"]}\n",
      "\n",
      "# ROOT PATH FOR MODELS CLIMATOLOGIES\n",
      "test_data_path = 'demo_data/CMIP5_demo_clims/'\n",
      "# ROOT PATH FOR OBSERVATIONS\n",
      "# Note that atm/mo/%(variable)/ac will be added to this\n",
      "reference_data_path = 'demo_data/obs4MIPs_PCMDI_clims'\n",
      "\n",
      "# DIRECTORY WHERE TO PUT RESULTS\n",
      "metrics_output_path = os.path.join(\n",
      "    'demo_output',\n",
      "    \"%(case_id)\")\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(\"basic_param.py\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run the mean climate driver, use the following command in the terminal. This will generate a metrics file based on the models, observations, and other criteria in `basic_param.py`\n",
    "```\n",
    "mean_climate_driver.py -p basic_param.py\n",
    "```  \n",
    "In the next cell, bash cell magic is used to run this command as a subprocess:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-09 16:38:41,260 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:38:41,260 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:39::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_basicTest.json\n",
      "2023-11-09 16:39:14,932 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_basicTest.json\n",
      "2023-11-09 16:39:14,932 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_basicTest.json\n",
      "2023-11-09 16:39:14,958 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:39:14,958 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:39::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_basicTest.json\n",
      "2023-11-09 16:39:50,007 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_basicTest.json\n",
      "2023-11-09 16:39:50,007 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_basicTest.json\n",
      "2023-11-09 16:39:55,027 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:39:55,027 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:40::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_basicTest.json\n",
      "2023-11-09 16:40:29,591 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_basicTest.json\n",
      "2023-11-09 16:40:29,591 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_basicTest.json\n",
      "2023-11-09 16:40:29,627 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:40:29,627 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:41::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_basicTest.json\n",
      "2023-11-09 16:41:05,136 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_basicTest.json\n",
      "2023-11-09 16:41:05,136 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_basicTest.json\n",
      "INFO::2023-11-09 16:41::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut_2.5x2.5_regrid2_metrics_basicTest.json\n",
      "2023-11-09 16:41:05,142 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut_2.5x2.5_regrid2_metrics_basicTest.json\n",
      "2023-11-09 16:41:05,142 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/basicTest/rlut_2.5x2.5_regrid2_metrics_basicTest.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "case_id:  basicTest \n",
      " test_data_set: ['ACCESS1-0', 'CanCM4'] \n",
      " realization:  \n",
      " vars: ['rlut'] \n",
      " varname_in_test_data: None \n",
      " reference_data_set: ['all'] \n",
      " target_grid: 2.5x2.5 \n",
      " regrid_tool: regrid2 \n",
      " regrid_tool_ocn: esmf \n",
      " save_test_clims: False \n",
      " test_clims_interpolated_output: None \n",
      " filename_template: cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc \n",
      " sftlf_filename_template: sftlf_%(model_version).nc \n",
      " generate_sftlf: True \n",
      " regions_specs: {'global': {}, 'NHEX': {'domain': {'latitude': (30.0, 90)}}, 'SHEX': {'domain': {'latitude': (-90.0, -30)}}, 'TROPICS': {'domain': {'latitude': (-30.0, 30)}}, '90S50S': {'domain': {'latitude': (-90.0, -50)}}, '50S20S': {'domain': {'latitude': (-50.0, -20)}}, '20S20N': {'domain': {'latitude': (-20.0, 20)}}, '20N50N': {'domain': {'latitude': (20.0, 50)}}, '50N90N': {'domain': {'latitude': (50.0, 90)}}, 'CONUS': {'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'land': {'value': 100}, 'land_NHEX': {'value': 100, 'domain': {'latitude': (30.0, 90)}}, 'land_SHEX': {'value': 100, 'domain': {'latitude': (-90.0, -30)}}, 'land_TROPICS': {'value': 100, 'domain': {'latitude': (-30.0, 30)}}, 'land_CONUS': {'value': 100, 'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'ocean': {'value': 0}, 'ocean_NHEX': {'value': 0, 'domain': {'latitude': (30.0, 90)}}, 'ocean_SHEX': {'value': 0, 'domain': {'latitude': (-90.0, -30)}}, 'ocean_TROPICS': {'value': 0, 'domain': {'latitude': (30.0, 30)}}, 'ocean_50S50N': {'value': 0.0, 'domain': {'latitude': (-50.0, 50)}}, 'ocean_50S20S': {'value': 0.0, 'domain': {'latitude': (-50.0, -20)}}, 'ocean_20S20N': {'value': 0.0, 'domain': {'latitude': (-20.0, 20)}}, 'ocean_20N50N': {'value': 0.0, 'domain': {'latitude': (20.0, 50)}}, 'NAM': {'domain': {'latitude': (20.0, 90), 'longitude': (-180, 180)}}, 'NAO': {'domain': {'latitude': (20.0, 80), 'longitude': (-90, 40)}}, 'SAM': {'domain': {'latitude': (-20.0, -90), 'longitude': (0, 360)}}, 'PNA': {'domain': {'latitude': (20.0, 85), 'longitude': (120, 240)}}, 'PDO': {'domain': {'latitude': (20.0, 70), 'longitude': (110, 260)}}, 'AllMW': {'domain': {'latitude': (-40.0, 45.0), 'longitude': (0.0, 360.0)}}, 'AllM': {'domain': {'latitude': (-45.0, 45.0), 'longitude': (0.0, 360.0)}}, 'NAMM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (210.0, 310.0)}}, 'SAMM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (240.0, 330.0)}}, 'NAFM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (310.0, 60.0)}}, 'SAFM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (0.0, 90.0)}}, 'ASM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (60.0, 180.0)}}, 'AUSM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (90.0, 160.0)}}, 'AIR': {'domain': {'latitude': (7.0, 25.0), 'longitude': (65.0, 85.0)}}, 'AUS': {'domain': {'latitude': (-20.0, -10.0), 'longitude': (120.0, 150.0)}}, 'Sahel': {'domain': {'latitude': (13.0, 18.0), 'longitude': (-10.0, 10.0)}}, 'GoG': {'domain': {'latitude': (0.0, 5.0), 'longitude': (-10.0, 10.0)}}, 'NAmo': {'domain': {'latitude': (20.0, 37.0), 'longitude': (-112.0, -103.0)}}, 'SAmo': {'domain': {'latitude': (-20.0, 2.5), 'longitude': (-65.0, -40.0)}}} \n",
      " regions: {'rlut': ['Global']} \n",
      " test_data_path: demo_data/CMIP5_demo_clims/ \n",
      " reference_data_path: demo_data/obs4MIPs_PCMDI_clims \n",
      " metrics_output_path: demo_output/basicTest \n",
      " diagnostics_output_path: demo_output/basicTest \n",
      " debug: False \n",
      "\n",
      "--- prepare mean climate metrics calculation ---\n",
      "--- start mean climate metrics calculation ---\n",
      "varname: rlut\n",
      "level: None\n",
      "reference_data_set (all):  ['alternate1', 'default']\n",
      "ref: alternate1\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.46854119190699\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.438416968613225\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 1.1384451737888808\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.770012023689812\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  8.044682445768467 7.9637213813349925\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9693780836555635\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.46854119190699\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 35.67665337064905\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -1.1635463134324031\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 6.331106438822097\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  9.610248337339145 9.539550989529207\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9518659641205703\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "ref: default\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-1/v20210804/rlut_mon_CERES-EBAF-4-1_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.41842190556023\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.438416968613225\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 1.1365878854524063\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.763493973471603\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  8.033820554038556 7.953014565127818\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9694728084795681\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.41842190556023\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 35.67665337064905\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -1.1654036017688774\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 6.333692053819536\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  9.608637289548378 9.537701243333409\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9518850115349531\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "pmp mean clim driver completed\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mean_climate_driver.py  -p basic_param.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the mean climate driver produces an output json file in the demo output directory. The metrics are stored in the \"RESULTS\" object of the json, viewed here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"ACCESS1-0\": {\n",
      "    \"alternate1\": {\n",
      "      \"source\": \"CERES-EBAF-4-0\",\n",
      "      \"\": {\n",
      "        \"InputClimatologyFileName\": \"cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\",\n",
      "        \"Global\": {\n",
      "          \"bias_xy\": {\n",
      "            \"ann\": \"1.138\",\n",
      "            \"djf\": \"1.675\",\n",
      "            \"mam\": \"1.392\",\n",
      "            \"jja\": \"0.859\",\n",
      "            \"son\": \"0.629\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"1.709\",\n",
      "              \"1.208\",\n",
      "              \"1.200\",\n",
      "              \"1.359\",\n",
      "              \"1.606\",\n",
      "              \"1.287\",\n",
      "              \"0.820\",\n",
      "              \"0.470\",\n",
      "              \"0.184\",\n",
      "              \"0.398\",\n",
      "              \"1.299\",\n",
      "              \"2.122\"\n",
      "            ]\n",
      "          },\n",
      "          \"cor_xy\": {\n",
      "            \"ann\": \"0.969\",\n",
      "            \"djf\": \"0.96\",\n",
      "            \"mam\": \"0.95\",\n",
      "            \"jja\": \"0.96\",\n",
      "            \"son\": \"0.96\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"0.96\",\n",
      "              \"0.96\",\n",
      "              \"0.95\",\n",
      "              \"0.95\",\n",
      "              \"0.93\",\n",
      "              \"0.94\",\n",
      "              \"0.95\",\n",
      "              \"0.96\",\n",
      "              \"0.96\",\n",
      "              \"0.95\",\n",
      "              \"0.95\",\n",
      "              \"0.95\"\n",
      "            ]\n",
      "          },\n",
      "          \"mae_xy\": {\n",
      "            \"ann\": \"5.770\",\n",
      "            \"djf\": \"7.157\",\n",
      "            \"mam\": \"7.247\",\n",
      "            \"jja\": \"7.517\",\n",
      "            \"son\": \"6.448\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"7.774\",\n",
      "              \"7.641\",\n",
      "              \"8.017\",\n",
      "              \"7.776\",\n",
      "              \"7.820\",\n",
      "              \"7.989\",\n",
      "              \"8.508\",\n",
      "              \"8.006\",\n",
      "              \"7.362\",\n",
      "              \"7.296\",\n",
      "              \"7.280\",\n",
      "              \"7.719\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean-obs_xy\": {\n",
      "            \"ann\": \"240.316\",\n",
      "            \"djf\": \"237.540\",\n",
      "            \"mam\": \"239.327\",\n",
      "            \"jja\": \"243.879\",\n",
      "            \"son\": \"240.536\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"237.511\",\n",
      "              \"237.974\",\n",
      "              \"238.287\",\n",
      "              \"238.996\",\n",
      "              \"240.648\",\n",
      "              \"242.912\",\n",
      "              \"244.361\",\n",
      "              \"244.378\",\n",
      "              \"242.922\",\n",
      "              \"240.568\",\n",
      "              \"238.119\",\n",
      "              \"237.120\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean_xy\": {\n",
      "            \"ann\": \"241.455\",\n",
      "            \"djf\": \"239.215\",\n",
      "            \"mam\": \"240.719\",\n",
      "            \"jja\": \"244.738\",\n",
      "            \"son\": \"241.166\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"239.221\",\n",
      "              \"239.182\",\n",
      "              \"239.487\",\n",
      "              \"240.356\",\n",
      "              \"242.254\",\n",
      "              \"244.199\",\n",
      "              \"245.181\",\n",
      "              \"244.848\",\n",
      "              \"243.106\",\n",
      "              \"240.966\",\n",
      "              \"239.418\",\n",
      "              \"239.242\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_devzm\": {\n",
      "            \"ann\": \"5.808\"\n",
      "          },\n",
      "          \"rms_xy\": {\n",
      "            \"ann\": \"8.045\",\n",
      "            \"djf\": \"10.232\",\n",
      "            \"mam\": \"10.775\",\n",
      "            \"jja\": \"10.443\",\n",
      "            \"son\": \"9.260\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.121\",\n",
      "              \"10.901\",\n",
      "              \"11.602\",\n",
      "              \"11.489\",\n",
      "              \"12.612\",\n",
      "              \"11.981\",\n",
      "              \"11.852\",\n",
      "              \"11.344\",\n",
      "              \"10.657\",\n",
      "              \"10.753\",\n",
      "              \"11.132\",\n",
      "              \"11.892\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_xyt\": {\n",
      "            \"ann\": \"nan\"\n",
      "          },\n",
      "          \"rms_y\": {\n",
      "            \"ann\": \"5.566\"\n",
      "          },\n",
      "          \"rmsc_xy\": {\n",
      "            \"ann\": \"7.964\",\n",
      "            \"djf\": \"10.094\",\n",
      "            \"mam\": \"10.684\",\n",
      "            \"jja\": \"10.407\",\n",
      "            \"son\": \"9.238\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"10.989\",\n",
      "              \"10.834\",\n",
      "              \"11.539\",\n",
      "              \"11.409\",\n",
      "              \"12.509\",\n",
      "              \"11.912\",\n",
      "              \"11.823\",\n",
      "              \"11.334\",\n",
      "              \"10.655\",\n",
      "              \"10.745\",\n",
      "              \"11.056\",\n",
      "              \"11.701\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy\": {\n",
      "            \"ann\": \"29.647\",\n",
      "            \"djf\": \"32.686\",\n",
      "            \"mam\": \"30.818\",\n",
      "            \"jja\": \"35.375\",\n",
      "            \"son\": \"31.289\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"33.135\",\n",
      "              \"33.734\",\n",
      "              \"33.073\",\n",
      "              \"31.262\",\n",
      "              \"31.721\",\n",
      "              \"34.451\",\n",
      "              \"36.365\",\n",
      "              \"36.271\",\n",
      "              \"34.820\",\n",
      "              \"32.053\",\n",
      "              \"30.983\",\n",
      "              \"31.985\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy_devzm\": {\n",
      "            \"ann\": \"12.983\"\n",
      "          },\n",
      "          \"std-obs_xyt\": {\n",
      "            \"ann\": \"33.469\"\n",
      "          },\n",
      "          \"std_xy\": {\n",
      "            \"ann\": \"31.966\",\n",
      "            \"djf\": \"36.191\",\n",
      "            \"mam\": \"34.089\",\n",
      "            \"jja\": \"37.564\",\n",
      "            \"son\": \"33.813\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"36.778\",\n",
      "              \"37.344\",\n",
      "              \"36.641\",\n",
      "              \"34.890\",\n",
      "              \"34.366\",\n",
      "              \"36.411\",\n",
      "              \"38.933\",\n",
      "              \"39.053\",\n",
      "              \"37.396\",\n",
      "              \"34.686\",\n",
      "              \"33.932\",\n",
      "              \"35.562\"\n",
      "            ]\n",
      "          },\n",
      "          \"std_xy_devzm\": {\n",
      "            \"ann\": \"12.475\"\n",
      "          },\n",
      "          \"std_xyt\": {\n",
      "            \"ann\": \"36.438\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    },\n",
      "    \"units\": \"W m-2\",\n",
      "    \"default\": {\n",
      "      \"source\": \"CERES-EBAF-4-1\",\n",
      "      \"\": {\n",
      "        \"InputClimatologyFileName\": \"cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\",\n",
      "        \"Global\": {\n",
      "          \"bias_xy\": {\n",
      "            \"ann\": \"1.137\",\n",
      "            \"djf\": \"1.644\",\n",
      "            \"mam\": \"1.325\",\n",
      "            \"jja\": \"0.866\",\n",
      "            \"son\": \"0.712\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"1.687\",\n",
      "              \"1.207\",\n",
      "              \"1.166\",\n",
      "              \"1.304\",\n",
      "              \"1.496\",\n",
      "              \"1.239\",\n",
      "              \"0.819\",\n",
      "              \"0.540\",\n",
      "              \"0.263\",\n",
      "              \"0.517\",\n",
      "              \"1.349\",\n",
      "              \"2.052\"\n",
      "            ]\n",
      "          },\n",
      "          \"cor_xy\": {\n",
      "            \"ann\": \"0.969\",\n",
      "            \"djf\": \"0.96\",\n",
      "            \"mam\": \"0.95\",\n",
      "            \"jja\": \"0.96\",\n",
      "            \"son\": \"0.96\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"0.96\",\n",
      "              \"0.96\",\n",
      "              \"0.95\",\n",
      "              \"0.95\",\n",
      "              \"0.93\",\n",
      "              \"0.94\",\n",
      "              \"0.95\",\n",
      "              \"0.96\",\n",
      "              \"0.96\",\n",
      "              \"0.95\",\n",
      "              \"0.95\",\n",
      "              \"0.95\"\n",
      "            ]\n",
      "          },\n",
      "          \"mae_xy\": {\n",
      "            \"ann\": \"5.763\",\n",
      "            \"djf\": \"7.164\",\n",
      "            \"mam\": \"7.307\",\n",
      "            \"jja\": \"7.560\",\n",
      "            \"son\": \"6.379\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"7.812\",\n",
      "              \"7.649\",\n",
      "              \"8.087\",\n",
      "              \"7.828\",\n",
      "              \"7.843\",\n",
      "              \"8.028\",\n",
      "              \"8.551\",\n",
      "              \"8.058\",\n",
      "              \"7.319\",\n",
      "              \"7.211\",\n",
      "              \"7.159\",\n",
      "              \"7.554\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean-obs_xy\": {\n",
      "            \"ann\": \"240.318\",\n",
      "            \"djf\": \"237.570\",\n",
      "            \"mam\": \"239.394\",\n",
      "            \"jja\": \"243.872\",\n",
      "            \"son\": \"240.454\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"237.534\",\n",
      "              \"237.974\",\n",
      "              \"238.321\",\n",
      "              \"239.051\",\n",
      "              \"240.758\",\n",
      "              \"242.960\",\n",
      "              \"244.362\",\n",
      "              \"244.308\",\n",
      "              \"242.844\",\n",
      "              \"240.449\",\n",
      "              \"238.069\",\n",
      "              \"237.190\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean_xy\": {\n",
      "            \"ann\": \"241.455\",\n",
      "            \"djf\": \"239.215\",\n",
      "            \"mam\": \"240.719\",\n",
      "            \"jja\": \"244.738\",\n",
      "            \"son\": \"241.166\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"239.221\",\n",
      "              \"239.182\",\n",
      "              \"239.487\",\n",
      "              \"240.356\",\n",
      "              \"242.254\",\n",
      "              \"244.199\",\n",
      "              \"245.181\",\n",
      "              \"244.848\",\n",
      "              \"243.106\",\n",
      "              \"240.966\",\n",
      "              \"239.418\",\n",
      "              \"239.242\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_devzm\": {\n",
      "            \"ann\": \"5.808\"\n",
      "          },\n",
      "          \"rms_xy\": {\n",
      "            \"ann\": \"8.034\",\n",
      "            \"djf\": \"10.240\",\n",
      "            \"mam\": \"10.872\",\n",
      "            \"jja\": \"10.488\",\n",
      "            \"son\": \"9.202\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.139\",\n",
      "              \"10.909\",\n",
      "              \"11.669\",\n",
      "              \"11.580\",\n",
      "              \"12.718\",\n",
      "              \"12.062\",\n",
      "              \"11.913\",\n",
      "              \"11.353\",\n",
      "              \"10.565\",\n",
      "              \"10.660\",\n",
      "              \"11.013\",\n",
      "              \"11.727\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_xyt\": {\n",
      "            \"ann\": \"nan\"\n",
      "          },\n",
      "          \"rms_y\": {\n",
      "            \"ann\": \"5.551\"\n",
      "          },\n",
      "          \"rmsc_xy\": {\n",
      "            \"ann\": \"7.953\",\n",
      "            \"djf\": \"10.107\",\n",
      "            \"mam\": \"10.791\",\n",
      "            \"jja\": \"10.452\",\n",
      "            \"son\": \"9.175\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.010\",\n",
      "              \"10.842\",\n",
      "              \"11.610\",\n",
      "              \"11.506\",\n",
      "              \"12.630\",\n",
      "              \"11.998\",\n",
      "              \"11.885\",\n",
      "              \"11.340\",\n",
      "              \"10.562\",\n",
      "              \"10.647\",\n",
      "              \"10.930\",\n",
      "              \"11.546\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy\": {\n",
      "            \"ann\": \"29.644\",\n",
      "            \"djf\": \"32.736\",\n",
      "            \"mam\": \"30.775\",\n",
      "            \"jja\": \"35.361\",\n",
      "            \"son\": \"31.194\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"33.142\",\n",
      "              \"33.727\",\n",
      "              \"32.912\",\n",
      "              \"31.203\",\n",
      "              \"31.767\",\n",
      "              \"34.483\",\n",
      "              \"36.342\",\n",
      "              \"36.157\",\n",
      "              \"34.665\",\n",
      "              \"31.899\",\n",
      "              \"30.965\",\n",
      "              \"32.014\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy_devzm\": {\n",
      "            \"ann\": \"12.976\"\n",
      "          },\n",
      "          \"std-obs_xyt\": {\n",
      "            \"ann\": \"33.418\"\n",
      "          },\n",
      "          \"std_xy\": {\n",
      "            \"ann\": \"31.966\",\n",
      "            \"djf\": \"36.191\",\n",
      "            \"mam\": \"34.089\",\n",
      "            \"jja\": \"37.564\",\n",
      "            \"son\": \"33.813\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"36.778\",\n",
      "              \"37.344\",\n",
      "              \"36.641\",\n",
      "              \"34.890\",\n",
      "              \"34.366\",\n",
      "              \"36.411\",\n",
      "              \"38.933\",\n",
      "              \"39.053\",\n",
      "              \"37.396\",\n",
      "              \"34.686\",\n",
      "              \"33.932\",\n",
      "              \"35.562\"\n",
      "            ]\n",
      "          },\n",
      "          \"std_xy_devzm\": {\n",
      "            \"ann\": \"12.475\"\n",
      "          },\n",
      "          \"std_xyt\": {\n",
      "            \"ann\": \"36.438\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  },\n",
      "  \"CanCM4\": {\n",
      "    \"alternate1\": {\n",
      "      \"source\": \"CERES-EBAF-4-0\",\n",
      "      \"\": {\n",
      "        \"InputClimatologyFileName\": \"cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\",\n",
      "        \"Global\": {\n",
      "          \"bias_xy\": {\n",
      "            \"ann\": \"-1.164\",\n",
      "            \"djf\": \"-0.905\",\n",
      "            \"mam\": \"-0.959\",\n",
      "            \"jja\": \"-1.403\",\n",
      "            \"son\": \"-1.393\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"-1.057\",\n",
      "              \"-0.731\",\n",
      "              \"-0.594\",\n",
      "              \"-0.959\",\n",
      "              \"-1.302\",\n",
      "              \"-1.321\",\n",
      "              \"-1.516\",\n",
      "              \"-1.374\",\n",
      "              \"-1.579\",\n",
      "              \"-1.463\",\n",
      "              \"-1.138\",\n",
      "              \"-0.927\"\n",
      "            ]\n",
      "          },\n",
      "          \"cor_xy\": {\n",
      "            \"ann\": \"0.952\",\n",
      "            \"djf\": \"0.95\",\n",
      "            \"mam\": \"0.93\",\n",
      "            \"jja\": \"0.95\",\n",
      "            \"son\": \"0.93\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"0.95\",\n",
      "              \"0.94\",\n",
      "              \"0.94\",\n",
      "              \"0.92\",\n",
      "              \"0.92\",\n",
      "              \"0.94\",\n",
      "              \"0.95\",\n",
      "              \"0.95\",\n",
      "              \"0.94\",\n",
      "              \"0.92\",\n",
      "              \"0.92\",\n",
      "              \"0.94\"\n",
      "            ]\n",
      "          },\n",
      "          \"mae_xy\": {\n",
      "            \"ann\": \"6.331\",\n",
      "            \"djf\": \"7.493\",\n",
      "            \"mam\": \"8.024\",\n",
      "            \"jja\": \"7.628\",\n",
      "            \"son\": \"7.940\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"8.015\",\n",
      "              \"8.693\",\n",
      "              \"8.520\",\n",
      "              \"8.889\",\n",
      "              \"8.656\",\n",
      "              \"7.961\",\n",
      "              \"8.217\",\n",
      "              \"8.243\",\n",
      "              \"8.657\",\n",
      "              \"8.891\",\n",
      "              \"8.307\",\n",
      "              \"7.976\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean-obs_xy\": {\n",
      "            \"ann\": \"240.316\",\n",
      "            \"djf\": \"237.540\",\n",
      "            \"mam\": \"239.327\",\n",
      "            \"jja\": \"243.879\",\n",
      "            \"son\": \"240.536\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"237.511\",\n",
      "              \"237.974\",\n",
      "              \"238.287\",\n",
      "              \"238.996\",\n",
      "              \"240.648\",\n",
      "              \"242.912\",\n",
      "              \"244.361\",\n",
      "              \"244.378\",\n",
      "              \"242.922\",\n",
      "              \"240.568\",\n",
      "              \"238.119\",\n",
      "              \"237.120\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean_xy\": {\n",
      "            \"ann\": \"239.153\",\n",
      "            \"djf\": \"236.635\",\n",
      "            \"mam\": \"238.368\",\n",
      "            \"jja\": \"242.476\",\n",
      "            \"son\": \"239.144\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"236.454\",\n",
      "              \"237.243\",\n",
      "              \"237.693\",\n",
      "              \"238.037\",\n",
      "              \"239.346\",\n",
      "              \"241.591\",\n",
      "              \"242.845\",\n",
      "              \"243.005\",\n",
      "              \"241.343\",\n",
      "              \"239.106\",\n",
      "              \"236.981\",\n",
      "              \"236.193\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_devzm\": {\n",
      "            \"ann\": \"9.363\"\n",
      "          },\n",
      "          \"rms_xy\": {\n",
      "            \"ann\": \"9.610\",\n",
      "            \"djf\": \"10.951\",\n",
      "            \"mam\": \"11.786\",\n",
      "            \"jja\": \"11.219\",\n",
      "            \"son\": \"12.519\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.383\",\n",
      "              \"12.617\",\n",
      "              \"12.416\",\n",
      "              \"13.174\",\n",
      "              \"13.060\",\n",
      "              \"12.071\",\n",
      "              \"12.151\",\n",
      "              \"12.224\",\n",
      "              \"12.989\",\n",
      "              \"14.060\",\n",
      "              \"13.062\",\n",
      "              \"11.749\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_xyt\": {\n",
      "            \"ann\": \"nan\"\n",
      "          },\n",
      "          \"rms_y\": {\n",
      "            \"ann\": \"2.168\"\n",
      "          },\n",
      "          \"rmsc_xy\": {\n",
      "            \"ann\": \"9.540\",\n",
      "            \"djf\": \"10.913\",\n",
      "            \"mam\": \"11.746\",\n",
      "            \"jja\": \"11.131\",\n",
      "            \"son\": \"12.441\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.334\",\n",
      "              \"12.596\",\n",
      "              \"12.402\",\n",
      "              \"13.139\",\n",
      "              \"12.995\",\n",
      "              \"11.998\",\n",
      "              \"12.056\",\n",
      "              \"12.146\",\n",
      "              \"12.893\",\n",
      "              \"13.984\",\n",
      "              \"13.013\",\n",
      "              \"11.713\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy\": {\n",
      "            \"ann\": \"29.647\",\n",
      "            \"djf\": \"32.686\",\n",
      "            \"mam\": \"30.818\",\n",
      "            \"jja\": \"35.375\",\n",
      "            \"son\": \"31.289\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"33.135\",\n",
      "              \"33.734\",\n",
      "              \"33.073\",\n",
      "              \"31.262\",\n",
      "              \"31.721\",\n",
      "              \"34.451\",\n",
      "              \"36.365\",\n",
      "              \"36.271\",\n",
      "              \"34.820\",\n",
      "              \"32.053\",\n",
      "              \"30.983\",\n",
      "              \"31.985\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy_devzm\": {\n",
      "            \"ann\": \"12.983\"\n",
      "          },\n",
      "          \"std-obs_xyt\": {\n",
      "            \"ann\": \"33.469\"\n",
      "          },\n",
      "          \"std_xy\": {\n",
      "            \"ann\": \"31.123\",\n",
      "            \"djf\": \"34.351\",\n",
      "            \"mam\": \"33.081\",\n",
      "            \"jja\": \"37.357\",\n",
      "            \"son\": \"33.759\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"35.083\",\n",
      "              \"36.318\",\n",
      "              \"35.860\",\n",
      "              \"33.875\",\n",
      "              \"34.141\",\n",
      "              \"36.390\",\n",
      "              \"38.237\",\n",
      "              \"38.696\",\n",
      "              \"37.492\",\n",
      "              \"34.881\",\n",
      "              \"32.548\",\n",
      "              \"33.053\"\n",
      "            ]\n",
      "          },\n",
      "          \"std_xy_devzm\": {\n",
      "            \"ann\": \"14.246\"\n",
      "          },\n",
      "          \"std_xyt\": {\n",
      "            \"ann\": \"35.677\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    },\n",
      "    \"units\": \"W m-2\",\n",
      "    \"default\": {\n",
      "      \"source\": \"CERES-EBAF-4-1\",\n",
      "      \"\": {\n",
      "        \"InputClimatologyFileName\": \"cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\",\n",
      "        \"Global\": {\n",
      "          \"bias_xy\": {\n",
      "            \"ann\": \"-1.165\",\n",
      "            \"djf\": \"-0.936\",\n",
      "            \"mam\": \"-1.026\",\n",
      "            \"jja\": \"-1.395\",\n",
      "            \"son\": \"-1.310\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"-1.080\",\n",
      "              \"-0.732\",\n",
      "              \"-0.628\",\n",
      "              \"-1.014\",\n",
      "              \"-1.412\",\n",
      "              \"-1.370\",\n",
      "              \"-1.517\",\n",
      "              \"-1.303\",\n",
      "              \"-1.501\",\n",
      "              \"-1.343\",\n",
      "              \"-1.088\",\n",
      "              \"-0.997\"\n",
      "            ]\n",
      "          },\n",
      "          \"cor_xy\": {\n",
      "            \"ann\": \"0.952\",\n",
      "            \"djf\": \"0.95\",\n",
      "            \"mam\": \"0.93\",\n",
      "            \"jja\": \"0.95\",\n",
      "            \"son\": \"0.93\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"0.95\",\n",
      "              \"0.94\",\n",
      "              \"0.94\",\n",
      "              \"0.92\",\n",
      "              \"0.92\",\n",
      "              \"0.94\",\n",
      "              \"0.95\",\n",
      "              \"0.95\",\n",
      "              \"0.94\",\n",
      "              \"0.92\",\n",
      "              \"0.92\",\n",
      "              \"0.94\"\n",
      "            ]\n",
      "          },\n",
      "          \"mae_xy\": {\n",
      "            \"ann\": \"6.334\",\n",
      "            \"djf\": \"7.487\",\n",
      "            \"mam\": \"8.034\",\n",
      "            \"jja\": \"7.646\",\n",
      "            \"son\": \"7.998\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"8.026\",\n",
      "              \"8.696\",\n",
      "              \"8.557\",\n",
      "              \"8.897\",\n",
      "              \"8.659\",\n",
      "              \"7.953\",\n",
      "              \"8.219\",\n",
      "              \"8.271\",\n",
      "              \"8.655\",\n",
      "              \"8.928\",\n",
      "              \"8.362\",\n",
      "              \"7.874\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean-obs_xy\": {\n",
      "            \"ann\": \"240.318\",\n",
      "            \"djf\": \"237.570\",\n",
      "            \"mam\": \"239.394\",\n",
      "            \"jja\": \"243.872\",\n",
      "            \"son\": \"240.454\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"237.534\",\n",
      "              \"237.974\",\n",
      "              \"238.321\",\n",
      "              \"239.051\",\n",
      "              \"240.758\",\n",
      "              \"242.960\",\n",
      "              \"244.362\",\n",
      "              \"244.308\",\n",
      "              \"242.844\",\n",
      "              \"240.449\",\n",
      "              \"238.069\",\n",
      "              \"237.190\"\n",
      "            ]\n",
      "          },\n",
      "          \"mean_xy\": {\n",
      "            \"ann\": \"239.153\",\n",
      "            \"djf\": \"236.635\",\n",
      "            \"mam\": \"238.368\",\n",
      "            \"jja\": \"242.476\",\n",
      "            \"son\": \"239.144\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"236.454\",\n",
      "              \"237.243\",\n",
      "              \"237.693\",\n",
      "              \"238.037\",\n",
      "              \"239.346\",\n",
      "              \"241.591\",\n",
      "              \"242.845\",\n",
      "              \"243.005\",\n",
      "              \"241.343\",\n",
      "              \"239.106\",\n",
      "              \"236.981\",\n",
      "              \"236.193\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_devzm\": {\n",
      "            \"ann\": \"9.363\"\n",
      "          },\n",
      "          \"rms_xy\": {\n",
      "            \"ann\": \"9.609\",\n",
      "            \"djf\": \"10.919\",\n",
      "            \"mam\": \"11.801\",\n",
      "            \"jja\": \"11.248\",\n",
      "            \"son\": \"12.596\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.412\",\n",
      "              \"12.621\",\n",
      "              \"12.476\",\n",
      "              \"13.176\",\n",
      "              \"13.057\",\n",
      "              \"12.083\",\n",
      "              \"12.152\",\n",
      "              \"12.268\",\n",
      "              \"13.012\",\n",
      "              \"14.098\",\n",
      "              \"13.152\",\n",
      "              \"11.583\"\n",
      "            ]\n",
      "          },\n",
      "          \"rms_xyt\": {\n",
      "            \"ann\": \"nan\"\n",
      "          },\n",
      "          \"rms_y\": {\n",
      "            \"ann\": \"2.159\"\n",
      "          },\n",
      "          \"rmsc_xy\": {\n",
      "            \"ann\": \"9.538\",\n",
      "            \"djf\": \"10.878\",\n",
      "            \"mam\": \"11.757\",\n",
      "            \"jja\": \"11.161\",\n",
      "            \"son\": \"12.527\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"11.361\",\n",
      "              \"12.600\",\n",
      "              \"12.461\",\n",
      "              \"13.137\",\n",
      "              \"12.980\",\n",
      "              \"12.005\",\n",
      "              \"12.057\",\n",
      "              \"12.199\",\n",
      "              \"12.925\",\n",
      "              \"14.034\",\n",
      "              \"13.107\",\n",
      "              \"11.540\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy\": {\n",
      "            \"ann\": \"29.644\",\n",
      "            \"djf\": \"32.736\",\n",
      "            \"mam\": \"30.775\",\n",
      "            \"jja\": \"35.361\",\n",
      "            \"son\": \"31.194\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"33.142\",\n",
      "              \"33.727\",\n",
      "              \"32.912\",\n",
      "              \"31.203\",\n",
      "              \"31.767\",\n",
      "              \"34.483\",\n",
      "              \"36.342\",\n",
      "              \"36.157\",\n",
      "              \"34.665\",\n",
      "              \"31.899\",\n",
      "              \"30.965\",\n",
      "              \"32.014\"\n",
      "            ]\n",
      "          },\n",
      "          \"std-obs_xy_devzm\": {\n",
      "            \"ann\": \"12.976\"\n",
      "          },\n",
      "          \"std-obs_xyt\": {\n",
      "            \"ann\": \"33.418\"\n",
      "          },\n",
      "          \"std_xy\": {\n",
      "            \"ann\": \"31.123\",\n",
      "            \"djf\": \"34.351\",\n",
      "            \"mam\": \"33.081\",\n",
      "            \"jja\": \"37.357\",\n",
      "            \"son\": \"33.759\",\n",
      "            \"CalendarMonths\": [\n",
      "              \"35.083\",\n",
      "              \"36.318\",\n",
      "              \"35.860\",\n",
      "              \"33.875\",\n",
      "              \"34.141\",\n",
      "              \"36.390\",\n",
      "              \"38.237\",\n",
      "              \"38.696\",\n",
      "              \"37.492\",\n",
      "              \"34.881\",\n",
      "              \"32.548\",\n",
      "              \"33.053\"\n",
      "            ]\n",
      "          },\n",
      "          \"std_xy_devzm\": {\n",
      "            \"ann\": \"14.246\"\n",
      "          },\n",
      "          \"std_xyt\": {\n",
      "            \"ann\": \"35.677\"\n",
      "          }\n",
      "        }\n",
      "      }\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import os\n",
    "output_path = os.path.join(demo_output_directory,\"basicTest/rlut_2.5x2.5_regrid2_metrics_basicTest.json\")\n",
    "with open(output_path) as f:\n",
    "    metric = json.load(f)[\"RESULTS\"]\n",
    "print(json.dumps(metric, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customizing parameters in the mean climate driver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to override the parameter file from the command line. Use `mean_climate_driver.py --help` to see all the flag options.  \n",
    "  \n",
    "This next cell demonstrates how to change 'case_id' and 'regrid_tool' on the command line. Changing the 'case_id' is helpful because these results will be stored in a folder with that name, separate from other runs.  \n",
    "\n",
    "Both the model data sets and observations are regridded to a 2.5 by 2.5 degree grid before producing statistics. To interpolate to a different grid, the user should provide a [cdms2 grid object](https://cdms.readthedocs.io/en/latest/manual/cdms_2.html#id9) as the `target_grid`.\n",
    "\n",
    "The two regrid tools available are 'regrid2' and ['xesmf'](https://xesmf.readthedocs.io/en/latest/). 'regrid2' is recommended, but 'xesmf' must be used with non-rectangular grids.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nimport sys\\n!conda install --yes --prefix {sys.prefix} -c conda-forge xesmf\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for conda\n",
    "\"\"\"\n",
    "import sys\n",
    "!conda install --yes --prefix {sys.prefix} -c conda-forge xesmf\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OR delete the triple quotations on lines 2&5 from this cell to install with pip:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nimport sys\\n!{sys.executable} -m pip install git+https://github.com/pangeo-data/xesmf.git\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for pip\n",
    "\"\"\"\n",
    "import sys\n",
    "!{sys.executable} -m pip install git+https://github.com/pangeo-data/xesmf.git\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-09 16:41:28,762 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:41:28,762 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:42::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_ACCESS1-0__2.5x2.5_xesmf_metrics_alternate1_Ex2.json\n",
      "2023-11-09 16:42:31,697 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_ACCESS1-0__2.5x2.5_xesmf_metrics_alternate1_Ex2.json\n",
      "2023-11-09 16:42:31,697 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_ACCESS1-0__2.5x2.5_xesmf_metrics_alternate1_Ex2.json\n",
      "2023-11-09 16:42:31,736 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:42:31,736 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:43::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_CanCM4__2.5x2.5_xesmf_metrics_alternate1_Ex2.json\n",
      "2023-11-09 16:43:43,018 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_CanCM4__2.5x2.5_xesmf_metrics_alternate1_Ex2.json\n",
      "2023-11-09 16:43:43,018 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_CanCM4__2.5x2.5_xesmf_metrics_alternate1_Ex2.json\n",
      "2023-11-09 16:43:47,139 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:43:47,139 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:45::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_ACCESS1-0__2.5x2.5_xesmf_metrics_default_Ex2.json\n",
      "2023-11-09 16:45:02,114 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_ACCESS1-0__2.5x2.5_xesmf_metrics_default_Ex2.json\n",
      "2023-11-09 16:45:02,114 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_ACCESS1-0__2.5x2.5_xesmf_metrics_default_Ex2.json\n",
      "2023-11-09 16:45:02,157 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:45:02,157 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:46::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_CanCM4__2.5x2.5_xesmf_metrics_default_Ex2.json\n",
      "2023-11-09 16:46:12,238 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_CanCM4__2.5x2.5_xesmf_metrics_default_Ex2.json\n",
      "2023-11-09 16:46:12,238 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut/rlut_CanCM4__2.5x2.5_xesmf_metrics_default_Ex2.json\n",
      "INFO::2023-11-09 16:46::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut_2.5x2.5_xesmf_metrics_Ex2.json\n",
      "2023-11-09 16:46:33,802 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut_2.5x2.5_xesmf_metrics_Ex2.json\n",
      "2023-11-09 16:46:33,802 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex2/rlut_2.5x2.5_xesmf_metrics_Ex2.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "case_id:  Ex2 \n",
      " test_data_set: ['ACCESS1-0', 'CanCM4'] \n",
      " realization:  \n",
      " vars: ['rlut'] \n",
      " varname_in_test_data: None \n",
      " reference_data_set: ['all'] \n",
      " target_grid: 2.5x2.5 \n",
      " regrid_tool: xesmf \n",
      " regrid_tool_ocn: esmf \n",
      " save_test_clims: False \n",
      " test_clims_interpolated_output: None \n",
      " filename_template: cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc \n",
      " sftlf_filename_template: sftlf_%(model_version).nc \n",
      " generate_sftlf: True \n",
      " regions_specs: {'global': {}, 'NHEX': {'domain': {'latitude': (30.0, 90)}}, 'SHEX': {'domain': {'latitude': (-90.0, -30)}}, 'TROPICS': {'domain': {'latitude': (-30.0, 30)}}, '90S50S': {'domain': {'latitude': (-90.0, -50)}}, '50S20S': {'domain': {'latitude': (-50.0, -20)}}, '20S20N': {'domain': {'latitude': (-20.0, 20)}}, '20N50N': {'domain': {'latitude': (20.0, 50)}}, '50N90N': {'domain': {'latitude': (50.0, 90)}}, 'CONUS': {'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'land': {'value': 100}, 'land_NHEX': {'value': 100, 'domain': {'latitude': (30.0, 90)}}, 'land_SHEX': {'value': 100, 'domain': {'latitude': (-90.0, -30)}}, 'land_TROPICS': {'value': 100, 'domain': {'latitude': (-30.0, 30)}}, 'land_CONUS': {'value': 100, 'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'ocean': {'value': 0}, 'ocean_NHEX': {'value': 0, 'domain': {'latitude': (30.0, 90)}}, 'ocean_SHEX': {'value': 0, 'domain': {'latitude': (-90.0, -30)}}, 'ocean_TROPICS': {'value': 0, 'domain': {'latitude': (30.0, 30)}}, 'ocean_50S50N': {'value': 0.0, 'domain': {'latitude': (-50.0, 50)}}, 'ocean_50S20S': {'value': 0.0, 'domain': {'latitude': (-50.0, -20)}}, 'ocean_20S20N': {'value': 0.0, 'domain': {'latitude': (-20.0, 20)}}, 'ocean_20N50N': {'value': 0.0, 'domain': {'latitude': (20.0, 50)}}, 'NAM': {'domain': {'latitude': (20.0, 90), 'longitude': (-180, 180)}}, 'NAO': {'domain': {'latitude': (20.0, 80), 'longitude': (-90, 40)}}, 'SAM': {'domain': {'latitude': (-20.0, -90), 'longitude': (0, 360)}}, 'PNA': {'domain': {'latitude': (20.0, 85), 'longitude': (120, 240)}}, 'PDO': {'domain': {'latitude': (20.0, 70), 'longitude': (110, 260)}}, 'AllMW': {'domain': {'latitude': (-40.0, 45.0), 'longitude': (0.0, 360.0)}}, 'AllM': {'domain': {'latitude': (-45.0, 45.0), 'longitude': (0.0, 360.0)}}, 'NAMM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (210.0, 310.0)}}, 'SAMM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (240.0, 330.0)}}, 'NAFM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (310.0, 60.0)}}, 'SAFM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (0.0, 90.0)}}, 'ASM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (60.0, 180.0)}}, 'AUSM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (90.0, 160.0)}}, 'AIR': {'domain': {'latitude': (7.0, 25.0), 'longitude': (65.0, 85.0)}}, 'AUS': {'domain': {'latitude': (-20.0, -10.0), 'longitude': (120.0, 150.0)}}, 'Sahel': {'domain': {'latitude': (13.0, 18.0), 'longitude': (-10.0, 10.0)}}, 'GoG': {'domain': {'latitude': (0.0, 5.0), 'longitude': (-10.0, 10.0)}}, 'NAmo': {'domain': {'latitude': (20.0, 37.0), 'longitude': (-112.0, -103.0)}}, 'SAmo': {'domain': {'latitude': (-20.0, 2.5), 'longitude': (-65.0, -40.0)}}} \n",
      " regions: {'rlut': ['Global']} \n",
      " test_data_path: demo_data/CMIP5_demo_clims/ \n",
      " reference_data_path: demo_data/obs4MIPs_PCMDI_clims \n",
      " metrics_output_path: demo_output/Ex2 \n",
      " diagnostics_output_path: demo_output/Ex2 \n",
      " debug: False \n",
      "\n",
      "--- prepare mean climate metrics calculation ---\n",
      "--- start mean climate metrics calculation ---\n",
      "varname: rlut\n",
      "level: None\n",
      "reference_data_set (all):  ['alternate1', 'default']\n",
      "ref: alternate1\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 38.93927790599767\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.60527090825452\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 2.849440160911837\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 7.557470849444263\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  22.607395028695382 22.427104154412426\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.7861025676947934\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 38.93927790599767\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 41.277546088282016\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -1.3070924524925915\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 9.860459737057527\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  31.29959559143669 31.27229114580701\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.6355158166285978\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "ref: default\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-1/v20210804/rlut_mon_CERES-EBAF-4-1_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 38.896082465222186\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.60527090825452\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 2.847343450867303\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 7.551058394040502\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  22.603314702394204 22.42325736391287\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.7861552158165768\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 38.896082465222186\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 41.277546088282016\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -1.3091891625371264\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 9.862415553120186\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  31.29710758581451 31.269713253797885\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.6355540591449562\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "pmp mean clim driver completed\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mean_climate_driver.py  -p basic_param.py --case_id 'Ex2' --regrid_tool 'xesmf'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specifying the model  \n",
    "It is easy to change the models or variables in the analysis. Whatever variables you choose must be present in both your model files and obervations. This example shows how to specify a variable while running only the ACCESS1-0 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-09 16:46:57,258 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:46:57,258 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:47::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex3.json\n",
      "2023-11-09 16:47:57,629 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex3.json\n",
      "2023-11-09 16:47:57,629 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex3.json\n",
      "2023-11-09 16:48:03,578 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:48:03,578 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:49::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex3.json\n",
      "2023-11-09 16:49:02,702 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex3.json\n",
      "2023-11-09 16:49:02,702 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex3.json\n",
      "INFO::2023-11-09 16:49::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut_2.5x2.5_regrid2_metrics_Ex3.json\n",
      "2023-11-09 16:49:21,917 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut_2.5x2.5_regrid2_metrics_Ex3.json\n",
      "2023-11-09 16:49:21,917 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex3/rlut_2.5x2.5_regrid2_metrics_Ex3.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "case_id:  Ex3 \n",
      " test_data_set: ['ACCESS1-0'] \n",
      " realization:  \n",
      " vars: ['rlut'] \n",
      " varname_in_test_data: None \n",
      " reference_data_set: ['all'] \n",
      " target_grid: 2.5x2.5 \n",
      " regrid_tool: regrid2 \n",
      " regrid_tool_ocn: esmf \n",
      " save_test_clims: False \n",
      " test_clims_interpolated_output: None \n",
      " filename_template: cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc \n",
      " sftlf_filename_template: sftlf_%(model_version).nc \n",
      " generate_sftlf: True \n",
      " regions_specs: {'global': {}, 'NHEX': {'domain': {'latitude': (30.0, 90)}}, 'SHEX': {'domain': {'latitude': (-90.0, -30)}}, 'TROPICS': {'domain': {'latitude': (-30.0, 30)}}, '90S50S': {'domain': {'latitude': (-90.0, -50)}}, '50S20S': {'domain': {'latitude': (-50.0, -20)}}, '20S20N': {'domain': {'latitude': (-20.0, 20)}}, '20N50N': {'domain': {'latitude': (20.0, 50)}}, '50N90N': {'domain': {'latitude': (50.0, 90)}}, 'CONUS': {'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'land': {'value': 100}, 'land_NHEX': {'value': 100, 'domain': {'latitude': (30.0, 90)}}, 'land_SHEX': {'value': 100, 'domain': {'latitude': (-90.0, -30)}}, 'land_TROPICS': {'value': 100, 'domain': {'latitude': (-30.0, 30)}}, 'land_CONUS': {'value': 100, 'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'ocean': {'value': 0}, 'ocean_NHEX': {'value': 0, 'domain': {'latitude': (30.0, 90)}}, 'ocean_SHEX': {'value': 0, 'domain': {'latitude': (-90.0, -30)}}, 'ocean_TROPICS': {'value': 0, 'domain': {'latitude': (30.0, 30)}}, 'ocean_50S50N': {'value': 0.0, 'domain': {'latitude': (-50.0, 50)}}, 'ocean_50S20S': {'value': 0.0, 'domain': {'latitude': (-50.0, -20)}}, 'ocean_20S20N': {'value': 0.0, 'domain': {'latitude': (-20.0, 20)}}, 'ocean_20N50N': {'value': 0.0, 'domain': {'latitude': (20.0, 50)}}, 'NAM': {'domain': {'latitude': (20.0, 90), 'longitude': (-180, 180)}}, 'NAO': {'domain': {'latitude': (20.0, 80), 'longitude': (-90, 40)}}, 'SAM': {'domain': {'latitude': (-20.0, -90), 'longitude': (0, 360)}}, 'PNA': {'domain': {'latitude': (20.0, 85), 'longitude': (120, 240)}}, 'PDO': {'domain': {'latitude': (20.0, 70), 'longitude': (110, 260)}}, 'AllMW': {'domain': {'latitude': (-40.0, 45.0), 'longitude': (0.0, 360.0)}}, 'AllM': {'domain': {'latitude': (-45.0, 45.0), 'longitude': (0.0, 360.0)}}, 'NAMM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (210.0, 310.0)}}, 'SAMM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (240.0, 330.0)}}, 'NAFM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (310.0, 60.0)}}, 'SAFM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (0.0, 90.0)}}, 'ASM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (60.0, 180.0)}}, 'AUSM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (90.0, 160.0)}}, 'AIR': {'domain': {'latitude': (7.0, 25.0), 'longitude': (65.0, 85.0)}}, 'AUS': {'domain': {'latitude': (-20.0, -10.0), 'longitude': (120.0, 150.0)}}, 'Sahel': {'domain': {'latitude': (13.0, 18.0), 'longitude': (-10.0, 10.0)}}, 'GoG': {'domain': {'latitude': (0.0, 5.0), 'longitude': (-10.0, 10.0)}}, 'NAmo': {'domain': {'latitude': (20.0, 37.0), 'longitude': (-112.0, -103.0)}}, 'SAmo': {'domain': {'latitude': (-20.0, 2.5), 'longitude': (-65.0, -40.0)}}} \n",
      " regions: {'rlut': ['Global']} \n",
      " test_data_path: demo_data/CMIP5_demo_clims/ \n",
      " reference_data_path: demo_data/obs4MIPs_PCMDI_clims \n",
      " metrics_output_path: demo_output/Ex3 \n",
      " diagnostics_output_path: demo_output/Ex3 \n",
      " debug: False \n",
      "\n",
      "--- prepare mean climate metrics calculation ---\n",
      "--- start mean climate metrics calculation ---\n",
      "varname: rlut\n",
      "level: None\n",
      "reference_data_set (all):  ['alternate1', 'default']\n",
      "ref: alternate1\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.46854119190699\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.438416968613225\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 1.1384451737888808\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.770012023689812\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  8.044682445768467 7.9637213813349925\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9693780836555635\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "ref: default\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-1/v20210804/rlut_mon_CERES-EBAF-4-1_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.41842190556023\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.438416968613225\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 1.1365878854524063\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.763493973471603\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  8.033820554038556 7.953014565127818\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9694728084795681\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "pmp mean clim driver completed\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mean_climate_driver.py -p basic_param.py --case_id 'Ex3' --test_data_set 'ACCESS1-0' --vars \"rlut\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using custom regions  \n",
    "This example specifies additional regions for the analysis. The predefined regions that can be set by the `--regions` flag can be found in [default_regions.py](https://github.com/PCMDI/pcmdi_metrics/blob/master/share/default_regions.py). By default, the mean climate driver will run \"global\", \"NHEX\", \"SHEX\", and \"TROPICS\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-09 16:49:45,266 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:49:45,266 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:50::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex4.json\n",
      "2023-11-09 16:50:41,150 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex4.json\n",
      "2023-11-09 16:50:41,150 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex4.json\n",
      "2023-11-09 16:50:41,183 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:50:41,183 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:51::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex4.json\n",
      "2023-11-09 16:51:35,777 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex4.json\n",
      "2023-11-09 16:51:35,777 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex4.json\n",
      "2023-11-09 16:51:42,513 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:51:42,513 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:52::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex4.json\n",
      "2023-11-09 16:52:38,464 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex4.json\n",
      "2023-11-09 16:52:38,464 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex4.json\n",
      "2023-11-09 16:52:38,494 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:52:38,494 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:53::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_Ex4.json\n",
      "2023-11-09 16:53:33,822 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_Ex4.json\n",
      "2023-11-09 16:53:33,822 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_Ex4.json\n",
      "INFO::2023-11-09 16:53::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut_2.5x2.5_regrid2_metrics_Ex4.json\n",
      "2023-11-09 16:53:54,053 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut_2.5x2.5_regrid2_metrics_Ex4.json\n",
      "2023-11-09 16:53:54,053 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex4/rlut_2.5x2.5_regrid2_metrics_Ex4.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "case_id:  Ex4 \n",
      " test_data_set: ['ACCESS1-0', 'CanCM4'] \n",
      " realization:  \n",
      " vars: ['rlut'] \n",
      " varname_in_test_data: None \n",
      " reference_data_set: ['all'] \n",
      " target_grid: 2.5x2.5 \n",
      " regrid_tool: regrid2 \n",
      " regrid_tool_ocn: esmf \n",
      " save_test_clims: False \n",
      " test_clims_interpolated_output: None \n",
      " filename_template: cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc \n",
      " sftlf_filename_template: sftlf_%(model_version).nc \n",
      " generate_sftlf: True \n",
      " regions_specs: {'MyDomain': {'domain': {'latitude': (20.0, 30)}}} \n",
      " regions: {'rlut': ['MyDomain']} \n",
      " test_data_path: demo_data/CMIP5_demo_clims/ \n",
      " reference_data_path: demo_data/obs4MIPs_PCMDI_clims \n",
      " metrics_output_path: demo_output/Ex4 \n",
      " diagnostics_output_path: demo_output/Ex4 \n",
      " debug: False \n",
      "\n",
      "--- prepare mean climate metrics calculation ---\n",
      "--- start mean climate metrics calculation ---\n",
      "varname: rlut\n",
      "level: None\n",
      "reference_data_set (all):  ['alternate1', 'default']\n",
      "ref: alternate1\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: MyDomain\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 19.18138602499969\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 22.561827391039035\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 0.5946991696333115\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 4.150349791215329\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  5.291037658821646 5.2575100955401215\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9522270778028106\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: MyDomain\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 19.18138602499969\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 22.245208810088847\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 0.6638373054433819\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.889229301904497\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  7.248559470317926 7.218097701377931\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9066905099914794\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "ref: default\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-1/v20210804/rlut_mon_CERES-EBAF-4-1_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: MyDomain\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 19.049296637854248\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 22.561827391039035\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 0.5913453737090093\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 4.160780868714007\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  5.294374298855912 5.261246037335545\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9522958639707434\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: MyDomain\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 19.049296637854248\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 22.245208810088847\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 0.6604835095190796\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.886274128348927\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  7.247367618574384 7.217208527707502\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9068727312282716\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "pmp mean clim driver completed\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mean_climate_driver.py -p basic_param.py \\\n",
    "--case_id 'Ex4' --regions '{\"rlut\": [\"MyDomain\"]}' \\\n",
    "--regions_specs \"{'MyDomain': {'domain': {'latitude': (20.0, 30)}}}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The region value `None` is another way of indicating \"global\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Custom domain is controlled by the variable `regions_specs` in the parameter file or command line. For example, a custom region for Antarctica over land could be defined with `regions_specs = {'ANT': {'value': 100, 'domain': {'latitude': (-90, -60)}}}` in the parameter file. `value` indicate masking, 100: land only, 0: ocean only.\n",
    "\n",
    "The command to use the custom region would look like `--regions '{\"rlut\": [\"ANT\"]}'` in the command line or `regions = {\"rlut\": [\"ANT\"]}` in the parameter file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Land/sea masks  \n",
    "Land/Sea masking is required for some regions (e.g. \"land\"). This is controlled by variables `sftlf_filename_template` and `generate_sftlf`. The metrics package expects the land/sea mask files to be located with the model data under a fixed field variable ('fx') with a file name that follows the pattern in `sftlf_filename_template`. The example data does not come with a land/sea mask, so we set `generate_sftlf` to `True` in the basic parameter file to have the PMP generate a land/sea mask where 0 represents ocean and 100 represents land."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract vertical level  and customize observations\n",
    "This example shows how to extract a vertical level from a 4-D variable. The 500 hPa level of zg is specified by adding \"\\_500\" to zg.  \n",
    "\n",
    "There are several sets of observations available in the standard PCMDI set. This case uses the 'alternate1' observations for zg with the flag `reference_data_set`. The test and observations path are specified with `test_data_path` and `reference_data_path`, respectively.\n",
    "\n",
    "To see which datasets are the default and which are alternates, check out [pcmdi_metrics/doc/obs_info_dictionary.json](https://github.com/PCMDI/pcmdi_metrics/blob/main/doc/obs_info_dictionary.json)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-09 16:54:19,804 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:54:19,804 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:55::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:55:21,394 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:55:21,394 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "INFO::2023-11-09 16:55::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:55:54,625 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:55:54,625 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "INFO::2023-11-09 16:56::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:56:38,981 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:56:38,981 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "INFO::2023-11-09 16:57::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:57:15,946 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "2023-11-09 16:57:15,946 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500/zg_500_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex5.json\n",
      "INFO::2023-11-09 16:57::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500_2.5x2.5_regrid2_metrics_Ex5.json\n",
      "2023-11-09 16:57:35,313 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500_2.5x2.5_regrid2_metrics_Ex5.json\n",
      "2023-11-09 16:57:35,313 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex5/zg_500_2.5x2.5_regrid2_metrics_Ex5.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "case_id:  Ex5 \n",
      " test_data_set: ['CanCM4'] \n",
      " realization:  \n",
      " vars: ['zg_500'] \n",
      " varname_in_test_data: None \n",
      " reference_data_set: ['alternate1'] \n",
      " target_grid: 2.5x2.5 \n",
      " regrid_tool: regrid2 \n",
      " regrid_tool_ocn: esmf \n",
      " save_test_clims: False \n",
      " test_clims_interpolated_output: None \n",
      " filename_template: cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc \n",
      " sftlf_filename_template: sftlf_%(model_version).nc \n",
      " generate_sftlf: True \n",
      " regions_specs: {'global': {}, 'NHEX': {'domain': {'latitude': (30.0, 90)}}, 'SHEX': {'domain': {'latitude': (-90.0, -30)}}, 'TROPICS': {'domain': {'latitude': (-30.0, 30)}}, '90S50S': {'domain': {'latitude': (-90.0, -50)}}, '50S20S': {'domain': {'latitude': (-50.0, -20)}}, '20S20N': {'domain': {'latitude': (-20.0, 20)}}, '20N50N': {'domain': {'latitude': (20.0, 50)}}, '50N90N': {'domain': {'latitude': (50.0, 90)}}, 'CONUS': {'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'land': {'value': 100}, 'land_NHEX': {'value': 100, 'domain': {'latitude': (30.0, 90)}}, 'land_SHEX': {'value': 100, 'domain': {'latitude': (-90.0, -30)}}, 'land_TROPICS': {'value': 100, 'domain': {'latitude': (-30.0, 30)}}, 'land_CONUS': {'value': 100, 'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'ocean': {'value': 0}, 'ocean_NHEX': {'value': 0, 'domain': {'latitude': (30.0, 90)}}, 'ocean_SHEX': {'value': 0, 'domain': {'latitude': (-90.0, -30)}}, 'ocean_TROPICS': {'value': 0, 'domain': {'latitude': (30.0, 30)}}, 'ocean_50S50N': {'value': 0.0, 'domain': {'latitude': (-50.0, 50)}}, 'ocean_50S20S': {'value': 0.0, 'domain': {'latitude': (-50.0, -20)}}, 'ocean_20S20N': {'value': 0.0, 'domain': {'latitude': (-20.0, 20)}}, 'ocean_20N50N': {'value': 0.0, 'domain': {'latitude': (20.0, 50)}}, 'NAM': {'domain': {'latitude': (20.0, 90), 'longitude': (-180, 180)}}, 'NAO': {'domain': {'latitude': (20.0, 80), 'longitude': (-90, 40)}}, 'SAM': {'domain': {'latitude': (-20.0, -90), 'longitude': (0, 360)}}, 'PNA': {'domain': {'latitude': (20.0, 85), 'longitude': (120, 240)}}, 'PDO': {'domain': {'latitude': (20.0, 70), 'longitude': (110, 260)}}, 'AllMW': {'domain': {'latitude': (-40.0, 45.0), 'longitude': (0.0, 360.0)}}, 'AllM': {'domain': {'latitude': (-45.0, 45.0), 'longitude': (0.0, 360.0)}}, 'NAMM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (210.0, 310.0)}}, 'SAMM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (240.0, 330.0)}}, 'NAFM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (310.0, 60.0)}}, 'SAFM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (0.0, 90.0)}}, 'ASM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (60.0, 180.0)}}, 'AUSM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (90.0, 160.0)}}, 'AIR': {'domain': {'latitude': (7.0, 25.0), 'longitude': (65.0, 85.0)}}, 'AUS': {'domain': {'latitude': (-20.0, -10.0), 'longitude': (120.0, 150.0)}}, 'Sahel': {'domain': {'latitude': (13.0, 18.0), 'longitude': (-10.0, 10.0)}}, 'GoG': {'domain': {'latitude': (0.0, 5.0), 'longitude': (-10.0, 10.0)}}, 'NAmo': {'domain': {'latitude': (20.0, 37.0), 'longitude': (-112.0, -103.0)}}, 'SAmo': {'domain': {'latitude': (-20.0, 2.5), 'longitude': (-65.0, -40.0)}}} \n",
      " regions: {'rlut': ['Global']} \n",
      " test_data_path: demo_data/CMIP5_demo_clims/ \n",
      " reference_data_path: demo_data/obs4MIPs_PCMDI_clims \n",
      " metrics_output_path: demo_output/Ex5 \n",
      " diagnostics_output_path: demo_output/Ex5 \n",
      " debug: False \n",
      "\n",
      "--- prepare mean climate metrics calculation ---\n",
      "--- start mean climate metrics calculation ---\n",
      "varname: zg\n",
      "level: 500.0\n",
      "ref: alternate1\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/zg/ERA-INT/v20210804/zg_mon_ERA-INT_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.zg.198101-200512.AC.v20200426.nc\n",
      "units: m\n",
      "load and regrid done\n",
      "region: global\n",
      "compute metrics start\n",
      "var:  zg\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 266.78874424242247\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 289.171909592503\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -26.72829436270523\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 27.31174888102911\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  36.86848787995158 25.39456002798049\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9982646048078304\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "region: NHEX\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  zg\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 217.77184929408025\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 243.07955688052704\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -33.98470754086787\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 36.12586072486697\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  43.4018204400982 26.99551205063858\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9934590261766263\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "region: SHEX\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  zg\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 283.22012506599816\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 296.6174627856965\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -53.00982429706322\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 53.00982429706322\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  56.8956886480462 20.665863512804005\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9981088912216388\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "region: TROPICS\n",
      "spatial subset done\n",
      "compute metrics start\n",
      "var:  zg\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 28.36627012760418\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 34.43418248863642\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -9.959322806445527\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 10.055655251093725\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  12.575853648293624 7.678800962410827\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9609284596570256\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "pmp mean clim driver completed\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "mean_climate_driver.py -p basic_param.py \\\n",
    "--case_id 'Ex5' \\\n",
    "--vars 'zg_500' \\\n",
    "--test_data_set 'CanCM4' \\\n",
    "--reference_data_set \"alternate1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a couple of steps for using an observational dataset that isn't included in **obs_info_dict.json**. The observational dataset mush be documented in a catalog json following the format used in **obs_info_dict.json**. Then set the variable `custom_observations` to the path for this catalog. The observation files need to be located under the path specified by `reference_data_path`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding notes and other options  \n",
    "This example adds a field to \"user_notes\" in the output json and saves the interpolated fields. Interpolated fields are saved at the same directory level as the parameter file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-11-09 16:58:00,109 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:58:00,109 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 16:58::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex6.json\n",
      "2023-11-09 16:58:59,465 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex6.json\n",
      "2023-11-09 16:58:59,465 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_alternate1_Ex6.json\n",
      "2023-11-09 16:58:59,513 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 16:58:59,513 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 17:00::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex6.json\n",
      "2023-11-09 17:00:04,816 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex6.json\n",
      "2023-11-09 17:00:04,816 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_alternate1_Ex6.json\n",
      "2023-11-09 17:00:10,978 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 17:00:10,978 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 17:01::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex6.json\n",
      "2023-11-09 17:01:07,454 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex6.json\n",
      "2023-11-09 17:01:07,454 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_ACCESS1-0__2.5x2.5_regrid2_metrics_default_Ex6.json\n",
      "2023-11-09 17:01:07,496 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "2023-11-09 17:01:07,496 [WARNING]: dataset.py(_is_decodable:670) >> 'time' does not have a 'units' attribute set so it could not be decoded. Try setting the 'units' attribute (`ds.{coords.name}.attrs['units']`) and try decoding again.\n",
      "INFO::2023-11-09 17:02::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_Ex6.json\n",
      "2023-11-09 17:02:06,691 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_Ex6.json\n",
      "2023-11-09 17:02:06,691 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut/rlut_CanCM4__2.5x2.5_regrid2_metrics_default_Ex6.json\n",
      "INFO::2023-11-09 17:02::pcmdi_metrics:: Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut_2.5x2.5_regrid2_metrics_Ex6.json\n",
      "2023-11-09 17:02:24,735 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut_2.5x2.5_regrid2_metrics_Ex6.json\n",
      "2023-11-09 17:02:24,735 [INFO]: base.py(write:250) >> Results saved to a json file: /Users/lee1043/Documents/Research/git/pcmdi_metrics_20230620_pcmdi/pcmdi_metrics/doc/jupyter/Demo/demo_output/Ex6/rlut_2.5x2.5_regrid2_metrics_Ex6.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "case_id:  Ex6 \n",
      " test_data_set: ['ACCESS1-0', 'CanCM4'] \n",
      " realization:  \n",
      " vars: ['rlut'] \n",
      " varname_in_test_data: None \n",
      " reference_data_set: ['all'] \n",
      " target_grid: 2.5x2.5 \n",
      " regrid_tool: regrid2 \n",
      " regrid_tool_ocn: esmf \n",
      " save_test_clims: True \n",
      " test_clims_interpolated_output: None \n",
      " filename_template: cmip5.historical.%(model_version).r1i1p1.mon.%(variable).198101-200512.AC.v20200426.nc \n",
      " sftlf_filename_template: sftlf_%(model_version).nc \n",
      " generate_sftlf: True \n",
      " regions_specs: {'global': {}, 'NHEX': {'domain': {'latitude': (30.0, 90)}}, 'SHEX': {'domain': {'latitude': (-90.0, -30)}}, 'TROPICS': {'domain': {'latitude': (-30.0, 30)}}, '90S50S': {'domain': {'latitude': (-90.0, -50)}}, '50S20S': {'domain': {'latitude': (-50.0, -20)}}, '20S20N': {'domain': {'latitude': (-20.0, 20)}}, '20N50N': {'domain': {'latitude': (20.0, 50)}}, '50N90N': {'domain': {'latitude': (50.0, 90)}}, 'CONUS': {'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'land': {'value': 100}, 'land_NHEX': {'value': 100, 'domain': {'latitude': (30.0, 90)}}, 'land_SHEX': {'value': 100, 'domain': {'latitude': (-90.0, -30)}}, 'land_TROPICS': {'value': 100, 'domain': {'latitude': (-30.0, 30)}}, 'land_CONUS': {'value': 100, 'domain': {'latitude': (24.7, 49.4), 'longitude': (-124.78, -66.92)}}, 'ocean': {'value': 0}, 'ocean_NHEX': {'value': 0, 'domain': {'latitude': (30.0, 90)}}, 'ocean_SHEX': {'value': 0, 'domain': {'latitude': (-90.0, -30)}}, 'ocean_TROPICS': {'value': 0, 'domain': {'latitude': (30.0, 30)}}, 'ocean_50S50N': {'value': 0.0, 'domain': {'latitude': (-50.0, 50)}}, 'ocean_50S20S': {'value': 0.0, 'domain': {'latitude': (-50.0, -20)}}, 'ocean_20S20N': {'value': 0.0, 'domain': {'latitude': (-20.0, 20)}}, 'ocean_20N50N': {'value': 0.0, 'domain': {'latitude': (20.0, 50)}}, 'NAM': {'domain': {'latitude': (20.0, 90), 'longitude': (-180, 180)}}, 'NAO': {'domain': {'latitude': (20.0, 80), 'longitude': (-90, 40)}}, 'SAM': {'domain': {'latitude': (-20.0, -90), 'longitude': (0, 360)}}, 'PNA': {'domain': {'latitude': (20.0, 85), 'longitude': (120, 240)}}, 'PDO': {'domain': {'latitude': (20.0, 70), 'longitude': (110, 260)}}, 'AllMW': {'domain': {'latitude': (-40.0, 45.0), 'longitude': (0.0, 360.0)}}, 'AllM': {'domain': {'latitude': (-45.0, 45.0), 'longitude': (0.0, 360.0)}}, 'NAMM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (210.0, 310.0)}}, 'SAMM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (240.0, 330.0)}}, 'NAFM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (310.0, 60.0)}}, 'SAFM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (0.0, 90.0)}}, 'ASM': {'domain': {'latitude': (0.0, 45.0), 'longitude': (60.0, 180.0)}}, 'AUSM': {'domain': {'latitude': (-45.0, 0.0), 'longitude': (90.0, 160.0)}}, 'AIR': {'domain': {'latitude': (7.0, 25.0), 'longitude': (65.0, 85.0)}}, 'AUS': {'domain': {'latitude': (-20.0, -10.0), 'longitude': (120.0, 150.0)}}, 'Sahel': {'domain': {'latitude': (13.0, 18.0), 'longitude': (-10.0, 10.0)}}, 'GoG': {'domain': {'latitude': (0.0, 5.0), 'longitude': (-10.0, 10.0)}}, 'NAmo': {'domain': {'latitude': (20.0, 37.0), 'longitude': (-112.0, -103.0)}}, 'SAmo': {'domain': {'latitude': (-20.0, 2.5), 'longitude': (-65.0, -40.0)}}} \n",
      " regions: {'rlut': ['Global']} \n",
      " test_data_path: demo_data/CMIP5_demo_clims/ \n",
      " reference_data_path: demo_data/obs4MIPs_PCMDI_clims \n",
      " metrics_output_path: demo_output/Ex6 \n",
      " diagnostics_output_path: demo_output/Ex6 \n",
      " debug: False \n",
      "\n",
      "--- prepare mean climate metrics calculation ---\n",
      "--- start mean climate metrics calculation ---\n",
      "varname: rlut\n",
      "level: None\n",
      "reference_data_set (all):  ['alternate1', 'default']\n",
      "ref: alternate1\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-0/v20210804/rlut_mon_CERES-EBAF-4-0_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.46854119190699\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.438416968613225\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 1.1384451737888808\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.770012023689812\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  8.044682445768467 7.9637213813349925\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9693780836555635\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.46854119190699\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 35.67665337064905\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -1.1635463134324031\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 6.331106438822097\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  9.610248337339145 9.539550989529207\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9518659641205703\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "ref: default\n",
      "ref_data_full_path: demo_data/obs4MIPs_PCMDI_clims/rlut/CERES-EBAF-4-1/v20210804/rlut_mon_CERES-EBAF-4-1_PCMDI_gn.200301-201812.AC.v20210804.nc\n",
      "'DataArray' object has no attribute 'units'\n",
      "units: \n",
      "ref_data load_and_regrid done\n",
      "=================================\n",
      "model, runs, find_all_realizations: ACCESS1-0 [''] False\n",
      "-----------------------\n",
      "model, run: ACCESS1-0 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.ACCESS1-0.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.41842190556023\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 36.438416968613225\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: 1.1365878854524063\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 5.763493973471603\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  8.033820554038556 7.953014565127818\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9694728084795681\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "=================================\n",
      "model, runs, find_all_realizations: CanCM4 [''] False\n",
      "-----------------------\n",
      "model, run: CanCM4 \n",
      "test_data (model in this case) full_path: demo_data/CMIP5_demo_clims/cmip5.historical.CanCM4.r1i1p1.mon.rlut.198101-200512.AC.v20200426.nc\n",
      "units: W m-2\n",
      "load and regrid done\n",
      "region: Global\n",
      "compute metrics start\n",
      "var:  rlut\n",
      "compute_metrics-CALCULATE ANNUAL CYCLE SPACE-TIME RMS, CORRELATIONS and STD\n",
      "compute_metrics, rms_xyt\n",
      "compute_metrics, rms_xyt: nan\n",
      "compute_metrics, stdObs_xyt\n",
      "compute_metrics, stdObs_xyt: 33.41842190556023\n",
      "compute_metrics, std_xyt\n",
      "compute_metrics, std_xyt: 35.67665337064905\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN BIAS, bias_xy: -1.1654036017688774\n",
      "compute_metrics-CALCULATE MSE\n",
      "compute_metrics-CALCULATE MSE, mae_xy: 6.333692053819536\n",
      "compute_metrics-CALCULATE MEAN RMS\n",
      "compute_metrics-CALCULATE MEAN RMS: rms_xy, rmsc_xy:  9.608637289548378 9.537701243333409\n",
      "compute_metrics-CALCULATE MEAN CORR\n",
      "compute_metrics-CALCULATE MEAN CORR: cor_xy: 0.9518850115349531\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD STD\n",
      "compute_metrics-CALCULATE ANNUAL OBS AND MOD MEAN\n",
      "compute_metrics-CALCULATE ANNUAL MEANS\n",
      "compute_metrics-CALCULATE ANNUAL AND ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN RMS\n",
      "compute_metrics-CALCULATE ANNUAL MEAN DEVIATION FROM ZONAL MEAN STD\n",
      "pmp mean clim driver completed\n"
     ]
    }
   ],
   "source": [
    "%%bash \n",
    "mean_climate_driver.py -p basic_param.py \\\n",
    "--case_id 'Ex6' --user_notes 'Example note' --save_test_clims True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some other flags to note:  \n",
    "`--ext` '.nc' or '.xml'. NetCDF files should be post-processed annual climatologies  \n",
    "`--filename_template` Based on the format of your model files.  \n",
    "    - When the file name varies based on model, experiment, or another parameter, use %(parameter) as a placeholder   \n",
    "    (for example, \"CMIP5.historical.%(model_version)...\" in this demo's parameter file)  \n",
    "`--help` or `-h` Help command for more information about all flags."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pmp_devel_20230223] *",
   "language": "python",
   "name": "conda-env-pmp_devel_20230223-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "selected_variables": [],
  "vcdat_file_path": "",
  "vcdat_loaded_variables": [],
  "vcdat_variable_info": {}
 },
 "nbformat": 4,
 "nbformat_minor": 4
}