Skip to content

ADF python API

Jump to bottom Edit New page
Jesse Nusbaumer edited this page Jul 19, 2022 · 4 revisions

If you have a python script that can accept multi-variable CAM history (h0) files, single-variable time series files, or single-variable monthly climatology ("climo") files, then you can likely connect it to the ADF so that it runs with every ADF diagnostics run. This particular wiki page describes how to create that connection.

If you have any questions or issues with the instructions here, then please open up a discussion thread on the ADF discussions page.

Required script modifications

  • Any python script that is going to be called by the ADF must be stored under one of the ADF/scripts/XXX directories, where XXX corresponds to analysis, averaging, plotting, and regridding. The following guidelines should help in determining where your script should be placed:

    1. analysis -> Location for a script that performs some sort of mathematical or statistical analysis on the model output.
    2. averaging -> Location for a script that converts model history or time-series files into some sort of temporal or spatial average.
    3. plotting -> Location for any script which creates some sort of plot or visualization.
    4. regridding -> Location for any script which performs regridding.

  • Any python script that will be called by the ADF must be in one of the directories above and have the following internal function header for the script:

    def func_name(adf, optional_A = A, optional_B = B, ...):

    where func_name is the name of your script, adf is the ADF object that must always be passed, and optional_A and optional_B are example optional variables that can be passed using keyword arguments (these are not required).

  • Finally, the filename for the script should ideally be <func_name>.py, where <func_name> matches func_name in your function header.

Using the ADF object in a script

The ADF object currently provides access to all of the variables present in the config YAML file, including any extra variables you add to the file. If you want to access any config variables present under the diag_basic_info, diag_cam_climo, or diag_cam_baseline_climo sections, then you can use the following calls in your script (with the variables chosen acting as examples):

html_logical = adf.get_basic_info("create_html")  #To access variables in diag_basic_info

overwrite_climo = adf.get_cam_info("cam_overwrite_climo") #To access variables in diag_cam_climo

baseline_start_year = adf.get_baseline_info("start_year") #To access variables in `diag_cam_baseline_climo`

Please note that if a particular variable isn't found then these functions will simply return None. If you instead want the ADF run to abort with an error message if the variable isn't found, then you can pass the optional required variable to true, e.g.:

html_logical = adf.get_basic_info("create_html", required=True) #If create_html isn't present, then ADF run will die here.

Along with these access functions, there are two additional, special variables that one can use:

  • adf.diag_var_list -> List of all model variables as specified in diag_var_list in the config file.
  • adf.plot_location -> Special sub-directory where all plots/tables used in the website must be stored.

Please note that plot_location is currently only available for scripts that exist under the analysis or plotting script directories.

Finally, if you need to access a variable outside of those three sections, including a variable you added, then you can use the generic adf.read_config_var() method, which behaves the same as the adf.get_XXX methods above, but can grab any variable in the config file. Please note however that unless a sub-section is specified, this method can only grab top-level variables. For example, if you wanted to access the create_html logical using this method, then you would have to do the following:

basic_info_dict = adf.read_config_var("diag_basic_info")
html_logical = adf.read_config_var("create_html", conf_dict=basic_info_dict)

Adding script outputs to the generated ADF website

If your script creates either an image file (.e.g a png file of a plot) or a Pandas Data frame, then those results can be added to the auto-generated website created during an ADF run. You simply have to call the following somewhere in your script:

adf.add_website_data(web_data, web_name, case_name)

Where web_data is the path to an image file or a pandas dataframe, web_name is the "name" of the plot, which is usually either the variable being plotted or the name of the case being analyzed, and case_name, which is again the case name, or another dataset being analyzed. Additional optional inputs are:

  • category -> The "category" under which the plot or table will be included. If not present it will try to find the category in the defaults file for the given web_name, and if that doesn't work will simply default to no category yet.

  • season -> The season under which this particular plot or table should be included. If not present then it is assumed the plot needs no seasonal separation on the website.

  • plot_type -> The "type" of plot/table under which the figure should be included. If not present then it will be included under the Special type.

  • multi_case -> Logical which tells the website whether or not this particular plot or table can include multiple cases in the same figure. If this is set to True then case_name can also be set to None, in which case the plot or table will be included under a "multi-case" section.

For example, let's say I have a script that makes contour maps over North America for a given variable and season. I could add those plots to the ADF website by doing the following:

plot_path = "/path/to/NA_contour_map.png" #Could also be a "Pathlib" object.
variable  = "TS" #Plotting surface temperature, note that TS already has a category listed in "lib/adf_variable_defaults.yaml"
season    = "JJA" #Plot is for the June-July-August season
case      = adf.adf.get_baseline_info("cam_case_name") #Analyzing the baseline case

adf.add_website_data(plot_path, variable, case, season=season, plot_type="NA_contour_map")

Similarly, if I had a script that generated a pandas dataframe (called table_df) that contained the moisture budget values for all of the ADF test cases, then I could add it to the website by doing the following:

adf.add_website_data(table_df, "Moisture Budget", None, plot_type="Budget Tables", multi_case=True)

which will then add a link to a rendered table under the multi-case website section.

Adding debug output to your script

If you have print statements in your script that you believe are useful for debugging, but which probably shouldn't be output during normal runs, then you can use the ADF's debug_log method, which can be used in your script like so:

adf.debug_log("Some message that is useful during debugging")

Using this function means that if someone passes the --debug argument to run_adf_diag, then the message will be written to the ADF_debug.log file at the appropriate time. Otherwise the message will never be printed.

Testing an ADF script without running the ADF

During development you may find that you want to run your ADF script without running through the entire ADF itself (calculating climatology files, regridding those files, etc.). This can be done by initializing an ADF object inside a python script or juypter notebook, and then passing that object directly into your script. For example, let's say you have already done an ADF run using the config file awesome_possum.yaml, and you want to see if a new ADF-converted plotting script called polar_bare.py works as expected. That can be done via the following python code (assuming you are in the ADF directory):

#Import standard python modules
import os
import os.path
import sys

#Determine local directory path:
local_path = os.path.abspath(os.curdir)

#set path to ADF lib:
lib_path = os.path.join(local_path,"lib")

#set path to ADF plotting scripts directory:
plotting_scripts_path = os.path.join(local_path, "scripts", "plotting")

#Add paths to python path:
sys.path.append(lib_path)
sys.path.append(plotting_scripts_path)

#Import ADF diagnostics object:
from adf_diag import AdfDiag

#Import script being tested:
from polar_bare import polar_bare

#Set path to config YAML file:
config_fil = "/path/to/awesome_possum.yaml"

#Initialize ADF object:
adf = AdfDiag(config_fil)

#Run test_script:
polar_bare(adf)

You can also check that the script interfaces correctly by adding it to the plotting_scripts list in your config file, initializing the ADF object as shown above, and then running:

adf.create_plots()

to run all scripts listed under plotting_scripts for that particular config file, including the file you want to test.

Add a custom footer
Add a custom sidebar
Clone this wiki locally