-
I am working on an example of adding a user-defined plot. To do it, I put a new script into the scripts/plotting. Inside that script is the function that should get called. It should be able to use arguments as specified in in the yaml file. This is sort of working, but I'm running into an issue now that the code has a few classes that seem to obscure the configuration (i.e., yaml) information. What is the right way to have a user specify arguments, and how should AdfDiag get them? For example, in the function I am trying to implement I want to provide it with the case names, locations of the climo files, and the output plots directory. That is to say, this info from the yaml file:
When I first wrote the code that did this, I could look in the locals() namespace, but now these seem to be buried in nested structures. Is there some class that is floating around in there where these are all still just simple attributes? Or is there a better way to get this info? |
Beta Was this translation helpful? Give feedback.
Replies: 13 comments 6 replies
-
Hi @brianpm I am not sure where exactly in the calling order you are, but if you have access to the
Also note that if you are inside an
which represent the Finally, as an FYI, the reason this Hope that helps, and let me know if you run into any problems! |
Beta Was this translation helpful? Give feedback.
-
Hi @brianpm , Ahh, now I understand what your issue is. I think your fix is fine, and your code modifications look good to me! The only thing I might change would be to see if the In terms of a long-term plan to provide arguments to various plotting scripts, I have wondered if instead of passing a bunch of arguments we just passed the ADF object itself (along with optional kwargs that are plotting script specific), and then it would be up to the plotting scripts themselves to use As someone who has been on the plotting-scripts side what do you think of this? Would it help or hinder your scripts development? |
Beta Was this translation helpful? Give feedback.
-
I agree on both. The After working through this example, I also started thinking that giving users access to the ADF object might be the better way to go. As it stands now, the user is likely to have to write an "interface" function that calls their plotting function. Having a clean way to get the information from ADF might make this easier overall. It could make it a little harder for beginners, but this might save work in the long run because that ADF object could potentially bring a lot of useful methods with it (working with catalogs, etc.). |
Beta Was this translation helpful? Give feedback.
-
I'm getting a bit concerned watching this thread and looking at the code that this is steep learning curve for developers: I couldn't find the actual plotting code when I looked, and most of the discussion is well beyond my python knowledge. I recognize there are new standards here, but we need to consider how user-modifiable and user-developable we want this to be. And as a possible developer I don't even see where the plotting calls actually are in the scripts. Are we setting this bar too high for developers? How can we make it easier? |
Beta Was this translation helpful? Give feedback.
-
@brianpm yes, that is exactly what I was thinking! I am not sure what the specific ADF object functions will be yet (it could take a couple of iterations with DTF/AMP to get it set-up in away that makes everyone happy), but the general idea shouldn't change. Since I need to modify some of the plotting scripts anyway in order to enable Intake-ESM and the observations catalog, I can see about implementing a test version of this object-passing interface there. @andrewgettelman part of the underlying design for the ADF is to allow you to call sections of the ADF in python/Jupyter without having to run the entire ADF itself, with the goal being exactly what you outlined (loading ADF to get the object to run a plotting script, but avoiding all of the re-gridding, time series generation, etc.). The one downside to python as a language is that there is no easy way to query the object in python itself to know which functions are available in the object. To alleviate this, I hope to have only a few generic functions where you can query any variable you want, and then to just provide good documentation on the Github wiki or elsewhere describing how to use it. I suspect for most plotting scripts the requested variables will generally be the same, so hopefully the learning curve will be pretty shallow. |
Beta Was this translation helpful? Give feedback.
-
@andrewgettelman @brianpm I have made a working example of what this object-passing interface for a script would look like for the
with the original version here:
Obviously the new version requires extra lines to extract the necessary variables, but the actual function interface is much simplified (as well as the code in Anyways, let me know if you have any strong thoughts or opinions. I should note that script-specific keyword arguments ( Thanks! |
Beta Was this translation helpful? Give feedback.
-
Hi Jesse, I think that looks okay to me. Is there a way to query the adf object to see what's in it? I gather the principle is to pass all this information and let each diagnostic grab the files in their own way? Just to note that this code seems to be hardcoded to do a baseline or a cam case. Shouldn't it be more general (any case, especially if we go multi-case). Also Why does it care what case name it will get? How would it select a case name with the ADF object if there are multiple cases? That might require the old version and an interface. I'd like to see everything in ADF be extensible to multiple cases since it's high on our development list. |
Beta Was this translation helpful? Give feedback.
-
Hi @andrewgettelman , Right now the only variables that will be available are all contained within the config YAML file, plus a couple extra "convenience" variables that I will document in a wiki page that hopefully will describe how to interface a script with the ADF. Also all of the adf functions can behave like query functions, in that if one requests a variable and it isn't found then the function simply returns Finally, the ADF object maintains the variable type as contained within the YAML file. Thus if one converts That being said, some of the scripts (like the re-gridding example I showed) are hard-coded such that they expect only one test case (hence why that code looks like it does). It shouldn't take too much work to modify the scripts to expect multiple cases, but it still has to be done. |
Beta Was this translation helpful? Give feedback.
-
@brianpm in response to your original issue, the ADF has been updated such that now every script should have the ADF object passed to it, from which you can access any config file variable. Instructions for using the ADF object in a script can be found here: https://github.com/NCAR/ADF/wiki/ADF-python-API obviously that is a first pass at those instructions, so if there is any section that is confusing or needs to be worded better please let me know. Finally, if this new interface works for you then it would be great if you could mark this particular post as "the answer" for this discussion thread. That way if another user looks at this discussion they'll more likely see the URL to the script interface wiki page shown above. Thanks! |
Beta Was this translation helpful? Give feedback.
-
@nusbaume This is great. Thanks for doing this and for making the super-helpful wiki page. I very much like the part about how to run your diagnostic outside of the ADF (would be good to have a working example in a notebook, maybe even for the hackathon). I was a bit concerned with the several 'Please Note' comments on the wiki: I think structure is fine, but can see some of those 'please notes' (about what scripts can call plotting, variables in the configuration file being buried under sub-heads) adding too much complexity. Just another plug for flat and simple if we can. |
Beta Was this translation helpful? Give feedback.
-
@nusbaume and @cecilehannay , Where is a good place to put a list of variables we should have defaults for? Somewhere we can make a list and people can check it off. Maybe it's just a google sheet we can link to in an issue? Not sure if github discussions or the wiki would work. We now have a functional way to modify variable defaults for ranges and contour intervals, easy to customize, but it would be great to have a list of variables we want to attack. I started with a few of them I will soon put into a PR (issue #87) Ideas? |
Beta Was this translation helpful? Give feedback.
-
@andrewgettelman I would probably vote to make a new Gihub discussion with a link to a google sheet where people can add the variables they want, and discuss it on the new discussion thread itself. Then once it looks like all of the relevant folks are happy with the variable list we can convert the discussion thread into an issue and start the actual coding. Of course if you need any help with any of that just let me know! |
Beta Was this translation helpful? Give feedback.
-
Done: discussion #90 . Hopefully people will see this.... I mentioned you and Cecile, I'll flag @juliecaron, @JulioTBacmeister and @bitterbark here as well so they may go to the discussion of variables and the document for listing priorities.... |
Beta Was this translation helpful? Give feedback.
@andrewgettelman @brianpm I have made a working example of what this object-passing interface for a script would look like for the
regrid_example.py
script. Specifically, compare the version on glade here (with the ADF object interface):/glade/work/nusbaume/SE_projects/model_diagnostics/ADF_fork/scripts/regridding/regrid_example_new.py
with the original version here:
/glade/work/nusbaume/SE_projects/model_diagnostics/ADF_fork/scripts/regridding/regrid_example.py
Obviously the new version requires extra lines to extract the necessary variables, but the actual function interface is much simplified (as well as the code in
adf_diag.py
). Plus this method gives the script access to the ADF deb…