Plugins for zppy

[golaz]

Plugins for zppy

A number of users have expressed interest in adding new functionality to zppy. Until now, the process of adding new tasks to zppy required directly modifying the zppy source code to add the appropriate hooks in the python code and add new Jinja2 templates. This is unnecessarily complicated and could cause the zppy code base to become bloated over time with code that should live in a separate repository. The goal of zppy is to streamline the workflow, not to directly incorporate tools and diagnostics.

The new zppy plugin functionality aims to solve these issues by providing a very simple mechanism for users to call custom python code without any modifications to the zppy source code. The plugin is maintained in a separate location, and the user only needs to modify the zppy configuration file:

[default]
plugins = "/pathToMyGreatPlugin/myGreatPlugin.py" # list of one or more external plugins

# Add a new section for each external plugin

[myGreatPlugin]
# Needed onfiguration options go here...

Minimum directory structure of the plugin code:

myGreatPlugin.py    # main code with a function 'myGreatPlugin' that will be called by zppy
templates/
   # all the Jinja2 templates needed

Over time, we envision that certain user-created plugins will mature and gain sufficient adoption to become official zppy plugins. At that point, the plugin source code should be included into E3SM Unified and zppy will be modified to know its location without having to specify it explicitly in the configuration file. But testing and ensuring compatibility with new zppy releases will remain the responsibility of the plugin maintainer.

[forsyth2]

This will be great, thanks @golaz!

[golaz]

A few more ideas for consideration

1. Implement mechanism to validate and provide default values for [sections] in configuration file that correspond to plugin. The user plugin would have to provide its own default.ini.
2. In [default], plugins variable could point to a configuration file (json, yaml) instead of the actual plugin python file. This would provide more flexibility for naming convention and possibilities for future expansion.

[chengzhuzhang]

@golaz I'm very excited about this new feature! it can make zppy more extensible and effective to serve as an interface for needed tools.
I feel it might be useful to have a cookiecutter python package repository setup as a template for meeting requirement to be included into E3SM-unified (i.g a conda release) . @xylar is setting up a new repo following one example set up by@tomvothecoder. I think a more minimum example will be useful for each plugin maintainers.

[xylar]

@golaz and @tangq, I don't think providing a path to a plugin is going to be a good approach if you want plugins to become conda packages. The path to the code will be pretty convoluted, machine specific and not easy for a user to figure out. Instead, it would be better if it were a python package with a standard format and you simply gave the name of the python package. (It would be even better the python packages for zppy plugins all started with zppy- because that's a very common convention for python plugins.)

You can use importlib.resources to get file locations and read text from files in python packages. This would be my strong recommendation here.

[golaz]

@xylar, thank you for raising those good points.

We could certainly change how plugins are specified. As I was adapting @hsiangheleellnl and @tangq code, I looked for the simplest possible entry option to get users started interfacing their python code. As the development of a plugin matures, the code should become a standard python package. But it might be better to require this from the onset.

I like the idea of names starting with zppy-, not sure if some developers might prefer a standalone name though.

[xylar]

I see, it depends on if the package has a life if its own outside of being a zppy plugin. Then, the zppy prefix wouldn't make much sense.

[xylar]

We decided on chemdyg to match the repo name for that particular package. No zppy- prefix.

[golaz]

Unresolved: how do zppy plugins provide their dafault.ini settings, and how does zppy handle them.

[xylar]

Unresolved: how do zppy plugins provide their dafault.ini settings, and how does zppy handle them.

I would think you could have a standard filename (probably default.ini) that you look for in the root of each plugin package. Again, you can use importlib.resources to determine if there is such a file in the package and read it if so.

[tangq]

I think it is a good idea to have chemdyg's own default.ini if it is possible. As of now, all the chemdyg settings are in the cfg e3sm_chem_diags (will be renamed to chemdyg) session in this example.

[xylar]

@golaz, how will zppy's internally set config options be passed on to plugins? In particular, I would like to avoid duplicating this kind of code in the plugin if possible:

zppy/zppy/__main__.py

Lines 77 to 102 in c3f70dc

	if ("machine" not in config["default"]) or (config["default"]["machine"] == ""):
	# MachineInfo below will then call `discover_machine()`,
	# which only works on log-in nodes.
	machine = None
	else:
	# If `machine` is set, then MachineInfo can bypass the
	# `discover_machine()` function.
	machine = config["default"]["machine"]
	machine_info = MachineInfo(machine=machine)
	default_machine = machine_info.machine
	(
	default_account,
	default_partition,
	default_constraint,
	_,
	) = machine_info.get_account_defaults()
	unified_base = machine_info.config.get("e3sm_unified", "base_path")
	config["default"]["diagnostics_base_path"] = machine_info.config.get(
	"diagnostics", "base_path"
	)
	config["default"]["web_portal_base_path"] = machine_info.config.get(
	"web_portal", "base_path"
	)
	config["default"]["web_portal_base_url"] = machine_info.config.get(
	"web_portal", "base_url"
	)

Is there a way that zppy can pass the config object when it calls the plugin?

[xylar]

Sorry, silly question. This is clearly the case looking at the code.

[forsyth2]

@golaz Here's my commit of fixes you'll need: 3b2338e

I ran the integration tests. I had to add in a if "plugins" in user_config["default"].keys(): line to get zppy to run at all. Other than that, they look to be good in shape. The test_bash_generation, test_campaign, and test_defaults integration tests will need to have their expected results updated after merging (e.g., https://github.com/E3SM-Project/zppy/blob/main/tests/integration/generated/directions_chrysalis.md#commands-to-run-to-replace-outdated-expected-files).

The unit tests pass with the fixes in the commit.

I also ran the pre-commit checks and made two changes to get those passing.

[forsyth2]

I updated expected files for the integration tests, so it looks like those are all passing now.