Support setting MODULEPATH via config.yaml

[aekiss]

Our ACCESS-OM3 configurations (e.g. MOM6-CICE6) rely on the user doing

module use /g/data/ik11/spack/0.20.1/share/modules/linux-rocky8-cascadelake

prior to payu run in order to put the right versions of dependencies in the MODULEPATH environment variable (see https://github.com/COSIMA/access-om3/wiki/Quick-start#running and #337).

This is error-prone, especially as we will likely require particular spack versions to support particular model releases.

Would it be possible to modify payu to support a moduleuse list in config.yaml? It already supports modules, so it seems appropriate for config.yaml to also record the module use. This way it will show up in the git runlog etc.

[aidanheerdegen]

There is an undocumented feature** where you can set arbitrary environment variables with payu

payu/payu/experiment.py

Line 463 in aacfd92

env = self.config.get('env')

So in your case adding

    env:
        MODULEPATH: '/g/data/ik11/spack/0.20.1/share/modules/linux-rocky8-cascadelake'

should do what you want.

That doesn't mean your suggestion isn't worthwhile, but this is a work-around, and may be sufficient?

** I couldn't find any documentation for it, but would be happy to be corrected. If so it should be added to the documentation.

[aekiss]

Thanks @aidanheerdegen that sounds perfect - does this append to MODULEPATH rather than redefining it?

[aekiss]

looks like it redefines MODULEPATH

payu/payu/experiment.py

Lines 461 to 476 in aacfd92

	# Set MPI environment variables
	env = self.config.get('env')
	# Explicitly check for `None`, in case of an empty `env:` entry
	if env is None:
	env = {}
	for var in env:
	if env[var] is None:
	env_value = ''
	else:
	env_value = str(env[var])
	os.environ[var] = env_value

[aidanheerdegen]

Yeah it does redefine it. Seems by default:

$ echo $MODULEPATH
/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles

So yeah, probably have to put the whole lot in there. That is ugly.

Probably worth testing that it works though, and we can decide/work on incorporating something more elegant, like "if string has a leading : then append rather than replace". That is an existing idiom IIRC.

[aidanheerdegen]

That is an existing idiom IIRC.

Actually it isn't an existing idiom. But it can be, we just have to make it so!

[aidanheerdegen]

The implementation is outlined in the docs

https://payu.readthedocs.io/en/latest/config.html#miscellaneous

e.g.

modules:
   use:
      - /path/to/module/directory
   load:
      - netcdf-c-4.9.0
      - parallel-netcdf-1.12.3
      - xerces-c-3.2.3

[aekiss]

thanks @aidanheerdegen and @jo-basevi !