Loading software environment on blues

[rljacob]

We were having trouble running on blues. It worked fine if you did "qsub $CASE.run" but a user would get library path errors if they executed $CASE.submit.

@jayeshkrishna traced it to the way CIME loads the machine specific environment in the job submission scripts. This bug is repeated here for further discussion and to check the solution.

[rljacob]

Details from @jayeshkrishna:

"In blues we use the PBS job scheduler and the "-V" option is used within the
job submission scripts. The "-V" imports environment from the caller
processes' environment. PBS modifies some environment variables (like PATH)
in the user's (caller process's) environment while retaining (if "-V" option is used)
the other environment variables from the user's environment.

When submitting a job, the submit script ($CASE.submit),
loads the environment (using ModuleLoader) before submitting the run script
($CASE.run) using the qsub command.

The job submission perl script in CIME loads the environment by running (it reads the env
set by env_mach_specific and explicitly sets its own env accordingly) env_mach_specific.
The module loader in CIME has optimizations to avoid reloading the environment if it is already
loaded (see "CIME_MODULES_LOADED" in ModuleLoader.pm).

Since variables like PATH are modified by PBS and the run script retains the
environment of the submit script, when the run script tries to load the machine specific env
the module loader skips it due to the optimization mentioned above. Hence the PATH
(and other required environment variables) is not set correctly in the run script.
i.e., In the run script PATH is not set correctly while CIME_MODULES_LOADED is set."

[rljacob]

Again from @jayeshkrishna:

"There are two possible fixes to the problem,

• Remove the "-V" option passed to qsub (see settings for pbs in config_batch,xml)

<directive default="/bin/bash" > -S {{ shell }} -V </directive>

• Disable the optimization in CIME that prevents reloading of the user environment (See ModuleLoader.pm)

if(defined $ENV{CIME_MODULES_LOADED}) {return $self};

"

[rljacob]

We've decided to disable the optimization in CIME that prevents reloading of the user environment. Does anyone know a good reason to keep that optimization?

[jedwards4b]

I do - the user environment sometimes causes errors that are difficult to
debug and/or testers set up environments and forget to tell other people
that they need them. We want to keep the cime environment self contained
to the extent possible.

On Thu, Feb 4, 2016 at 4:29 PM, Robert Jacob notifications@github.com
wrote:

We've decided to disable the optimization in CIME that prevents reloading
of the user environment. Does anyone know a good reason to keep that
optimization?

—
Reply to this email directly or view it on GitHub
#56 (comment).

Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

[rljacob]

So you'd be in favor of taking out the -V ?

[jedwards4b]

Yes - I think that we've seen this problem on several machines and the
solution has been to take out the -V.

On Thu, Feb 4, 2016 at 5:36 PM, Robert Jacob notifications@github.com
wrote:

So you'd be in favor of taking out the -V ?

—
Reply to this email directly or view it on GitHub
#56 (comment).

Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

[rljacob]

Thanks. But is that line to prevent reloading in ModuleLoader.pm necessary for any reason?

[jedwards4b]

It was intended to avoid repeating tasks that had already been done. But
it's not in the latest CESM-development or ESMCI master.

On Thu, Feb 4, 2016 at 7:26 PM, Robert Jacob notifications@github.com
wrote:

Thanks. But is that line to prevent reloading in ModuleLoader.pm necessary
for any reason?

—
Reply to this email directly or view it on GitHub
#56 (comment).

Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

[rljacob]

Fixed in ACME.