Use env in drmaa.wrapper.run_job_locally

[ghost]

The job environment can be specified when jobs are run via

run_job_using_drmaa(... job_environment = None ...)

but not when the job is executed locally:

def run_job_locally (cmd_str, logger = None):

    process = subprocess.Popen(  cmd_str,
                                 cwd = os.getcwd(),
                                 shell = True,
                                 stdin = subprocess.PIPE,
                                 stdout = subprocess.PIPE,
                                 stderr = subprocess.PIPE )

If a tool requires specific PATHs to be present that are in conflict with current environment variables, this complicates switching between grid and local mode.

[bunbun]

Thanks for the bug report

drmaa_wrapper.run_job_locally() doesn't support setting the working directory either.
The changes to support both have been checked into the git hub repository and will be in the next release of Ruffus (2.6.3)

The unittest is test/test_drmaa_wrapper_run_job_locally.py

I have also taken the opportunity to tweak drmaa_wrapper.run_job_locally():

When running commands on a remote computational cluster (usually in parallel), one obviously should not expect to see the console results as they happen. drmaa_wrapper.run_job() returns the stdout and stderr results in two lists of strings.

This is annoying when running the job locally (e.g. for debugging) when it would be nice to see the results scrolling on the screen as they come out.

There is now a "local_echo" parameter that causes both stdout and stderr to be echoed to the local console, as well as saved in the resulting two lists

# stdout and sterr are two lists of lines (python strings)
stdout, stderr = ruffus.drmaa_wrapper.run_job(cmd_str = "ls -la",
                                                 # set environment
                                                 job_environment = environ,
                                                 # set working directory
                                                 working_directory = home_dir,
                                                 run_locally = True,
                                                 # run with echo
                                                 local_echo = False)

The echoed output, however, only guarantees that the output line order are preserved among stdout lines and among stderr lines. In rare cases, when there is interleaved stdout and stderr output, a stdout line will appear before a stderr or vice versa.

Leo

[ghost]

Thanks for the quick response.
Ran all unit tests successfully with Python 3.2.3 - works like a charm!

Peter

[ghost]

Hi Leo,
I just discovered that the problem still exists in 2.6.3 (Python 3.2), but in another location (it's not covered by any of the unit tests). This time, it's the def enqueue_output(out, queue, echo) method, where the iterator returns iter(out.readline, b''), but it should be str. It only happens when jobs are run locally with echo, that's why I hadn't noticed that before :-( Sorry for that!
Best
Peter

[bunbun]

Hi Peter,
Sorry about that.
This is fixed in the repository and will be in the next release.
I am always a bit surprised how fiddly this unicode / bytestring division is to get right.
Leo