Document condor job env variables

[titodalcanton]

Some of us have wondered or asked a few times how to control the number of Condor jobs submitted/in idle/running when Pegasus is used. I think this should go into the --help of pycbc_submit_dax, or in PyCBC's docs.

So just
export _CONDOR_DAGMAN_MAX_JOBS_SUBMITTED = 500
of whatever value before running pycbc_submit_dax

[spxiwh]

For the help we should probably suggest to user to specify the config value to pegasus that would get this set in the dag file (as Tom Downes mentioned in the email thread).

This would require a trip to the pegasus docs pages to figure out how to set variables like this in the pegasus config files. If it is then not already possible to set such a value through the PyCBC config files, I can make any necessary changes if needed.

[duncan-brown]

Yes, the right way to do this long term is to get pegasus to do it with a dagman profile.

On Oct 20, 2015, at 10:27 AM, spxiwh notifications@github.com wrote:

For the help we should probably suggest to user to specify the config value to pegasus that would get this set in the dag file (as Tom Downes mentioned in the email thread).

This would require a trip to the pegasus docs pages to figure out how to set variables like this in the pegasus config files. If it is then not already possible to set such a value through the PyCBC config files, I can make any necessary changes if needed.

—
Reply to this email directly or view it on GitHub.

Duncan Brown http://www.gravity.phy.syr.edu/~duncan
Associate Professor of Physics Room 263-1, Department of Physics
Director of the Graduate Program Syracuse University, NY 13244, USA
Phone: 315 443 5993 Fax: 315 443 9103

[duncan-brown]

The right way to do this is to add a command line option allowing the user to add profiles to the site catalog. I will fix this in the Pegasus development branch for merge #559

[duncan-brown]

This is fixed by duncan-brown@adda8e9 which allows the user to specify arbitrary properties in the site catalog using the syntax

--append-site-profile 'SITE:NAMESPACE|KEY:VALUE'

to pycbc_submit_dax. For example, if you want your workflow to take a really long time, pass the option

--append-site-profile 'local:dagman|maxjobs:10'

A complete list of profiles is at https://pegasus.isi.edu/wms/docs/latest/profiles.php