Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for limiting the number of concurrently executing jobs #27

Open
dan-blanchard opened this issue Jan 13, 2014 · 3 comments
Open
Milestone

Comments

@dan-blanchard
Copy link
Contributor

Currently there's no way to tell process_jobs (or consequently grid_map) that you only want a certain number of jobs to execute at a time. I think this should be fairly straightforward to add to the JobMonitor class by making it keep track of how many jobs are actively running and place jobs on hold or suspend them—I'm not sure what the difference in DRMAA is yet—when they request their inputs and function if the concurrent job limit is exceeded.

@cjw85
Copy link
Contributor

cjw85 commented Sep 15, 2015

I've not read through all of the JobMonitor code, is there a reason that the jobs aren't submitted collectively as a job array? The latter versions of grid engine provide the -tc submission flag which limits the number of concurrently executing tasks of a job array. Submitting individual jobs seems quite integral to how everything works at the moment though.

@dan-blanchard
Copy link
Contributor Author

@cjw85 The primary reason is that GridMap can be used with DRMs that don't support job arrays. We've tried to make as little of it Grid Engine specific as possible.

@cjw85
Copy link
Contributor

cjw85 commented Sep 15, 2015

Fair enough, I thought that might be the reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants