Different resources for individual jobs

[MalgorzataOles]

Dear BatchJobs Developers,
the jobs I am submitting require different memory size, therefore it is inefficient for me to submit them at once with the current version of the 'submitJobs' function. I would really appreciate if there was an opportunity to change the elements of the 'resources' list to vectors.
I would find it very useful.
Best wishes,
Gosia

[berndbischl]

Hi, yeah I know.
I had the same issue, I wrote a small wrapper around submitJobs to do this.
But I agree, we could offer that out of the box.

Can you write down how your preferred call would look like?

[MalgorzataOles]

The example call to the function would be:
submitJobs(reg, ids=c(1:4), resources=list(ncpus=c(2,4), memory=seq(10000,4000,-2000), queue="research-rh6"))
This submits 4 jobs with different memory and number of cpus requirements. Please note that vectors (here ncpus) are recycled in a regular R's fashion.

[berndbischl]

This does not really work. Because we cannot really assume further structure on the "resources" list, as this is completely cluster dependent.

The only option would be have a reources list with as many elements as "ids".
Which begins to look bit clumsy?

[berndbischl]

I might allow this:

resources can be a data.frame alternatively. nrows = length(ids).
If you do that, every row will be converted to a list and set as the actual resources of the job it corresponds to.

Would that be ok?

[berndbischl]

I also want the opinion of @mllg here

[mllg]

Are you sure you have a problem where every job requires a different amount of memory? Or do you have a few "classes"? We could go with a list, as Bernd suggested.

[berndbischl]

I will implement the data.frame option now. This seems most flexible and simple

[MalgorzataOles]

Thank you. data.frame with specified options per job will be great!
So far I was indeed dividing jobs into classes depending on the amount of memory they were using. However, I have a function that calculates this memory requirement for every job separately and this is unique for each of them. That is why having the data.frame with parameters would be very appreciated :)