Running tasks with varying "core" requirements in same batch job

[TomGlanzman]

It would be beneficial to run tasks with different core_per_worker requirements within a single batch job. The motivation is to run a heterogeneous set of tasks (tasks with differing core requirements) on the same compute node at NERSC. I have a workflow that generates many tasks (identical code, different data) to run under the same (htex) executor so that the tasks run in the same batch job. Each task, in general, needs a different number of cores. The Cori machine has batch "Haswell" nodes with either 32 cores (and 64 hw threads) or 68 cores (and 272 hw threads). To efficiently utilize the node, one must be able to keep as many core busy as possible.

This request is to support the ability for the user to specify the number of needed cores at task creation time and to have the appropriate bookkeeping performed to avoid oversubscribing a node.

For "SimpleLauncher", this would mean Parsl would have to handle bookkeeping (i.e., #cores available vs in-use) For "SrunLauncher", srun would presumably do the bookkeeping (potentially across multiple nodes).

[annawoodard]

Work Queue supports this functionality. This would be a great use case for the WorkQueueExecutor, but there may be a few tweaks needed in to fully exploit those features. cc @dthain, @btovar, @tjdasso

[annawoodard]

Cross ref #1326

[TomGlanzman]

@annawoodard does the WorkQueueExecutor have the same basic functionality as the htex? That is, is wqex a superset of htex or, if not, what functionality will be given up?

[tjdasso]

In regards to WorkQueue, each task can specify the number of cores required. However, the WorkQueue executor does not currently specify the number of cores per task, but it shouldn't be a hard feature to implement. With that being said, how will the Parsl app specify the number of cores it needs to run the task, when it is submitted to the WQExecutor?

[annawoodard]

One possibility is to add it as a keyword argument to the decorator, for example:

@python_app(cores=2)
def foo():
    return 'Hello, world!'

This would be defined here and here for the python and bash app decorators, which would pass it along to the app here. Then at call time before this we could just add it into the kwargs that will be serialized along with the function and passed to the submit method here, at which point WQ could pull it out of the kwargs and use it to specify the cores per task.

[danielskatz]

as a reminder, we generally have tried to avoid mixing resource info and program/app info. I don't know if there's any way to not do this in the context of this issue, however

[btovar]

If so, I would recommend:

@python_app(resources = {cores=2})
def etc...

as the list of resources may get long, and you may not want to have a super long list of attributes.

[annawoodard]

@danielskatz I don't think my proposal above is in conflict with our 'write once, run anywhere' aspiration. If you know your task has fixed resource requirements, then I don't see the problem with saying so in the code-- that's not going to change. The thing that does change is where you are running it, and that is still nicely factorized out in the config.

[btovar]

as a reminder, we generally have tried to avoid mixing resource info and program/app info. I don't know if there's any way to not do this in the context of this issue, however

I think one can make the case that 'resources' are really closer to describing the app (kind of like an argument to malloc), rather than the 'resource' where the app will run.

[danielskatz]

ok, ok ...

[annawoodard]

as the list of resources may get long, and you may not want to have a super long list of attributes.

@btovar I slightly favor keyword args because in my view it's a bit more natural to document the options, their types and defaults in the docstring:

python_app(..., cores=1, memory=None)
    Decorator function for making python apps.

    Parameters
    ----------
    ...
    cores: int
        Number of cores the task needs. Default is 1.
    memory: float
        Memory to provision per task in MB. Default is ...
    ...

which can be accessed in the interpreter via help(python_app). (Of course, it could also be documented as a single keyword arg, so maybe this isn't a particularly persuasive point.) The other advantage is that it makes it easier to specify and test that the passed type is what is expected in mypy (cc @benclifford).

[btovar]

@btovar I slightly favor keyword args because in my view it's a bit more natural to document the options, their types and defaults in the docstring:

Ah yes, that makes sense.

[annawoodard]

does the WorkQueueExecutor have the same basic functionality as the htex? That is, is wqex a superset of htex or, if not, what functionality will be given up?

@TomGlanzman The main differences that come to mind are are 1) at the moment WQ is not pip installable, so you would need to do that as a separate step (but my understanding is that it will be very soon), and 2) wqex was added recently so while WQ itself is mature and robust software, there may be a few kinks to iron out with the executor because it is so fresh it hasn't been extensively tested 'in the wild' yet.

[TomGlanzman]

Thanks @annawoodard. Is the suggestion that I attempt to migrate to the wqex at some point or that some of its functionality will be incorporated into the htex? (It is not clear to me how wqex might be used at NERSC.)

[dgasmith]

This is great functionality! With our use case we would need to set this on a per task basis not a per function basis. A simple example would be if I had a function the would GEMM different sized matrices, I would want to allocate more cores to larger matrices.

[tjdasso]

I've been working on implementing cores as an option to the decorator for python_apps and bash_apps, so that, at the least, you can specify core requirements by function, if not by individual task (yet). As @annawoodard suggested, I added the cores value to the kwargs of the function, which then gets passed to the submit() function of the executor. Because the kwargs are passed directly to the Parsl function later in the executor, the submit() function reads the value of cores from the kwargs and saves it, before popping it from kwargs immediately afterward. This allows us to pass kwargs directly to the Parsl function later in the executor (as most executors seem to do). However, if you were to implement cores as an option to the decorator that then gets added to the kwargs right before submission, I think this would affect a lot of the other executors, as they would need to handle this added cores option within the executor before the function gets submitted.

[tjdasso]

The bare-bones implementation can be found here. After testing, it seems to be working with the Work Queue executor, but as I mentioned above, the changes to parsl/app/app.py and parsl/app/python.py might have side effects on the other executors' functionality. To use, simply pass in the number of cores as a parameter to the Python app decorator, ie, @python_app(cores=2) will specify the task to require 2 cores on the Work Queue worker.

If this is of any use, we can also implement other resource options to the decorator, as Work Queue tasks can specify the amount of memory and disk allocation needed for execution as well.

[annawoodard]

@tjdasso that is fantastic! @btovar correctly pointed out at ParslFest that my recommendation was probably not quite careful enough because it could result in collisions with user kwargs passed to the app. I think the super careful way to deal with that would be to add another dictionary decorator_args or similar that we pass around analogously to how we pass around kwargs now, rather than inserting the decorator args into the app's keyword args. But that change would affect a lot of code and require a change to the executor interface. I think for now we should stick with the approach you've taken, but perhaps we can make it slightly safer by putting all of the resource specification into a single less-clobberable arg, so here instead of

        self.kwargs["cores"] = self.cores

you could have for example (thinking ahead to adding the other resources),

    self.kwargs['parsl_resource_specification'] = {
        'cores': self.cores,
        'disk': self.disk,
        'mem': self.mem
    }

(i.e., going with something like @btovar's original recommendation, just hiding it from the user in order to simplify the documentation.)

I think besides that tweak what you have implemented is great and we should add mem and disk and open a PR. I don't think we need to worry much about side effects to the other executors, as they can always just ignore the additional kwargs.

[annawoodard]

Thanks @annawoodard. Is the suggestion that I attempt to migrate to the wqex at some point or that some of its functionality will be incorporated into the htex? (It is not clear to me how wqex might be used at NERSC.)

@TomGlanzman Yes the idea here would be to try wqex. @tjdasso and others in the CCL team are working currently on getting WQ pip-installable.

With our use case we would need to set this on a per task basis not a per function basis. A simple example would be if I had a function the would GEMM different sized matrices, I would want to allocate more cores to larger matrices.

Sorry I missed this earlier @dgasmith. That's a really good point. I don't love our existing 'magic' keyword args ('stdout', 'stderr', etc) because I think people find it confusing, but your use case makes a lot of sense.

@tjdasso in light of @dgasmith's use case (and just to be more careful to avoid clobbering), perhaps it's sufficient to just check if parsl_resource_specification is already in the kwargs before inserting the ones passed from the decorator. That way the user could always pass their own to override the decorator args.

[tjdasso]

Ok, I implemented all three resource requirements (mem, cores, and disk) as options to the decorator, or as keyword argument parsl_resource_specification with a dictionary of resource specifications to the function itself, working with the per-task use case. Please see PR #1358. It looks like it is failing the testing in CI right now, because as I suspected, most executors pass in kwargs directly to their function when executing the function call, and since there is an additional kwarg (parsl_resource_specification), the function doesn't know how to handle the unexpected argument, and thus fails. Is there any way to only add the parsl_resource_specification if we are using the Work Queue executor? (The WQ Executor obviously handles this additional kwarg while the other executors do not.)

[annawoodard]

tagging @josephmoon, who has a related use case

[TomGlanzman]

I have been following this discussion (and #1358 ) with interest and have a couple of questions:

1. Has the dust settled sufficiently that I might try to use the wqex with the new resource management tools or are code and ideas still in flux?

2. Could issue Timing considerations for submitting new tasks to a running batch job #1325 be combined with this work? That is, could "time" be considered just another resource and handled appropriately by the job dispatching mechanism?

Thanks for your interest in this topic,
- Tom

[dthain]

Speaking from the WQ side of things:

1 - It's close, but we aren't there yet. We need to propagate the properties down to the WQ layer and then exploit them.

2 - Work Queue does not do currently do anything about the time property. However, it would not be hard to have the WQ worker accept an "end time", propagate that back to the master, and then skip scheduling tasks that run over the end time. I think the tricky part would be getting accurate info propagated from the scheduler.

[dthain]

@tjuedema is going to take over from @tjdasso on this issue (from the WQ side of things, anyhow.)

@tjuedema please look into getting the cores, memory, and disk from decorated Parsl tasks, and then pass them down to the WQ executor, so that each task can then be labelled appropriately. With that info in place, WQ should automatically "pack" multiple tasks into a single worker.

[annawoodard]

@tjuedema nice to 'meet' you! FYI, I put a sketch here that might be useful.

[benclifford]

closing this as the WorkQueueExecutor has been providing this functionality without problems in DESC