relationship between worker number, clustermq core and foreach n_jobs

[liutiming]

Hi @wlandau, thank you so much for creating and maintaining this package!

I will really appreciate if you could clarify the relationships between tar_make_clustermq(worker = n1), clustermq::register_dopar_cmq(n_jobs=n2) and #BSUB-n {{ cores | n3 }} in the tmpl file. There is a further complexity of tar_make(resource = list(cores = n4)) but I think n4 substitutes n3 for individual targets?

I asked @mschubert a similar question before but in this case it seems I have to increase both n1 and n3 for foreach to increase performance. Will be happy to benchmark! It seems to be a quite theoretical question at the moment.

related: mschubert/clustermq#225

[wlandau]

n1 is is just like clustermq::Q(), and it controls among-target parallelism. clustermq initializes n1 persistent workers, which could potentially correspond to individual compute nodes on a cluster, and then targets sends each target to a worker when the worker is available and the target's dependencies have checked or built. From the clustermq docs, it looks like n1 and n2 are the same (n2 should get passed to Q() as n_jobs). n4 and n3 control within-target parallelism: for example, if your target's command uses mclapply() and needs access to multiple physical cores on the compute node to have noticeable performance benefits. targets passes n4 to clustermq functions, which then pass it directly to n3.