Nested withPool issues

[lacan]

Hi all,

We have a project where we would need n threads to each work on k threads on a particular problem.

So nested withPool(n) and eachParallel calls seemed to make sense.

However it is using way too many threads, as per this example from 2014
http://markmail.org/message/ccr4zk4v2jh22g3n

https://gist.github.com/TioBorracho/caee991b0b43a5bd70b8

What would be the best way to have nested withPool calls?

All the best

[lacan]

@russel, thank you for adding the label. Are there any news on this?

[russel]

@lacan I am just emerging from a period taken up with conferences and other stuff. I should be able to get back to GPars stuff fairly soon.

In the interim: I am not sure I quite understand the phrasing. "n threads to each work on k threads" doesn't make sense to me. This may be my fault, but it doesn't alter the fact that I don't have a mental model of the problem you have.

[lacan]

Hi @russel
No problem. What I mean is that we were thinking of nesting two eachParallel calls. In my particular case

I have a certain amount of RAM which allows me to store n large image stacks at the same.

For each of these image stacks, I need to load individual slices. Each slice is compressed and takes some CPU power to decompress before it can be loaded to memory, creating a bottleneck during opening, not linked to disk read time, but overhead from the decompression (the files are small too)

For one image, I can run k threads that each open an image and put it in the right place in memory, and that works perfectly.

However, I wanted to run that in parallel for the n images as well, so it looks like this

GParsPool.withPool(n) {
  stacks.eachParallel{ stack ->
    //Allocate memory
    GParsPool.withPool(k) {
      image_files.eachParallel{ image_file ->
        //Open image, and place into stack

This currently works fine, however it creates more threads than what I would expect (n x k) at any given time. Again you can see the minimal example on my first post.

Thanks again for getting back to me.

[russel]

Hummm… an area of GPars I am not that familiar with. I wonder if GParsPool.withPool creates a new thread pool or just reuses the default one. There is also ForkJoinPool.withPool but I am not entirely certain if that is a distinct concept. There is also runForkJoin and forkOffChild to manipulate the fork/join system for creating a tree of jobs on the JVM threads.

What we should create here is a small test program using the nested GParsPool.withPool so as to add to the GPars test suite. Only with a reproducible test is it feasible to do work on this.

[orubel]

ah I see this was resolved using 'Gpars.withExistingPool'.