Question - Ray parallelization

[serbinsh]

Hello

I am experimenting with running hytools image corrections on our cluster and noticed that while it states running on 4 CPUs, most of the time the processes is rather sequential or low CPU usage? There are short periods where the number of CPUs selected (4) are used concurrently, but a lot of other times they are not?

Is this normal? It looks like the parallelization is running images on a CPU correct? Such that it would not help to have more CPUs availible than images?

[serbinsh]

Also FYI - here is a repo we are developing for running hytools jobs on our cluster. Still very much a WIP

https://github.com/TESTgroup-BNL/hytools_helpers

[oldmanye007]

Basically, it is a coarse-grained parallelism. Yes, if available CPUs>number of images, only one CPU is paired to one image.

…

On Wed, Mar 9, 2022 at 1:40 PM Shawn P. Serbin ***@***.***> wrote: Hello I am experimenting with running hytools image corrections on our cluster and noticed that while it states running on 4 CPUs, most of the time the processes is rather sequential or low CPU usage? There are short periods where the number of CPUs selected (4) are used concurrently, but a lot of other times they are not? [image: Screen Shot 2022-03-09 at 2 35 32 PM] <https://user-images.githubusercontent.com/3067369/157519388-aa035466-fe83-4ad1-9033-cffda0a954a5.png> [image: Screen Shot 2022-03-09 at 2 37 37 PM] <https://user-images.githubusercontent.com/3067369/157519448-110caaed-b13b-4734-8573-ae4e0909d035.png> Is this normal? It looks like the parallelization is running images on a CPU correct? Such that it would not help to have more CPUs availible than images? — Reply to this email directly, view it on GitHub <#9>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABEO2NVWJSUUJ4VHJRKYVO3U7D5BVANCNFSM5QKQXDQA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[serbinsh]

OK thanks @oldmanye007