Pinning affects an external OpenMP library

[PetrKryslUCSD]

TLDR: An external OpenMP-equipped C++ library shows no parallel speedup when thread pinning is used. Disabling pinning fixed the issue.

https://discourse.julialang.org/t/openmp-in-a-shared-library-opened-in-julia/113880/9

[carstenbauer]

Are you perhaps oversubscribing CPU cores? That is, are Julia threads and the external OpenMP threads maybe running on the same set of CPU cores? This would explain why it works better without thread pinning, because the OS scheduler then has the flexibility to avoid this (as much as possible).

If that's the case you should either avoid thread pinning (as you did) or pin the OpenMP threads as well (e.g. through environment variables) - to different cores, of course.

Otherwise, pinthreads shouldn't effect any OpenMP threads: Julia threads aren't even OpenMP threads, they are pthreads.

[PetrKryslUCSD]

I do not think the cpu cores are oversubscribed. When the library is called, only a single thread is running.

[carstenbauer]

I don't understand, I thought you have multiple threads? Why would pinning or not pinning make a big/any difference if there is only a single thread running?

In any case, pinthreads doesn't change anything about other OpenMP threads. Hence, AFAICS, this isn't a bug in ThreadPinning.jl.

(Also, you neither provide much information nor a MWE, which makes this hard/impossible to figure out.)

[PetrKryslUCSD]

The issue was intended as a heads up. I'm working on a minimal working example.

The application runs with multiple threads. However at the point where the library gets called, only a single thread is running. The issue is real: enabling or disabling pin threading affects the performance of the external library.

[carstenbauer]

The issue is real: enabling or disabling pin threading affects the performance of the external library.

To be clear, I'm not arguing that your performance issue isn't real. I'm saying that it (almost certainly) isn't a bug in ThreadPinning.jl.

Of course, feel free to continue the discussion in the Discourse thread. However, much more information will be needed, like:

• When you call into the library from - IIUC - a single Julia thread, how many OpenMP threads are active?
• Where are the OpenMP threads running? Have you pinned them? If so, how?
• Which system are you running on? Laptop? Cluster? Which CPU?
• Is data passed between Julia and the library? If so, is the data, perhaps, local to the Julia thread but far away from the OpenMP thread(s)?
• ...

[PetrKryslUCSD]

Here is the minimal working example (https://www.dropbox.com/scl/fi/3dezt0lp10pdflcq9479z/mwe_eclgraphcolor.tgz?rlkey=14w67j59xz3uk3fczergzhhmu&dl=0).

Here is a sample output:

pkrysl@horntail:~/mwe_eclgraphcolor$ . ./run_mwe.sh
================================================================================
With 1 thread
================================================================================
  Activating project at `~/mwe_eclgraphcolor`
[ Info: Loading ThreadPinning
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.560385 seconds
  Activating project at `~/mwe_eclgraphcolor`
[ Info: NO ThreadPinning
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.498791 seconds
================================================================================
With 2 threads
================================================================================
  Activating project at `~/mwe_eclgraphcolor`
[ Info: Loading ThreadPinning
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.564154 seconds
  Activating project at `~/mwe_eclgraphcolor`
[ Info: NO ThreadPinning
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.280195 seconds
================================================================================
With 4 threads
================================================================================
  Activating project at `~/mwe_eclgraphcolor`
[ Info: Loading ThreadPinning
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.595618 seconds
  Activating project at `~/mwe_eclgraphcolor`
[ Info: NO ThreadPinning
    Updating git-repo `https://github.com/PetrKryslUCSD/ECLGraphColor.jl.git`
   Resolving package versions...
  No Changes to `~/mwe_eclgraphcolor/Project.toml`
  No Changes to `~/mwe_eclgraphcolor/Manifest.toml`
    Building ECLGraphColor → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/28351fdf0c4c6de0974f0f560c914d6bfd9411dc/build.log`
  0.155279 seconds
pkrysl@horntail:~/mwe_eclgraphcolor$

Two Julia files are provided: one adds, uses ThreadPinning (mwe_w.jl). This one does not get any speed up on the graph calculation.

The other one does (mwe_wo.jl).

[PetrKryslUCSD]

@carstenbauer Carsten, I am not saying that it is a bug in ThreadPinning. It would be interesting to find out why thread pinning affects OpenMP threads, though. Wouldn't you agree?