OMP: Error #15 (Duplicate OpenMP runtime) when running heir-opt tests via Bazel

[crockeea]

I am encountering an OpenMP runtime error when running tests for heir-opt using Bazel. The error indicates that multiple copies of the OpenMP runtime (libomp) have been linked into the binary, causing the program to abort during initialization.

Steps to Reproduce (intermittent, may need a failing test?)

Run the lwe-to-polynomial tests (or potentially any test invoking heir-opt with complex lowerings). E.g.,
bazel test //tests/Dialect/LWE/Conversions/lwe_to_polynomial:decrypt_ops.mlir.test

OMP: Error #15: Initializing libomp.so.5, but found unknown library already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/ error: command failed with exit status: -6

I have reproduced this error with the tip of main.

According to AI

It appears that heir-opt is linking against the OpenMP runtime twice. This often happens if:

The toolchain provides -fopenmp implicitly.

A dependency (possibly in @llvm-project) explicitly links a static archive of OpenMP.

Expected Behavior

The tests should run without conflicting OpenMP runtimes. This error makes debugging [even] harder.

[j2kun]

Could you let me know what platform you're running this on?

[crockeea]

WSL with Ubuntu 24.04

[j2kun]

I think this is a mistake in my migration of OpenFHE to the bazel central registry.

https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/openfhe/1.4.2.bcr.1/overlay/BUILD

There you can see linkstatic=True in each of the OpenFHE build targets. I had the same setting in our HEIR-internal build files, but I suspect this is the culprit. I can try to remove them, but I won't have access to my own WSL machine to test it for a while (my basement where my office usually sits is currently under construction).

[j2kun]

bazelbuild/bazel-central-registry#7403 removes linkstatic, and if you would like to try building against that version to test it before that PR is merged, I can give instructions for how to do that. Basically it would be: clone my fork for the BCR repo, update the openfhe version in HEIR's MODULE.bazel to 1.4.2.bcr.2, then build using bazel shutdown && bazel build --enable_bzlmod --registry="file:///path/to/bazel-central-registry" --lockfile_mode=off @heir//:target

[crockeea]

I managed to reproduce the bug before applying the fix.

1. Make a change a comment in a CKKSOps.cpp
2. Introduced a bug into a test inside the Transforms folder.
3. Run bazel test //tests/Transforms/...

I ran it maybe five times and got the OMP error twice. Look for OMP: Error #15: Initializing libomp.so.5, but found unknown:

$ time bazel test //tests/Transforms/...
INFO: Analyzed 470 targets (0 packages loaded, 0 targets configured).
FAIL: //tests/Transforms/apply_folders:empty_patterns.mlir.test (Exit 1) (see /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/execroot/_main/bazel-out/k8-dbg/testlogs/tests/Transforms/apply_folders/empty_patterns.mlir.test/test.log)
INFO: From Testing //tests/Transforms/apply_folders:empty_patterns.mlir.test:
==================== Test output for //tests/Transforms/apply_folders:empty_patterns.mlir.test:
-- Testing: 1 tests, 1 workers --
FAIL: heir :: Transforms/apply_folders/empty_patterns.mlir (1 of 1)
******************** TEST 'heir :: Transforms/apply_folders/empty_patterns.mlir' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
heir-opt --apply-folders --split-input-file /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir | FileCheck /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir
# executed command: heir-opt --apply-folders --split-input-file /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir
# .---command stderr------------
# | /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir:10:26: error: use of value '%0' expects different type than prior uses: 'tensor<2x2xf32>' vs 'tensor<2x3xf32>'
# |   %1 = arith.addf %arg0, %0 : tensor<2x2xf32>
# |                          ^
# | /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir:9:3: note: prior use here
# |   %0 = arith.constant dense<0.0> : tensor<2x3xf32>
# |   ^
# `-----------------------------
# error: command failed with exit status: 1
# executed command: FileCheck /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir
# .---command stderr------------
# | /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir:5:11: error: CHECK: expected string not found in input
# | // CHECK: func.func @add_zero
# |           ^
# | <stdin>:1:1: note: scanning from here
# | // -----
# | ^
# | <stdin>:3:2: note: possible intended match here
# |  func.func @add_empty(%arg0: tensor<2x2xf32>) -> tensor<2x2xf32> {
# |  ^
# | 
# | Input file: <stdin>
# | Check file: /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5114/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/apply_folders/empty_patterns.mlir.test.runfiles/_main/tests/Transforms/apply_folders/empty_patterns.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |            1: // ----- 
# | check:5'0     X~~~~~~~~ error: no match found
# |            2: module { 
# | check:5'0     ~~~~~~~~~
# |            3:  func.func @add_empty(%arg0: tensor<2x2xf32>) -> tensor<2x2xf32> { 
# | check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:5'1      ?                                                                  possible intended match
# |            4:  return %arg0 : tensor<2x2xf32> 
# | check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            5:  } 
# | check:5'0     ~~~
# |            6: } 
# | check:5'0     ~~
# |            7:  
# | check:5'0     ~
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

********************
********************
Failed Tests (1):
  heir :: Transforms/apply_folders/empty_patterns.mlir


Testing Time: 0.22s

Total Discovered Tests: 1
  Failed: 1 (100.00%)
================================================================================
FAIL: //tests/Transforms/convert_to_ciphertext_semantics:tensor_insert.mlir.test (Exit 1) (see /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/execroot/_main/bazel-out/k8-dbg/testlogs/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir.test/test.log)
INFO: From Testing //tests/Transforms/convert_to_ciphertext_semantics:tensor_insert.mlir.test:
==================== Test output for //tests/Transforms/convert_to_ciphertext_semantics:tensor_insert.mlir.test:
-- Testing: 1 tests, 1 workers --
FAIL: heir :: Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir (1 of 1)
******************** TEST 'heir :: Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir' FAILED ********************
Exit Code: 2

Command Output (stdout):
--
# RUN: at line 1
heir-opt /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5142/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir.test.runfiles/_main/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir --convert-to-ciphertext-semantics=ciphertext-size=16 | FileCheck /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5142/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir.test.runfiles/_main/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir
# executed command: heir-opt /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5142/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir.test.runfiles/_main/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir --convert-to-ciphertext-semantics=ciphertext-size=16
# .---command stderr------------
# | OMP: Error #15: Initializing libomp.so.5, but found unknown library already initialized.
# | OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
# `-----------------------------
# error: command failed with exit status: -6
# executed command: FileCheck /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5142/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir.test.runfiles/_main/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir
# .---command stderr------------
# | FileCheck error: '<stdin>' is empty.
# | FileCheck command line:  FileCheck /home/ericrockett/.cache/bazel/_bazel_ericrockett/c04ad07bb8cb01ca87786ae56c1bdac8/sandbox/linux-sandbox/5142/execroot/_main/bazel-out/k8-dbg/bin/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir.test.runfiles/_main/tests/Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir
# `-----------------------------
# error: command failed with exit status: 2

--

********************
********************
Failed Tests (1):
  heir :: Transforms/convert_to_ciphertext_semantics/tensor_insert.mlir


Testing Time: 3.59s

Total Discovered Tests: 1
  Failed: 1 (100.00%)
================================================================================
[5,512 / 5,514] 229 / 231 tests, 2 failed; 2 actions running; last test: //tests/Transforms/heir_simd_vectorizer:gx_kernel_16x16.mlir.test
    Testing //tests/Transforms/mlir_to_tfhe_rs:variance.mlir.test; 15s linux-sandbox
    Testing //tests/Transforms/yosys_optimizer:micro_speech_for.mlir.test; 11s linux-sandbox

[crockeea]

Since your PR was merged to BCR:main, here's what I tried (guided by AI), starting at a root directory ~:

1. Edit line 44 of MODULE.bazel to read bazel_dep(name = "openfhe", version = "1.4.2.bcr.2")
2. Execute the following:
   ~$ git clone https://github.com/bazelbuild/bazel-central-registry.git
   ~$ cd heir
   ~/heir$ bazel shutdown
   ~/heir$ bazel test --registry="file://~/bazel-central-registry" --lockfile_mode=off //tests/Transforms/...

I got the OMP error again on the first try.

[j2kun]

OK my next guess is that your system is linking in both GCC's and LLVM's openmp here. While I'm not quite sure how to avoid this yet, let's try to confirm or refute the situation:

Check if you've got both openmps on your system:

find /usr -name "libomp*.so*" -o -name "libgomp*.so*" -o -name "libiomp*.so*" 2>/dev/null

For example, on my Ubuntu 24.04 laptop I see the following (libgomp is GCC's, and libomp.so.5 is LLVM's)

/usr/lib/llvm-18/lib/libomptarget.rtl.amdgpu.so.18.1
/usr/lib/llvm-18/lib/libomptarget.rtl.x86_64.so
/usr/lib/llvm-18/lib/libomptarget.so
/usr/lib/llvm-18/lib/libomp-18.so.5
/usr/lib/llvm-18/lib/libomp.so
/usr/lib/llvm-18/lib/libiomp5.so
/usr/lib/llvm-18/lib/libomptarget.rtl.cuda.so
/usr/lib/llvm-18/lib/libomp.so.5
/usr/lib/llvm-18/lib/libomptarget.so.18.1
/usr/lib/llvm-18/lib/libomptarget.rtl.x86_64.so.18.1
/usr/lib/llvm-18/lib/libomptarget.rtl.amdgpu.so
/usr/lib/llvm-18/lib/libomptarget.rtl.cuda.so.18.1
/usr/lib/llvm-18/lib/libgomp.so
/usr/lib/llvm-18/lib/libompd.so
/usr/lib/x86_64-linux-gnu/libgomp.so.1
/usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
/usr/lib/x86_64-linux-gnu/libiomp5.so
/usr/lib/x86_64-linux-gnu/libomp.so.5
/usr/lib/x86_64-linux-gnu/libomp5.so
/usr/lib/gcc/x86_64-linux-gnu/13/libgomp.so
/usr/lib/gcc/x86_64-linux-gnu/10/libgomp.so

Check what heir-opt has linked in it

ldd bazel-bin/tools/heir-opt | grep omp

I see just

libomp.so.5 => /lib/x86_64-linux-gnu/libomp.so.5 (0x00007d552cad3000)

[crockeea]

My output is about the same:

$ find /usr -name "libomp*.so*" -o -name "libgomp*.so*" -o -name "libiomp*.so*" 2>/dev/null
/usr/lib/gcc/x86_64-linux-gnu/13/libgomp.so
/usr/lib/gcc/x86_64-linux-gnu/14/libgomp.so
/usr/lib/x86_64-linux-gnu/libiomp5.so
/usr/lib/x86_64-linux-gnu/openmpi/lib/openmpi3/libompi_dbg_msgq.so
/usr/lib/x86_64-linux-gnu/openmpi/lib/libompitrace.so
/usr/lib/x86_64-linux-gnu/libomp.so.5
/usr/lib/x86_64-linux-gnu/libompitrace.so.40.30.1
/usr/lib/x86_64-linux-gnu/libompitrace.so.40
/usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0
/usr/lib/x86_64-linux-gnu/libgomp.so.1
/usr/lib/x86_64-linux-gnu/libomp5.so
/usr/lib/x86_64-linux-gnu/libompitrace.so
/usr/lib/llvm-18/lib/libiomp5.so
/usr/lib/llvm-18/lib/libomp.so.5
/usr/lib/llvm-18/lib/libompd.so
/usr/lib/llvm-18/lib/libomptarget.so
/usr/lib/llvm-18/lib/libomptarget.rtl.x86_64.so
/usr/lib/llvm-18/lib/libomp-18.so.5
/usr/lib/llvm-18/lib/libomptarget.rtl.cuda.so
/usr/lib/llvm-18/lib/libomptarget.rtl.cuda.so.18.1
/usr/lib/llvm-18/lib/libomptarget.rtl.x86_64.so.18.1
/usr/lib/llvm-18/lib/libgomp.so
/usr/lib/llvm-18/lib/libomptarget.rtl.amdgpu.so
/usr/lib/llvm-18/lib/libomptarget.so.18.1
/usr/lib/llvm-18/lib/libomp.so
/usr/lib/llvm-18/lib/libomptarget.rtl.amdgpu.so.18.1

$ ldd bazel-bin/tools/heir-opt | grep omp
        libomp.so.5 => /lib/x86_64-linux-gnu/libomp.so.5 (0x00007e60e217f000)

[j2kun]

OK now I'm rather confused. From your earlier stderr I still think that it's complaining about heir-opt

executed command: heir-opt [...] OMP: Error #15: Initializing libomp.so.5, but found unknown library already initialized.

But this suggests that heir-opt does not have another openmp linked...

[crockeea]

After spending a long time with Gemini, I found that I am unable to reproduce the issue if I run bazel test //tests/Transforms/... --strategy=TestRunner=sandboxed --sandbox_tmpfs_path=/dev/shm. Gemini suggests adding

test --strategy=TestRunner=sandboxed
test --sandbox_tmpfs_path=/dev/shm

to .bazelrc.

I was able to trigger the bug with sandboxing alone; Gemini suggests the need for a sandbox_tmpfs_path implies that the root cause is "stale IPC (Inter-Process Communication) artifacts".

[crockeea]

Resolved by #2619 (comment)