MLIR Regression: 54 tests started to fail recently

[tuliom]

I've been seeing these errors for almost 2 weeks now.
They have been reproduced on aarch64-linux and x86_64-linux.

I have been able to bisect these failures to #141248.

This is the output of one of the tests:

******************** TEST 'MLIR :: Transforms/constant-fold-debuginfo.mlir' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/mlir-opt /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir -split-input-file -test-constant-fold -mlir-print-debuginfo | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir
# executed command: /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/mlir-opt /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir -split-input-file -test-constant-fold -mlir-print-debuginfo
# executed command: /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir
# .---command stderr------------
# | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir:5:17: error: CHECK-NEXT: expected string not found in input
# |  // CHECK-NEXT: [[C:%.+]] = arith.constant 6 : i32 loc(#[[UnknownLoc:.*]])
# |                 ^
# | <stdin>:2:27: note: scanning from here
# |  func.func @fold_and_merge() -> (i32, i32) {
# |                           ^
# | <stdin>:3:4: note: possible intended match here
# |  %c6_i32 = arith.constant 6 : i32
# |    ^
# | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir:20:12: error: CHECK: expected string not found in input
# |  // CHECK: arith.constant 1.{{0*}}e+00 : f32 loc(#[[UnknownLoc:.*]])
# |            ^
# | <stdin>:10:42: note: scanning from here
# |  func.func @materialize_different_dialect() -> (f32, f32) {
# |                                          ^
# | <stdin>:11:9: note: possible intended match here
# |  %cst = arith.constant 1.000000e+00 : f32
# |         ^
# | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir:33:17: error: CHECK-NEXT: expected string not found in input
# |  // CHECK-NEXT: arith.constant 6 : i32 loc(#[[UnknownLoc:.*]])
# |                 ^
# | <stdin>:18:33: note: scanning from here
# |  func.func @materialize_in_front(%arg0: memref<8xi32>) {
# |                                 ^
# | <stdin>:19:12: note: possible intended match here
# |  %c6_i32 = arith.constant 6 : i32
# |            ^
# | 
# | Input file: <stdin>
# | Check file: /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | <<<<<<
# |             1: module { 
# |             2:  func.func @fold_and_merge() -> (i32, i32) { 
# | next:5'0                                 X~~~~~~~~~~~~~~~~~~ error: no match found
# |             3:  %c6_i32 = arith.constant 6 : i32 
# | next:5'0       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:5'1          ?                               possible intended match
# |             4:  return %c6_i32, %c6_i32 : i32, i32 
# | next:5'0       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             5:  } 
# | next:5'0       ~~~
# |             6: } 
# | next:5'0       ~~
# |             7:  
# | next:5'0       ~
# |             8: // ----- 
# | next:5'0       ~~~~~~~~~
# |             9: module { 
# | next:5'0       ~~~~~~~~~
# |            10:  func.func @materialize_different_dialect() -> (f32, f32) { 
# | next:5'0       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:20'0                                              X~~~~~~~~~~~~~~~~~~ error: no match found
# |            11:  %cst = arith.constant 1.000000e+00 : f32 
# | check:20'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:20'1             ?                                  possible intended match
# |            12:  return %cst, %cst : f32, f32 
# | check:20'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            13:  } 
# | check:20'0     ~~~
# |            14: } 
# | check:20'0     ~~
# |            15:  
# | check:20'0     ~
# |            16: // ----- 
# | check:20'0     ~~~~~~~~~
# |            17: module { 
# | check:20'0     ~~~~~~~~~
# |            18:  func.func @materialize_in_front(%arg0: memref<8xi32>) { 
# | check:20'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:33'0                                      X~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |            19:  %c6_i32 = arith.constant 6 : i32 
# | next:33'0      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:33'1                 ?                       possible intended match
# |            20:  affine.for %arg1 = 0 to 8 { 
# | next:33'0      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            21:  memref.store %c6_i32, %arg0[%arg1] : memref<8xi32> 
# | next:33'0      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            22:  } 
# | next:33'0      ~~~
# |            23:  return 
# | next:33'0      ~~~~~~~~
# |            24:  } 
# | next:33'0      ~~~
# |             .
# |             .
# |             .
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

[llvmbot]

@llvm/issue-subscribers-mlir

Author: Tulio Magno Quites Machado Filho (tuliom)

I've been seeing these errors for almost 2 weeks now. They have been reproduced on aarch64-linux and x86_64-linux.

I have been able to bisect these failures to #141248.

This is the output of one of the tests:

******************** TEST 'MLIR :: Transforms/constant-fold-debuginfo.mlir' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 1
/builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/mlir-opt /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir -split-input-file -test-constant-fold -mlir-print-debuginfo | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir
# executed command: /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/mlir-opt /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir -split-input-file -test-constant-fold -mlir-print-debuginfo
# executed command: /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/llvm/redhat-linux-build/bin/FileCheck /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir
# .---command stderr------------
# | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir:5:17: error: CHECK-NEXT: expected string not found in input
# |  // CHECK-NEXT: [[C:%.+]] = arith.constant 6 : i32 loc(#[[UnknownLoc:.*]])
# |                 ^
# | <stdin>:2:27: note: scanning from here
# |  func.func @<!-- -->fold_and_merge() -&gt; (i32, i32) {
# |                           ^
# | &lt;stdin&gt;:3:4: note: possible intended match here
# |  %c6_i32 = arith.constant 6 : i32
# |    ^
# | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir:20:12: error: CHECK: expected string not found in input
# |  // CHECK: arith.constant 1.{{0*}}e+00 : f32 loc(#[[UnknownLoc:.*]])
# |            ^
# | &lt;stdin&gt;:10:42: note: scanning from here
# |  func.func @<!-- -->materialize_different_dialect() -&gt; (f32, f32) {
# |                                          ^
# | &lt;stdin&gt;:11:9: note: possible intended match here
# |  %cst = arith.constant 1.000000e+00 : f32
# |         ^
# | /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir:33:17: error: CHECK-NEXT: expected string not found in input
# |  // CHECK-NEXT: arith.constant 6 : i32 loc(#[[UnknownLoc:.*]])
# |                 ^
# | &lt;stdin&gt;:18:33: note: scanning from here
# |  func.func @<!-- -->materialize_in_front(%arg0: memref&lt;8xi32&gt;) {
# |                                 ^
# | &lt;stdin&gt;:19:12: note: possible intended match here
# |  %c6_i32 = arith.constant 6 : i32
# |            ^
# | 
# | Input file: &lt;stdin&gt;
# | Check file: /builddir/build/BUILD/llvm-21.0.0_pre20250526.gc76e2800e3d78f-build/llvm-project-c76e2800e3d78fa6e8618431a6893f66f2fd3496/mlir/test/Transforms/constant-fold-debuginfo.mlir
# | 
# | -dump-input=help explains the following input dump.
# | 
# | Input was:
# | &lt;&lt;&lt;&lt;&lt;&lt;
# |             1: module { 
# |             2:  func.func @<!-- -->fold_and_merge() -&gt; (i32, i32) { 
# | next:5'0                                 X~~~~~~~~~~~~~~~~~~ error: no match found
# |             3:  %c6_i32 = arith.constant 6 : i32 
# | next:5'0       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:5'1          ?                               possible intended match
# |             4:  return %c6_i32, %c6_i32 : i32, i32 
# | next:5'0       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |             5:  } 
# | next:5'0       ~~~
# |             6: } 
# | next:5'0       ~~
# |             7:  
# | next:5'0       ~
# |             8: // ----- 
# | next:5'0       ~~~~~~~~~
# |             9: module { 
# | next:5'0       ~~~~~~~~~
# |            10:  func.func @<!-- -->materialize_different_dialect() -&gt; (f32, f32) { 
# | next:5'0       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:20'0                                              X~~~~~~~~~~~~~~~~~~ error: no match found
# |            11:  %cst = arith.constant 1.000000e+00 : f32 
# | check:20'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | check:20'1             ?                                  possible intended match
# |            12:  return %cst, %cst : f32, f32 
# | check:20'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            13:  } 
# | check:20'0     ~~~
# |            14: } 
# | check:20'0     ~~
# |            15:  
# | check:20'0     ~
# |            16: // ----- 
# | check:20'0     ~~~~~~~~~
# |            17: module { 
# | check:20'0     ~~~~~~~~~
# |            18:  func.func @<!-- -->materialize_in_front(%arg0: memref&lt;8xi32&gt;) { 
# | check:20'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:33'0                                      X~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# |            19:  %c6_i32 = arith.constant 6 : i32 
# | next:33'0      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:33'1                 ?                       possible intended match
# |            20:  affine.for %arg1 = 0 to 8 { 
# | next:33'0      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            21:  memref.store %c6_i32, %arg0[%arg1] : memref&lt;8xi32&gt; 
# | next:33'0      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |            22:  } 
# | next:33'0      ~~~
# |            23:  return 
# | next:33'0      ~~~~~~~~
# |            24:  } 
# | next:33'0      ~~~
# |             .
# |             .
# |             .
# | &gt;&gt;&gt;&gt;&gt;&gt;
# `-----------------------------
# error: command failed with exit status: 1

[tuliom]

Full build log of these failures: builder-live.log.gz

[tuliom]

Cc @hhkit @Moxinilian

[joker-eph]

What is you environment? (distro, host compiler version, libstdc++ version, libc version, ...)?
What kind of cmake flags are you using?

The test fails like if the -mlir-print-debuginfo option has no effect.
I'm struggling to see how the commit you're linking to can be the issue, I suspect some undefined behavior because of some sort of ODR violation across shared objects or something like that.

[nikic]

The test fails like if the -mlir-print-debuginfo option has no effect. I'm struggling to see how the commit you're linking to can be the issue, I suspect some undefined behavior because of some sort of ODR violation across shared objects or something like that.

Not sure if this may be related, but https://github.com/llvm/llvm-project/blob/main/mlir/test/lib/Dialect/TestIRDLToCpp/CMakeLists.txt should be using mlir_target_link_libraries instead of LINK_LIBS...

[hhkit]

Using mlir_target_link_libraries is certainly worth a shot but I was unable to reproduce the failures in my environment so I can't verify if it fixes the issue. Is standard procedure to simply attempt to push up a hotfix?

[Moxinilian]

@tuliom could you let us know if this fixes the issue on your side?

@nikic Thanks for the heads up. Do you know where one can read about the rationale behind this set up? The build scripts for MLIR are hard to decipher after the facts!

[tuliom]

@Moxinilian Sure! I'm running a test that takes a couple of hours to run.
I hope to have an update by tomorrow.

[tuliom]

@nikic's suggestion did solve this issue.
I've just created PR #143435 with the patch I tested.