Backport d56ef5e into 14.x (ArithmeticOps and TestPatterns Object files differ between Phase 2 and 3)

[quinnlp]

Running test-release.sh for 14.0.0-rc1 on PowerPC Ubuntu and RHEL I'm seeing differences in these files between Phase 2 and Phase 3:

# Comparing Phase 2 and Phase 3 files
file ArithmeticOps.cpp.o differs between phase 2 and phase 3
file TestPatterns.cpp.o differs between phase 2 and phase 3

[llvmbot]

@llvm/issue-subscribers-backend-powerpc

[quinnlp]

I have used git bisect to find that f3798ad5fa84 is the first commit with the differing object files. In both object files, the only differences are function labels. I've attached files containing the full objdump of each file for Phase2 and Phase3.

Phase2-ArithmeticOps.cpp.o.txt
Phase3-ArithmeticOps.cpp.o.txt
Phase2-TestPatterns.cpp.o.txt
Phase3-TestPatterns.cpp.o.txt

[quinnlp]

Here is the diff between Phase2-ArithmeticOps.cpp.o.txt and Phase3-ArithmeticOps.cpp.o.txt (I have only included lines with the differing function label definitions and not the branches within these functions which are also different but match the label):

...
65238c65238
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_160__mlir_ods_local_type_constraint_ArithmeticCanonicalization0ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_14EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_160__mlir_ods_local_type_constraint_ArithmeticCanonicalization1ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_15EEvlS3_>:
...
72450c72450
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_160__mlir_ods_local_type_constraint_ArithmeticCanonicalization1ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_15EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_160__mlir_ods_local_type_constraint_ArithmeticCanonicalization0ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_14EEvlS3_>:
...

It seems that the only difference is the swapping of the function order between ArithmeticCanonicalization0 and ArithmeticCanonicalization1. 0 is first in Phase2 but 1 is first in Phase3.

[quinnlp]

Here is the diff between Phase2-TestPatterns.cpp.o.txt and Phase3-TestPatterns.cpp.o.txt (I have only included lines with the differing function label definitions and not the branches within these functions which are also different but match the label):

...
15802c15802
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps3ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_5EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps9ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_11EEvlS3_>:
...
21404c21404
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps6ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_8EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps10ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_12EEvlS3_>:
...
22503c22503
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps21ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_22EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps11ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_13EEvlS3_>:
...
24231c24231
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps7ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_9EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps12ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_14EEvlS3_>:
...
24789c24789
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps0ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_2EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps7ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_9EEvlS3_>:
...
28209c28209
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps20ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_21EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps13ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_15EEvlS3_>:
...
29266c29266
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps5ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_7EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps15ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_17EEvlS3_>:
...
32138c32138
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps17ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_19EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps14ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_16EEvlS3_>:
...
34284c34284
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps8ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_10EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps1ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_3EEvlS3_>:
...
35071c35071
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps2ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_4EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps0ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_2EEvlS3_>:
...
36829c36829
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps11ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_13EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps20ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_21EEvlS3_>:
...
38160c38160
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps14ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_16EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps5ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_7EEvlS3_>:
...
39507c39507
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps12ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_14EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps17ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_18EEvlS3_>:
...
40512c40512
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps19ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_20EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps4ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_6EEvlS3_>:
...
41196c41196
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps4ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_6EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps8ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_10EEvlS3_>:
...
42206c42206
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps13ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_15EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps18ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_19EEvlS3_>:
...
47572c47572
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps9ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_11EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps3ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_5EEvlS3_>:
...
48165c48165
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps22ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_23EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps6ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_8EEvlS3_>:
...
49319c49319
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps15ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_17EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps19ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_20EEvlS3_>:
...
50004c50004
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps1ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_3EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_141__mlir_ods_local_type_constraint_TestOps2ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE3$_4EEvlS3_>:
...
50577c50577
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps16ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_18EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps22ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_23EEvlS3_>:
...
51202c51202
< 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps10ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_12EEvlS3_>:
---
> 0000000000000000 <_ZN4llvm12function_refIFvRN4mlir10DiagnosticEEE11callback_fnIZN12_GLOBAL__N_142__mlir_ods_local_type_constraint_TestOps21ERNS1_15PatternRewriterEPNS1_9OperationENS1_9AttributeENS_9StringRefEE4$_22EEvlS3_>:
...

Similar to ArithmeticCanonicalization in the previous file, we see that the TestOps functions are in a different order. Also, Phase2 does not have a function named TestOps18 while Phase3 does not have a function named TestOps16.

[quinnlp]

I've also taken a look at these functions in tools/mlir/lib/Dialect/Arithmetic/IR/ArithmeticCanonicalization.inc and tools/mlir/test/lib/Dialect/Test/TestPatterns.inc respectively. I've attached the full .inc files below. I found that the function definitions within these .inc files have been swapped around between Phase2 and Phase3. For example, the definition of ArithmeticCanonicalization0 in Phase2 is identical to the definition of ArithmeticCanonicalization1 in Phase3 and vice versa. All calls to these functions have also been swapped between Phase2 and Phase3. These swaps match up with the different labels that I showed above for both object files. So, it seems like there may to some non-determinism in naming theArithmeticCanonicalization and TestOps functions created by table gen for mlir.

Phase2-ArithmeticCanonicalization.inc.txt
Phase3-ArithmeticCanonicalization.inc.txt
Phase2-TestPatterns.inc.txt
Phase3-TestPatterns.inc.txt

[llvmbot]

@llvm/issue-subscribers-mlir

[joker-eph]

I don’t quite understand how it can be a bug in MLIR?
Are you sure this is deterministic: something else is going on here

[ChiaHungDuan]

Sorry I'm not familiar with the release process. By doing a quick glance of the test-release.sh, it seems that it builds the target with same configuration three times and expects these three should be identical, right?

The way we generate those functions has been used prior to f3798ad5fa84, so just have the same question as @joker-eph

[joker-eph]

It would be nice to check the build artifacts when test-release.sh fails to confirm if the output of TableGen (the C++ code) is identical or not.
It is possible that the change makes TableGen itself non-deterministic maybe?

If the source code in the build directory is identical, then I don't see how it can be anything else than a clang/llvm issue.

[quinnlp]

@ChiaHungDuan Yes that is correct, it is a bootstrap build so the compiler built by the first compilation is used for the second compilation, and so on.

@joker-eph I'm not really sure what you mean by build artifacts, but the .inc files that TableGen produces are not identical which causes the object files to not be identical.

Before f3798ad5fa84 the include files are always identical between Phase2 and Phase3. But, after f3798ad5fa84 (which makes changes to mlir TableGen) the TableGen for tools/mlir/lib/Dialect/Arithmetic/IR/ArithmeticCanonicalization.inc and tools/mlir/test/lib/Dialect/Test/TestPatterns.inc is non deterministic. You can take a look at the .inc files attached to my previous message.

Here are some more examples of non determinism in the .inc files that I obtained on a separate run:
Run2-Phase2-ArithmeticCanonicalization.inc.txt
Run2-Phase3-ArithmeticCanonicalization.inc.txt
Run2-Phase2-TestPatterns.inc.txt
Run2-Phase3-TestPatterns.inc.txt

[ChiaHungDuan]

@quinnlp , Can you share the configuration/environment of how you run the script? I tried ./llvm/utils/release/test-release.sh -triple powerpc64-unknown-linux-gnu -release 14.0.0 -rc 1 and this is what I got

# Comparing Phase 2 and Phase 3 files                                            
file asan_rtl_x86_64.S.o differs between phase 2 and phase 3
file hwasan_setjmp_aarch64.S.o differs between phase 2 and phase 3
file hwasan_tag_mismatch_aarch64.S.o differs between phase 2 and phase 3
file hwasan_setjmp_aarch64.S.o differs between phase 2 and phase 3
file hwasan_tag_mismatch_aarch64.S.o differs between phase 2 and phase 3
file hwasan_setjmp_aarch64.S.o differs between phase 2 and phase 3
file hwasan_tag_mismatch_aarch64.S.o differs between phase 2 and phase 3
file hwasan_setjmp_aarch64.S.o differs between phase 2 and phase 3
file hwasan_tag_mismatch_aarch64.S.o differs between phase 2 and phase 3
file fuzzer_no_main.o differs between phase 2 and phase 3
file fuzzer_interceptors.o differs between phase 2 and phase 3
file fuzzer.o differs between phase 2 and phase 3
file fuzzer_no_main.o differs between phase 2 and phase 3
file fuzzer_interceptors.o differs between phase 2 and phase 3
file fuzzer.o differs between phase 2 and phase 3
file ProfiledBinary.cpp.o differs between phase 2 and phase 3               

I didn't see the same failure and I think it may be caused by different configuration and environment. Do you have any recommend way to let me reproduce it and verify the fix?