[mlir] gpu-to-llvm Pass crashed with error message "Assertion failed: (!replacements.count(op) && "operation was already replaced")"

[Colloportus0]

MLIR built at commit 12ebfca
Reproduced with:
mlir-opt --gpu-to-llvm temp.mlir

temp.mlir:

module {
  func.func @func1(){
    %c1 = arith.constant 1 : index
    %alloc_6 = memref.alloc() : memref<2x3xf32>
    memref.alloca_scope  {
      %41 = memref.generic_atomic_rmw %alloc_6[%c1, %c1] : memref<2x3xf32> {
      ^bb0(%arg0: f32):
        memref.atomic_yield %arg0 : f32
      }
    }
    return
  }
}

trace:

Assertion failed: (!replacements.count(op) && "operation was already replaced"), function notifyOpReplaced, file DialectConversion.cpp, line 1404.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: mlir-opt -gpu-to-llvm temp.mlir
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  mlir-opt                 0x00000001055fff5c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 72
1  mlir-opt                 0x0000000105600478 PrintStackTraceSignalHandler(void*) + 28
2  mlir-opt                 0x00000001055fe588 llvm::sys::RunSignalHandlers() + 148
3  mlir-opt                 0x0000000105601d48 SignalHandler(int) + 252
4  libsystem_platform.dylib 0x000000019fc254c4 _sigtramp + 56
5  libsystem_pthread.dylib  0x000000019fc0dee0 pthread_kill + 288
6  libsystem_c.dylib        0x000000019fb48340 abort + 168
7  libsystem_c.dylib        0x000000019fb47754 err + 0
8  mlir-opt                 0x0000000109346380 mlir::detail::ConversionPatternRewriterImpl::notifyOpReplaced(mlir::Operation*, mlir::ValueRange) + 208
9  mlir-opt                 0x0000000109347758 mlir::ConversionPatternRewriter::eraseOp(mlir::Operation*) + 300
10 mlir-opt                 0x000000010834561c (anonymous namespace)::GenericAtomicRMWOpLowering::moveOpsRange(mlir::ValueRange, mlir::ValueRange, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void>, false, false>, llvm::ilist_iterator<llvm::ilist_detail::node_options<mlir::Operation, true, false, void>, false, false>, mlir::ConversionPatternRewriter&) const + 304
11 mlir-opt                 0x0000000108344630 (anonymous namespace)::GenericAtomicRMWOpLowering::matchAndRewrite(mlir::memref::GenericAtomicRMWOp, mlir::memref::GenericAtomicRMWOpAdaptor, mlir::ConversionPatternRewriter&) const + 1600
12 mlir-opt                 0x0000000108343f28 mlir::ConvertOpToLLVMPattern<mlir::memref::GenericAtomicRMWOp>::matchAndRewrite(mlir::Operation*, llvm::ArrayRef<mlir::Value>, mlir::ConversionPatternRewriter&) const + 192
13 mlir-opt                 0x0000000109348b50 mlir::ConversionPattern::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&) const + 368
14 mlir-opt                 0x000000010a06cbb8 mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) + 1356
15 mlir-opt                 0x0000000109369a1c (anonymous namespace)::OperationLegalizer::legalizeWithPattern(mlir::Operation*, mlir::ConversionPatternRewriter&) + 328
16 mlir-opt                 0x0000000109369170 (anonymous namespace)::OperationLegalizer::legalize(mlir::Operation*, mlir::ConversionPatternRewriter&) + 996
17 mlir-opt                 0x0000000109368760 (anonymous namespace)::OperationConverter::convert(mlir::ConversionPatternRewriter&, mlir::Operation*) + 64
18 mlir-opt                 0x000000010934cc60 (anonymous namespace)::OperationConverter::convertOperations(llvm::ArrayRef<mlir::Operation*>, llvm::function_ref<void (mlir::Diagnostic&)>) + 568
19 mlir-opt                 0x000000010934c990 mlir::applyPartialConversion(llvm::ArrayRef<mlir::Operation*>, mlir::ConversionTarget&, mlir::FrozenRewritePatternSet const&, llvm::DenseSet<mlir::Operation*, llvm::DenseMapInfo<mlir::Operation*, void>>*) + 124
20 mlir-opt                 0x000000010934ce84 mlir::applyPartialConversion(mlir::Operation*, mlir::ConversionTarget&, mlir::FrozenRewritePatternSet const&, llvm::DenseSet<mlir::Operation*, llvm::DenseMapInfo<mlir::Operation*, void>>*) + 72
21 mlir-opt                 0x00000001080bc8d4 (anonymous namespace)::GpuToLLVMConversionPass::runOnOperation() + 384
22 mlir-opt                 0x000000010920c524 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) + 512
23 mlir-opt                 0x000000010920cbf4 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) + 364
24 mlir-opt                 0x000000010920ee38 mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) + 108
25 mlir-opt                 0x000000010920ec10 mlir::PassManager::run(mlir::Operation*) + 864
26 mlir-opt                 0x00000001091f4010 performActions(llvm::raw_ostream&, bool, bool, llvm::SourceMgr&, mlir::MLIRContext*, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, bool, bool) + 560
27 mlir-opt                 0x00000001091f3ba4 processBuffer(llvm::raw_ostream&, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, bool, bool, bool, bool, bool, bool, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, llvm::ThreadPool*) + 496
28 mlir-opt                 0x00000001091f396c mlir::MlirOptMain(llvm::raw_ostream&, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool)::$_0::operator()(std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) const + 204
29 mlir-opt                 0x00000001091f3880 mlir::LogicalResult llvm::function_ref<mlir::LogicalResult (std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>::callback_fn<mlir::MlirOptMain(llvm::raw_ostream&, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool)::$_0>(long, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) + 80
30 mlir-opt                 0x00000001093fea44 llvm::function_ref<mlir::LogicalResult (std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>::operator()(std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) const + 96
31 mlir-opt                 0x00000001093fe528 mlir::splitAndProcessBuffer(std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>, llvm::raw_ostream&, bool, bool) + 128
32 mlir-opt                 0x00000001091f12c0 mlir::MlirOptMain(llvm::raw_ostream&, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool) + 320
33 mlir-opt                 0x00000001091f14c8 mlir::MlirOptMain(llvm::raw_ostream&, std::__1::unique_ptr<llvm::MemoryBuffer, std::__1::default_delete<llvm::MemoryBuffer>>, mlir::PassPipelineCLParser const&, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool, bool) + 296
34 mlir-opt                 0x00000001091f208c mlir::MlirOptMain(int, char**, llvm::StringRef, mlir::DialectRegistry&, bool) + 2912
35 mlir-opt                 0x0000000104e21fa0 main + 148
36 dyld                     0x0000000122ab5088 start + 516
zsh: abort      mlir-opt -gpu-to-llvm temp.mlir

[llvmbot]

@llvm/issue-subscribers-mlir

[joker-eph]

@ThomasRaoux may know who can look into this?

[ThomasRaoux]

This is not gpu specific, looks like a bug in the lowering of generic_atomic_rmw to llvm. This code was added by @pifon2a who is probably the best person to look at it.

[Colloportus0]

Hi @pifon2a, I want to kindly bring this to your attention in case you forgot to look at this issue.

[pifon2a]

I did not even see it. Thanks for pinging me. I ll try to fix it on Friday or Monday.

[pifon2a]

Here is the fix for the main issue here https://reviews.llvm.org/D145901, but it won't fix the lowering completely, because cmpxchg does not accept floats. @ftynse, is there any workaround to make it work? I found this related discussion https://lists.llvm.org/pipermail/llvm-dev/2020-August/144314.html

[ftynse]

Can't the lowering go to llvm.atomic_rmw?

[Colloportus0]

Now this pass still doesn't work, as @pifon2a said

cmpxchg does not accept floats.

error: 'llvm.cmpxchg' op operand #1 must be integer or LLVM pointer type, but got 'f32'
      %41 = memref.generic_atomic_rmw %alloc_6[%c1, %c1] : memref<2x3xf32> {

[joker-eph]

This isn't crashing anymore? Can we close this and file another issue for the incomplete support in the lowering?