Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA transform dialect reduction tests are flakey. #15892

Closed
benvanik opened this issue Dec 12, 2023 · 5 comments
Closed

CUDA transform dialect reduction tests are flakey. #15892

benvanik opened this issue Dec 12, 2023 · 5 comments
Assignees
@qedawkins
Labels
bug 🐞 Something isn't working codegen Shared code generation infrastructure and dialects

Comments

@benvanik
Copy link
Collaborator

I've seen several of the reduction tests fail at random on the CI (now up to 2-10 per day). They've been disabled. First I noticed a main branch CI failure was around the 10th but may have appeared slightly before then.

Example log (all stacks seem to be in about the same place):

302/362 Test  #765: iree/tests/transform_dialect/cuda/reduction.mlir.test ...............................................***Failed    1.55 sec
-- Testing: 1 tests, 1 workers --
FAIL: IREE :: transform_dialect/cuda/reduction.mlir (1 of 1)
******************** TEST 'IREE :: transform_dialect/cuda/reduction.mlir' FAILED ********************
Exit Code: 2

Command Output (stderr):
--
RUN: at line 21: iree-opt /work/tests/transform_dialect/cuda/reduction.mlir --iree-hal-target-backends=cuda      --iree-abi-transformation-pipeline      --iree-flow-transformation-pipeline       --iree-stream-transformation-pipeline      --iree-hal-configuration-pipeline |  iree-opt --pass-pipeline='builtin.module(hal.executable(hal.executable.variant(iree-codegen-materialize-user-configs, iree-llvmgpu-lower-executable-target)))'      --iree-codegen-llvmgpu-enable-transform-dialect-jit=false      --iree-codegen-transform-dialect-library=/work/tests/transform_dialect/cuda/reduction_codegen_spec.mlir      --iree-codegen-use-transform-dialect-strategy=codegen |  FileCheck /work/tests/transform_dialect/cuda/reduction.mlir --check-prefix=CHECK
+ iree-opt /work/tests/transform_dialect/cuda/reduction.mlir --iree-hal-target-backends=cuda --iree-abi-transformation-pipeline --iree-flow-transformation-pipeline --iree-stream-transformation-pipeline --iree-hal-configuration-pipeline
+ iree-opt '--pass-pipeline=builtin.module(hal.executable(hal.executable.variant(iree-codegen-materialize-user-configs, iree-llvmgpu-lower-executable-target)))' --iree-codegen-llvmgpu-enable-transform-dialect-jit=false --iree-codegen-transform-dialect-library=/work/tests/transform_dialect/cuda/reduction_codegen_spec.mlir --iree-codegen-use-transform-dialect-strategy=codegen
+ FileCheck /work/tests/transform_dialect/cuda/reduction.mlir --check-prefix=CHECK
iree-opt: /work/third_party/llvm-project/llvm/include/llvm/Support/Casting.h:650: decltype(auto) llvm::dyn_cast(const From &) [To = mlir::OpResult, From = mlir::Value]: Assertion `detail::isPresent(Val) && "dyn_cast on a non-existent value"' failed.
Please report issues to https://github.com/openxla/iree/issues and include the crash backtrace.
Stack dump:
0.	Program arguments: iree-opt "--pass-pipeline=builtin.module(hal.executable(hal.executable.variant(iree-codegen-materialize-user-configs, iree-llvmgpu-lower-executable-target)))" --iree-codegen-llvmgpu-enable-transform-dialect-jit=false --iree-codegen-transform-dialect-library=/work/tests/transform_dialect/cuda/reduction_codegen_spec.mlir --iree-codegen-use-transform-dialect-strategy=codegen
 #0 0x00007f9d3bb4790b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /work/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:13
 #1 0x00007f9d3bb45960 llvm::sys::RunSignalHandlers() /work/third_party/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x00007f9d3bb47fef SignalHandler(int) /work/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1
 #3 0x00007f9d433d6420 __restore_rt (/usr/lib/x86_64-linux-gnu/libpthread.so.0+0x14420)
 #4 0x00007f9d34e6f00b raise (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4300b)
 #5 0x00007f9d34e4e859 abort (/usr/lib/x86_64-linux-gnu/libc.so.6+0x22859)
 #6 0x00007f9d34e4e729 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x22729)
 #7 0x00007f9d34e5ffd6 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x33fd6)
 #8 0x00007f9d3bc48924 (/work/full-build-dir/lib/libIREECompiler.so+0x6a48924)
 #9 0x00007f9d3d9b73e8 llvm::SmallVectorBase<unsigned int>::empty() const /work/third_party/llvm-project/llvm/include/llvm/ADT/SmallVector.h:94:46
#10 0x00007f9d3d9b73e8 mlir::DiagnosedSilenceableFailure::isSilenceableFailure() const /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/Utils/DiagnosedSilenceableFailure.h:85:59
#11 0x00007f9d3d9b73e8 mlir::DiagnosedSilenceableFailure::attachNote(std::optional<mlir::Location>) /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/Utils/DiagnosedSilenceableFailure.h:141:5
#12 0x00007f9d3d9b73e8 mlir::ErrorCheckingTrackingListener::notifyPayloadReplacementNotFound(mlir::Operation*, mlir::ValueRange, mlir::DiagnosedSilenceableFailure&&) /work/llvm-external-projects/iree-dialects/lib/Dialect/LinalgTransform/IR/StructuredTransformOpsExt.cpp:367:12
#13 0x00007f9d3e17540a mlir::transform::TransformState::Extension::replacePayloadOp(mlir::Operation*, mlir::Operation*) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp:1042:10
#14 0x00007f9d3e17540a mlir::transform::TrackingListener::notifyOperationReplaced(mlir::Operation*, mlir::ValueRange) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp:1345:11
#15 0x00007f9d3f1f6fac (anonymous namespace)::GreedyPatternRewriteDriver::notifyOperationReplaced(mlir::Operation*, mlir::ValueRange) /work/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:584:1
#16 0x00007f9d3c11b4a1 mlir::Operation::getResults() /work/third_party/llvm-project/mlir/include/mlir/IR/Operation.h:411:12
#17 0x00007f9d3c11b4a1 mlir::RewriterBase::replaceOp(mlir::Operation*, mlir::ValueRange) /work/third_party/llvm-project/mlir/lib/IR/PatternMatch.cpp:277:32
#18 0x00007f9d3e760855 llvm::SmallVectorTemplateCommon<mlir::Value, void>::isSmall() const /work/third_party/llvm-project/llvm/include/llvm/ADT/SmallVector.h:146:39
#19 0x00007f9d3e760855 llvm::SmallVectorImpl<mlir::Value>::~SmallVectorImpl() /work/third_party/llvm-project/llvm/include/llvm/ADT/SmallVector.h:607:16
#20 0x00007f9d3e760855 llvm::SmallVector<mlir::Value, 6u>::~SmallVector() /work/third_party/llvm-project/llvm/include/llvm/ADT/SmallVector.h:1207:3
#21 0x00007f9d3e760855 (anonymous namespace)::WarpOpDeadResult::matchAndRewrite(mlir::vector::WarpExecuteOnLane0Op, mlir::PatternRewriter&) const /work/third_party/llvm-project/mlir/lib/Dialect/Vector/Transforms/VectorDistribute.cpp:984:3
#22 0x00007f9d3f23486e mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>)::$_2::operator()() const /work/third_party/llvm-project/mlir/lib/Rewrite/PatternApplicator.cpp:206:13
#23 0x00007f9d3f2310e4 mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:0:12
#24 0x00007f9d3f1f8b2f (anonymous namespace)::GreedyPatternRewriteDriver::processWorklist() /work/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:0:0
#25 0x00007f9d3f1f55d1 (anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) &&::$_8::operator()() const /work/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:693:11
#26 0x00007f9d3f1f55d1 void llvm::function_ref<void ()>::callback_fn<(anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) &&::$_8>(long) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#27 0x00007f9d3f1f55d1 llvm::function_ref<void ()>::operator()() const /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#28 0x00007f9d3f1f55d1 void mlir::MLIRContext::executeAction<(anonymous namespace)::GreedyPatternRewriteIteration, long&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, long&) /work/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#29 0x00007f9d3f1f55d1 (anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) && /work/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:691:10
#30 0x00007f9d3f1f55d1 mlir::applyPatternsAndFoldGreedily(mlir::Region&, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig, bool*) /work/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:[727](https://github.com/openxla/iree/actions/runs/7184958206/job/19567949093#step:7:728):47
#31 0x00007f9d3d8d2313 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#32 0x00007f9d3d8d2313 mlir::applyPatternsAndFoldGreedily(mlir::Operation*, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig, bool*) /work/third_party/llvm-project/mlir/include/mlir/Transforms/GreedyPatternRewriteDriver.h:124:14
#33 0x00007f9d3d8d2313 mlir::iree_compiler::IREE::transform_dialect::VectorWarpDistributionOp::applyToOne(mlir::transform::TransformRewriter&, mlir::func::FuncOp, mlir::transform::ApplyToEachResultList&, mlir::transform::TransformState&) /work/compiler/src/iree/compiler/Codegen/LLVMGPU/TransformExtensions/LLVMGPUExtensions.cpp:651:11
#34 0x00007f9d3d8fc879 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#35 0x00007f9d3d8fc879 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#36 0x00007f9d3d8fc879 mlir::DiagnosedSilenceableFailure::isDefiniteFailure() const /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/Utils/DiagnosedSilenceableFailure.h:81:12
#37 0x00007f9d3d8fc879 mlir::DiagnosedSilenceableFailure mlir::transform::detail::applyTransformToEach<mlir::iree_compiler::IREE::transform_dialect::VectorWarpDistributionOp, llvm::iterator_range<llvm::filter_iterator_impl<mlir::Operation* const*, mlir::transform::TransformState::getPayloadOps(mlir::Value) const::'lambda'(mlir::Operation*), std::bidirectional_iterator_tag> >&>(mlir::iree_compiler::IREE::transform_dialect::VectorWarpDistributionOp, mlir::transform::TransformRewriter&, llvm::iterator_range<llvm::filter_iterator_impl<mlir::Operation* const*, mlir::transform::TransformState::getPayloadOps(mlir::Value) const::'lambda'(mlir::Operation*), std::bidirectional_iterator_tag> >&, llvm::SmallVectorImpl<mlir::transform::ApplyToEachResultList>&, mlir::transform::TransformState&) /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/IR/TransformInterfaces.h:1477:13
#38 0x00007f9d3d8fc3af mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#39 0x00007f9d3d8fc3af mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#40 0x00007f9d3d8fc3af mlir::DiagnosedSilenceableFailure::isDefiniteFailure() const /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/Utils/DiagnosedSilenceableFailure.h:81:12
#41 0x00007f9d3d8fc3af mlir::transform::TransformEachOpTrait<mlir::iree_compiler::IREE::transform_dialect::VectorWarpDistributionOp>::apply(mlir::transform::TransformRewriter&, mlir::transform::TransformResults&, mlir::transform::TransformState&) /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/IR/TransformInterfaces.h:1551:14
#42 0x00007f9d3d8fc162 mlir::transform::detail::TransformOpInterfaceInterfaceTraits::Model<mlir::iree_compiler::IREE::transform_dialect::VectorWarpDistributionOp>::apply(mlir::transform::detail::TransformOpInterfaceInterfaceTraits::Concept const*, mlir::Operation*, mlir::transform::TransformRewriter&, mlir::transform::TransformResults&, mlir::transform::TransformState&) /work/full-build-dir/llvm-project/tools/mlir/include/mlir/Dialect/Transform/IR/TransformInterfaces.h.inc:442:3
#43 0x00007f9d3e170eff mlir::transform::TransformState::applyTransform(mlir::transform::TransformOpInterface) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp:946:3
#44 0x00007f9d3e1c501d mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#45 0x00007f9d3e1c501d mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#46 0x00007f9d3e1c501d mlir::DiagnosedSilenceableFailure::isDefiniteFailure() const /work/third_party/llvm-project/mlir/include/mlir/Dialect/Transform/Utils/DiagnosedSilenceableFailure.h:81:12
#47 0x00007f9d3e1c501d applySequenceBlock(mlir::Block&, mlir::transform::FailurePropagationMode, mlir::transform::TransformState&, mlir::transform::TransformResults&) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformOps.cpp:1416:16
#48 0x00007f9d3e1ca6d9 mlir::transform::NamedSequenceOp::apply(mlir::transform::TransformRewriter&, mlir::transform::TransformResults&, mlir::transform::TransformState&) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformOps.cpp:0:10
#49 0x00007f9d3e161b63 mlir::transform::detail::TransformOpInterfaceInterfaceTraits::Model<mlir::transform::NamedSequenceOp>::apply(mlir::transform::detail::TransformOpInterfaceInterfaceTraits::Concept const*, mlir::Operation*, mlir::transform::TransformRewriter&, mlir::transform::TransformResults&, mlir::transform::TransformState&) /work/full-build-dir/llvm-project/tools/mlir/include/mlir/Dialect/Transform/IR/TransformInterfaces.h.inc:442:3
#50 0x00007f9d3e170eff mlir::transform::TransformState::applyTransform(mlir::transform::TransformOpInterface) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp:946:3
#51 0x00007f9d3e17baba mlir::transform::applyTransforms(mlir::Operation*, mlir::transform::TransformOpInterface, mlir::RaggedArray<llvm::PointerUnion<mlir::Operation*, mlir::Attribute, mlir::Value> > const&, mlir::transform::TransformOptions const&, bool) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/IR/TransformInterfaces.cpp:1999:42
#52 0x00007f9d3e141a48 mlir::transform::applyTransformNamedSequence(mlir::Operation*, mlir::Operation*, mlir::ModuleOp, mlir::transform::TransformOptions const&) /work/third_party/llvm-project/mlir/lib/Dialect/Transform/Transforms/TransformInterpreterUtils.cpp:203:10
#53 0x00007f9d3d6ecb5e mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#54 0x00007f9d3d6ecb5e mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#55 0x00007f9d3d6ecb5e (anonymous namespace)::TransformDialectInterpreterPass::runOnOperation() /work/compiler/src/iree/compiler/Codegen/Common/TransformDialectInterpreterPass.cpp:65:9
#56 0x00007f9d3bcec5c6 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7::operator()() const /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:503:17
#57 0x00007f9d3bcec5c6 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7>(long) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#58 0x00007f9d3bcec5c6 llvm::function_ref<void ()>::operator()() const /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#59 0x00007f9d3bcec5c6 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) /work/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#60 0x00007f9d3bcec5c6 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:497:21
#61 0x00007f9d3bcecef8 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#62 0x00007f9d3bcecef8 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#63 0x00007f9d3bcecef8 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:569:9
#64 0x00007f9d3bcf1ef1 mlir::LogicalResult llvm::function_ref<mlir::LogicalResult (mlir::OpPassManager&, mlir::Operation*)>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_6>(long, mlir::OpPassManager&, mlir::Operation*) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:5
#65 0x00007f9d3d230409 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#66 0x00007f9d3d230409 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#67 0x00007f9d3d230409 mlir::iree_compiler::(anonymous namespace)::LLVMGPULowerExecutableTargetPass::runOnOperation() /work/compiler/src/iree/compiler/Codegen/LLVMGPU/LLVMGPULowerExecutableTarget.cpp:122:7
#68 0x00007f9d3bcec5c6 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7::operator()() const /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:503:17
#69 0x00007f9d3bcec5c6 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7>(long) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#70 0x00007f9d3bcec5c6 llvm::function_ref<void ()>::operator()() const /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#71 0x00007f9d3bcec5c6 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) /work/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#72 0x00007f9d3bcec5c6 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:497:21
#73 0x00007f9d3bcecef8 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#74 0x00007f9d3bcecef8 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#75 0x00007f9d3bcecef8 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:569:9
#76 0x00007f9d3bcf30ee mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&) const /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:794:5
#77 0x00007f9d3bcee5bb mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#78 0x00007f9d3bcee5bb mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#79 0x00007f9d3bcee5bb mlir::LogicalResult mlir::failableParallelForEach<__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&) /work/third_party/llvm-project/mlir/include/mlir/IR/Threading.h:46:11
#80 0x00007f9d3bcee5bb mlir::LogicalResult mlir::failableParallelForEach<std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> >&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&>(mlir::MLIRContext*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> >&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&) /work/third_party/llvm-project/mlir/include/mlir/IR/Threading.h:92:10
#81 0x00007f9d3bcee5bb mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:799:14
#82 0x00007f9d3bcec[762](https://github.com/openxla/iree/actions/runs/7184958206/job/19567949093#step:7:763) mlir::detail::OpToOpPassAdaptor::runOnOperation(bool) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:690:5
#83 0x00007f9d3bcec762 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7::operator()() const /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:501:20
#84 0x00007f9d3bcec762 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7>(long) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#85 0x00007f9d3bcec762 llvm::function_ref<void ()>::operator()() const /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#86 0x00007f9d3bcec762 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) /work/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#87 0x00007f9d3bcec762 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:497:21
#88 0x00007f9d3bcecef8 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#89 0x00007f9d3bcecef8 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#90 0x00007f9d3bcecef8 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:569:9
#91 0x00007f9d3bcf30ee mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&) const /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:[794](https://github.com/openxla/iree/actions/runs/7184958206/job/19567949093#step:7:795):5
#92 0x00007f9d3bcee5bb mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#93 0x00007f9d3bcee5bb mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#94 0x00007f9d3bcee5bb mlir::LogicalResult mlir::failableParallelForEach<__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&) /work/third_party/llvm-project/mlir/include/mlir/IR/Threading.h:46:11
#95 0x00007f9d3bcee5bb mlir::LogicalResult mlir::failableParallelForEach<std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> >&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&>(mlir::MLIRContext*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> >&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_15&) /work/third_party/llvm-project/mlir/include/mlir/IR/Threading.h:92:10
#96 0x00007f9d3bcee5bb mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:[799](https://github.com/openxla/iree/actions/runs/7184958206/job/19567949093#step:7:800):14
#97 0x00007f9d3bcec762 mlir::detail::OpToOpPassAdaptor::runOnOperation(bool) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:690:5
#98 0x00007f9d3bcec762 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7::operator()() const /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:501:20
#99 0x00007f9d3bcec762 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7>(long) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#100 0x00007f9d3bcec762 llvm::function_ref<void ()>::operator()() const /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#101 0x00007f9d3bcec762 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) /work/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#102 0x00007f9d3bcec762 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:497:21
#103 0x00007f9d3bcef676 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#104 0x00007f9d3bcef676 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#105 0x00007f9d3bcef676 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:569:9
#106 0x00007f9d3bcef676 mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:880:10
#107 0x00007f9d3bcef41f mlir::PassManager::run(mlir::Operation*) /work/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:0:0
#108 0x00007f9d3bce09f3 mlir::LogicalResult::failed() const /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:44:33
#109 0x00007f9d3bce09f3 mlir::failed(mlir::LogicalResult) /work/third_party/llvm-project/mlir/include/mlir/Support/LogicalResult.h:72:58
#110 0x00007f9d3bce09f3 performActions(llvm::raw_ostream&, std::shared_ptr<llvm::SourceMgr> const&, mlir::MLIRContext*, mlir::MlirOptMainConfig const&) /work/third_party/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:384:7
#111 0x00007f9d3bcdfb9b processBuffer(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, mlir::MlirOptMainConfig const&, mlir::DialectRegistry&, llvm::ThreadPool*) /work/third_party/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:441:12
#112 0x00007f9d3bcdfb9b mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&)::$_2::operator()(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::raw_ostream&) const /work/third_party/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:512:12
#113 0x00007f9d3bcdfb9b mlir::LogicalResult llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::raw_ostream&)>::callback_fn<mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&)::$_2>(long, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::raw_ostream&) /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#114 0x00007f9d3bce157d llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::raw_ostream&)>::operator()(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::raw_ostream&) const /work/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#115 0x00007f9d3bce157d mlir::splitAndProcessBuffer(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, llvm::raw_ostream&)>, llvm::raw_ostream&, bool, bool) /work/third_party/llvm-project/mlir/lib/Support/ToolUtilities.cpp:28:12
#116 0x00007f9d3bcdca1a mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer> >, mlir::DialectRegistry&, mlir::MlirOptMainConfig const&) /work/third_party/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:515:10
#117 0x00007f9d3baa9080 ireeOptMainFromCL(int, char**, llvm::StringRef, mlir::DialectRegistry&) /work/compiler/src/iree/compiler/API/Internal/IREEOptToolEntryPoint.cpp:128:14
#118 0x00007f9d3baa9080 ireeOptRunMain /work/compiler/src/iree/compiler/API/Internal/IREEOptToolEntryPoint.cpp:150:14
#119 0x00007f9d34e50083 __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x24083)
#120 0x000000000020102e _start (/work/full-build-dir/tools/iree-opt+0x20102e)
FileCheck error: '<stdin>' is empty.
FileCheck command line:  FileCheck /work/tests/transform_dialect/cuda/reduction.mlir --check-prefix=CHECK

--

********************
********************
Failed Tests (1):
  IREE :: transform_dialect/cuda/reduction.mlir
@benvanik benvanik added bug 🐞 Something isn't working codegen Shared code generation infrastructure and dialects labels Dec 12, 2023
benvanik added a commit that referenced this issue Dec 12, 2023
@benvanik
Disabling CUDA transform dialect reduction tests as they are flakey.
f1f9fda 
Tracked in #15892.
@benvanik benvanik mentioned this issue Dec 12, 2023
Merged
@benvanik
Copy link
Collaborator Author

https://github.com/openxla/iree/actions/runs/7174379621/job/19536206069 may be the first hit dec 11th 2pm
(github log search is weak and I don't think I can see retried things, so may be wrong)

benvanik added a commit that referenced this issue Dec 12, 2023
@benvanik
Disabling CUDA transform dialect reduction tests as they are flakey.
8ba2619 
Tracked in #15892.
benvanik added a commit that referenced this issue Dec 12, 2023
@benvanik
Disabling CUDA transform dialect reduction tests as they are flakey. (#…
818a4a7 
…15893)

Tracked in #15892.
ramiro050 pushed a commit to ramiro050/iree that referenced this issue Dec 19, 2023
@benvanik @ramiro050
Disabling CUDA transform dialect reduction tests as they are flakey. (i…
d0a2b9e 
…ree-org#15893)

Tracked in iree-org#15892.
@ScottTodd
Copy link
Collaborator

Most of the tests in https://github.com/iree-org/iree/tree/main/tests/transform_dialect/cuda are still disabled. Should we delete them? Would they be useful to run on both CUDA and ROCm/HIP? The files themselves don't look specific to CUDA (besides --iree-hal-target-backends=cuda and --device=cuda in lit tests)

@qedawkins
Copy link
Contributor

Yeah lets delete. The only use for them at this point is as a reference for how to write a transform script. I can send a PR out for it quickly now as an EOD thing :P

@qedawkins
Copy link
Contributor

#17858

@qedawkins
Copy link
Contributor

Solved by #17858

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qedawkins qedawkins

Labels
bug 🐞 Something isn't working codegen Shared code generation infrastructure and dialects
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benvanik @ScottTodd @qedawkins