Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flang] Compilation of DATA statement for large COMPLEX arrays needs a lot of memory #63610

Closed
yus3710-fj opened this issue Jun 30, 2023 · 3 comments
Closed

[Flang] Compilation of DATA statement for large COMPLEX arrays needs a lot of memory #63610

yus3710-fj opened this issue Jun 30, 2023 · 3 comments
Assignees
@luporl
Labels
flang:ir

Comments

@yus3710-fj
Copy link
Contributor

This is an issue from Fujitsu testsuite.

Flang-new terminates abnormally when compiling the following program.
When I checked with the top command, %MEM gradually increased to 100%.
It seems that a lot of memory was used.

There is no problem with gfortran compiling on the same machine.

The following are the test program, Flang-new and gfortran compilation result, and the stack trace of Flang-new.

! test.f90
complex(kind=8),dimension(1000)     :: a8
complex(kind=8),dimension(1000,1000)  :: b8
complex(kind=16),dimension(1000)    :: a16
complex(kind=16),dimension(1000,1000) :: b16

data a8/1000*(0.0,0.0)/,a16/1000*(0.0,0.0)/
data b8/1000000*(0.0,0.0)/,b16/1000000*(0.0,0.0)/

end
$ flang-new -v test.f90
flang-new version 17.0.0 (https://github.com/llvm/llvm-project.git 93af6bdcaf4d499e9d4c1da5aa82f92f0acc1dbd)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /path/to/install/bin
Found candidate GCC installation: /usr/lib/gcc/aarch64-redhat-linux/8
Selected GCC installation: /usr/lib/gcc/aarch64-redhat-linux/8
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/path/to/install/bin/flang-new" -fc1 -triple aarch64-unknown-linux-gnu -emit-obj -fcolor-diagnostics -mrelocation-model pic -pic-level 2 -pic-is-pie -target-cpu generic -target-feature +neon -target-feature +v8a -o /tmp/test-6a351b.o -x f95-cpp-input test.f90
flang-new: error: unable to execute command: Killed
flang-new: error: flang frontend command failed due to signal (use -v to see invocation)
flang-new version 17.0.0 (https://github.com/llvm/llvm-project.git 93af6bdcaf4d499e9d4c1da5aa82f92f0acc1dbd)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /path/to/install/bin
flang-new: note: diagnostic msg: 
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
flang-new: note: diagnostic msg: /tmp/test-7d3e84
flang-new: note: diagnostic msg: /tmp/test-7d3e84.sh
flang-new: note: diagnostic msg: 

********************

$ gfortran -v test.f90
Driving: gfortran -v test.f90 -l gfortran -l m -shared-libgcc
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-redhat-linux/8/lto-wrapper
Target: aarch64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --disable-libmpx --enable-gnu-indirect-function --build=aarch64-redhat-linux
Thread model: posix
gcc version 8.3.1 20190507 (Red Hat 8.3.1-4) (GCC) 
 :
$

#0  llvm::rotr<unsigned long, void> (V=2994594782533612773, R=44)
    at /path/to/llvm-project/llvm/include/llvm/ADT/bit.h:378
#1  0x00007ffff0406d07 in llvm::hashing::detail::hash_state::mix_32_bytes (s=0x555573085270 "0\272eXUU",
    a=@0x7fffffffa0a8: 2994594782533612773, b=@0x7fffffffa0b0: 2397334626997450193)
    at /path/to/llvm-project/llvm/include/llvm/ADT/Hashing.h:295
#2  0x00007ffff0406ec3 in llvm::hashing::detail::hash_state::mix (this=0x7fffffffa090, s=0x555573085270 "0\272eXUU")
    at /path/to/llvm-project/llvm/include/llvm/ADT/Hashing.h:310
#3  0x00007ffff0406c30 in llvm::hashing::detail::hash_state::create (s=0x555573085270 "0\272eXUU",
    seed=18397679294719823053) at /path/to/llvm-project/llvm/include/llvm/ADT/Hashing.h:283
#4  0x00007ffff04a1366 in llvm::hashing::detail::hash_combine_range_impl<llvm::Constant* const> (first=0x555573085270,
    last=0x5555730871b0) at /path/to/llvm-project/llvm/include/llvm/ADT/Hashing.h:468
#5  0x00007ffff049c677 in llvm::hash_combine_range<llvm::Constant* const*> (first=0x555573085270, last=0x5555730871b0)
    at /path/to/llvm-project/llvm/include/llvm/ADT/Hashing.h:492
#6  0x00007ffff04a7c86 in llvm::ConstantAggrKeyType<llvm::ConstantArray>::getHash (this=0x7fffffffa1b8)
    at /path/to/llvm-project/llvm/lib/IR/ConstantsContext.h:338
#7  0x00007ffff04a274b in llvm::ConstantUniqueMap<llvm::ConstantArray>::MapInfo::getHashValue (Val={...})
    at /path/to/llvm-project/llvm/lib/IR/ConstantsContext.h:549
#8  0x00007ffff049d449 in llvm::ConstantUniqueMap<llvm::ConstantArray>::getOrCreate (this=0x5555555d8b70,
    Ty=0x555558681880, V=...) at /path/to/llvm-project/llvm/lib/IR/ConstantsContext.h:599
#9  0x00007ffff048abd7 in llvm::ConstantArray::get (Ty=0x555558681880, V=...)
    at /path/to/llvm-project/llvm/lib/IR/Constants.cpp:1238
#10 0x00007ffff0467493 in llvm::ConstantFoldInsertValueInstruction (Agg=0x555a2c3138e0, Val=0x55555852ccd0, Idxs=...)
    at /path/to/llvm-project/llvm/lib/IR/ConstantFold.cpp:802
#11 0x00007ffff062802a in llvm::ConstantFolder::FoldInsertValue (this=0x7fffffffba78, Agg=0x555a2c3138e0,
    Val=0x55555852ccd0, IdxList=...) at /path/to/llvm-project/llvm/include/llvm/IR/ConstantFolder.h:145
#12 0x00007fffe1460a71 in llvm::IRBuilderBase::CreateInsertValue (this=0x7fffffffba00, Agg=0x555a2c3138e0,
    Val=0x55555852ccd0, Idxs=..., Name=...) at /path/to/llvm-project/llvm/include/llvm/IR/IRBuilder.h:2436
#13 0x00007fffe1436283 in convertOperationImpl (opInst=..., builder=..., moduleTranslation=...)
    at /path/to/install/tools/mlir/include/mlir/Dialect/LLVMIR/LLVMConversions.inc:207
#14 0x00007fffe144f82d in (anonymous namespace)::LLVMDialectLLVMIRTranslationInterface::convertOperation (
    this=0x5555555ee360, op=0x555563908280, builder=..., moduleTranslation=...)
    at /path/to/llvm-project/mlir/lib/Target/LLVMIR/Dialect/LLVMIR/LLVMToLLVMIRTranslation.cpp:384
#15 0x00007fffe550ed13 in mlir::LLVM::ModuleTranslation::convertOperation (this=0x7fffffffbcc0, op=..., builder=...)
    at /path/to/llvm-project/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp:612
#16 0x00007fffe550fb60 in mlir::LLVM::ModuleTranslation::convertGlobals (this=0x7fffffffbcc0)
    at /path/to/llvm-project/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp:760
#17 0x00007fffe551435a in mlir::translateModuleToLLVMIR (module=0x5555584d0a10, llvmContext=..., name=...)
    at /path/to/llvm-project/mlir/lib/Target/LLVMIR/ModuleTranslation.cpp:1403
#18 0x00007ffff6c74542 in Fortran::frontend::CodeGenAction::generateLLVMIR (this=0x5555555a1080)
    at /path/to/llvm-project/flang/lib/Frontend/FrontendActions.cpp:721
#19 0x00007ffff6c75b4a in Fortran::frontend::CodeGenAction::executeAction (this=0x5555555a1080)
    at /path/to/llvm-project/flang/lib/Frontend/FrontendActions.cpp:965
#20 0x00007ffff6c6d1d8 in Fortran::frontend::FrontendAction::execute (this=0x5555555a1080)
    at /path/to/llvm-project/flang/lib/Frontend/FrontendAction.cpp:112
#21 0x00007ffff6c46f01 in Fortran::frontend::CompilerInstance::executeAction (this=0x55555558e630, act=...)
    at /path/to/llvm-project/flang/lib/Frontend/CompilerInstance.cpp:167
#22 0x00007ffff7566a35 in Fortran::frontend::executeCompilerInvocation (flang=0x55555558e630)
    at /path/to/llvm-project/flang/lib/FrontendTool/ExecuteCompilerInvocation.cpp:174
#23 0x00005555555631a0 in fc1_main (argv=...,
    argv0=0x7fffffffe30f "/path/to/install/bin/flang-new")
    at /path/to/llvm-project/flang/tools/flang-driver/fc1_main.cpp:67
#24 0x0000555555558031 in executeFC1Tool (argV=...)
    at /path/to/llvm-project/flang/tools/flang-driver/driver.cpp:68
#25 0x0000555555558506 in main (argc=22, argv=0x7fffffffdfe8)
    at /path/to/llvm-project/flang/tools/flang-driver/driver.cpp:111

The following is a similar program but an array size is smaller.

! small_test.f90
complex :: a(100)
data a/100*(0.0,0.0)/
end

That can be compiled normally, but Flang generates large MLIR with LLVM dialect.
I suspect this is the cause of the abnormal termination.

$ flang-new -Xflang -save-temps small_test.f90
$ cat small_test-llvmir.mlir
module attributes {dlti.dl_spec = #dlti.dl_spec<#dlti.dl_entry<i16, dense<[16, 32]> : vector<2xi32>>, #dlti.dl_entry<i64, dense<64> : vector<2xi32>>, #dlti.dl_entry<i128, dense<128> : vector<2xi32>>, #dlti.dl_entry<!llvm.ptr, dense<64> : vector<4xi32>>, #dlti.dl_entry<i1, dense<8> : vector<2xi32>>, #dlti.dl_entry<f16, dense<16> : vector<2xi32>>, #dlti.dl_entry<i32, dense<32> : vector<2xi32>>, #dlti.dl_entry<f64, dense<64> : vector<2xi32>>, #dlti.dl_entry<i8, dense<[8, 32]> : vector<2xi32>>, #dlti.dl_entry<f128, dense<128> : vector<2xi32>>, #dlti.dl_entry<"dlti.stack_alignment", 128 : i32>, #dlti.dl_entry<"dlti.endianness", "little">>, fir.defaultkind = "a1c4d8i4l4r4", fir.kindmap = "", llvm.data_layout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128", llvm.target_triple = "aarch64-unknown-linux-gnu"} {
  llvm.func @_QQmain() {
    llvm.return
  }
  llvm.mlir.global internal @_QFEa() {addr_space = 0 : i32} : !llvm.array<100 x struct<(f32, f32)>> {
    %0 = llvm.mlir.constant(0.000000e+00 : f32) : f32
    %1 = llvm.mlir.undef : !llvm.array<100 x struct<(f32, f32)>>
    %2 = llvm.mlir.undef : !llvm.struct<(f32, f32)>
    %3 = llvm.insertvalue %0, %2[0] : !llvm.struct<(f32, f32)> 
    %4 = llvm.insertvalue %0, %3[1] : !llvm.struct<(f32, f32)> 
    %5 = llvm.insertvalue %4, %1[0] : !llvm.array<100 x struct<(f32, f32)>> 
    %6 = llvm.insertvalue %4, %5[1] : !llvm.array<100 x struct<(f32, f32)>> 
    %7 = llvm.insertvalue %4, %6[2] : !llvm.array<100 x struct<(f32, f32)>> 
    %8 = llvm.insertvalue %4, %7[3] : !llvm.array<100 x struct<(f32, f32)>> 
    %9 = llvm.insertvalue %4, %8[4] : !llvm.array<100 x struct<(f32, f32)>> 
    %10 = llvm.insertvalue %4, %9[5] : !llvm.array<100 x struct<(f32, f32)>> 
    %11 = llvm.insertvalue %4, %10[6] : !llvm.array<100 x struct<(f32, f32)>> 
     :
    %103 = llvm.insertvalue %4, %102[98] : !llvm.array<100 x struct<(f32, f32)>> 
    %104 = llvm.insertvalue %4, %103[99] : !llvm.array<100 x struct<(f32, f32)>> 
    llvm.return %104 : !llvm.array<100 x struct<(f32, f32)>>
  }
  llvm.mlir.global external constant @_QQEnvironmentDefaults() {addr_space = 0 : i32} : !llvm.ptr<struct<(i32, ptr<array<0 x struct<(ptr<i8>, ptr<i8>)>>>)>> {
    %0 = llvm.mlir.null : !llvm.ptr<struct<(i32, ptr<array<0 x struct<(ptr<i8>, ptr<i8>)>>>)>>
    llvm.return %0 : !llvm.ptr<struct<(i32, ptr<array<0 x struct<(ptr<i8>, ptr<i8>)>>>)>>
  }
  llvm.func @llvm.stacksave() -> !llvm.ptr<i8> attributes {sym_visibility = "private"}
  llvm.func @llvm.stackrestore(!llvm.ptr<i8>) attributes {sym_visibility = "private"}
}
@llvmbot
Copy link
Collaborator

llvmbot commented Jun 30, 2023

@llvm/issue-subscribers-flang-ir

@luporl
Copy link
Contributor

luporl commented Jun 30, 2023

This looks similar to #60376. There it was fixed by emitting array constants as MLIR attributes. However, MLIR attributes don't support complex types.

@luporl luporl self-assigned this Jul 20, 2023
@luporl
Copy link
Contributor

luporl commented Jul 21, 2023

It turns out there is actually a way to represent complex array constants with MLIR dense attribute: https://reviews.llvm.org/D155951

luporl added a commit to luporl/llvm-project that referenced this issue Aug 28, 2023
@luporl
[flang] Add support for dense complex constants
334f83a 
Add support for representing complex array constants with MLIR
dense attribute. This improves compile time and greatly reduces
memory usage of programs with large complex array constants.

Fixes llvm#63610

Differential Revision: https://reviews.llvm.org/D155951
@luporl luporl closed this as completed in c8517f1 Aug 30, 2023
@yus3710-fj yus3710-fj mentioned this issue Sep 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@luporl luporl

Labels
flang:ir
Projects
Flang - issues from Fujitsu testsuite
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@EugeneZelenko @luporl @llvmbot @yus3710-fj