Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ClassicFlang][Driver] Reduce downstream delta #5

Merged
merged 1 commit into from
Nov 3, 2020

Conversation

bryanpkc
Copy link
Collaborator

This commit is motivated by reducing the merge burden by shrinking the
diff between llvm upstream and classic-flang-llvm-project.

Outside of Flang, Fortran code is fed through the Compile phase, and
the appropriate tooling is picked up through ToolChain::SelectTool.

Classic Flang introduced a FortranFrontend, but these days this seems
unnecessary. Fortran can go through the same machinery as everything else.

  • Use the Preprocess phase to preprocess Fortran code. This phase is
    always combined with the Compile phase.

  • Use the Compile phase to lower Fortran code to LLVM IR, and use the
    Backend phase to compile and optimize the IR. These phases are never
    combined.

  • Remove FortranFrontendJobClass.

  • Remove FortranFrontend tool (instead it's just the Flang tool, which
    in Classic Flang mode is Classic Flang).

  • Update tests which inspect the output of the Classic Flang tooling,
    and ensures that the driver does the right thing for various types of
    inputs.

Based on a patch by @peterwaller-arm (peterwaller-arm/classic-flang-llvm-project@10dacac072).

@bryanpkc
Copy link
Collaborator Author

@banach-space Thanks for volunteering to review.

Copy link
Contributor

@banach-space banach-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] @bryanpkc On systems where filenames are case-insensitive (e.g. OS X), classic_flang.F95 and classic_flang.f95 will be problematic (it's treated as one file). Just to be on the safe side, perhaps it's worth to rename them?

@bryanpkc
Copy link
Collaborator Author

Just to be on the safe side, perhaps it's worth to rename them?

@banach-space The use of uppercase file name extensions to indicate the need for preprocessing seems to be a common policy observed by many Fortran compiler vendors, including GCC, PGI, Cray, and Intel. I think we should open an issue to discuss how to accommodate case-insensitive file systems, but that work wouldn't belong in this PR.

@banach-space
Copy link
Contributor

@banach-space The use of uppercase file name extensions to indicate the need for preprocessing seems to be a common policy observed by many Fortran compiler vendors, including GCC, PGI, Cray, and Intel. I think we should open an issue to discuss how to accommodate case-insensitive file systems, but that work wouldn't belong in this PR.

Sorry, I should've been clearer :) I'm not suggesting any extra work nor discussion. I agree that that would be beyond the scope of this patch. I'm definitely not suggesting that we challenge Fortran's file-naming conventions.

I was suggesting that the test files are renamed (which, AFAIK, doesn't violate any of the file-naming conventions):

  • classic_flang.F95 -> classic_flang_not_preprocessed.F95
  • classic_flang.f95 -> classic_flang_preprocessed.f95
  • classic_flang.F -> classic_flang_not_preprocessed.F
  • classic_flang.f -> classic_flang_preprocessed.f

Otherwise, your patch/commit confuses the filesystem on OS X. However, AFAIK, OS X is not officially supported by Flang, so for me this renaming would be a nice to have.

This patch LGTM as is, thanks for working on this!

@gklimowicz
Copy link
Contributor

I was suggesting that the test files are renamed (which, AFAIK, doesn't violate any of the file-naming conventions):

  • classic_flang.F95 -> classic_flang_not_preprocessed.F95
  • classic_flang.f95 -> classic_flang_preprocessed.f95
  • classic_flang.F -> classic_flang_not_preprocessed.F
  • classic_flang.f -> classic_flang_preprocessed.f

Unless I'm letting myself get confused by your request, I think it's the other way around: .F and .F95 invoke preprocessing. .f and .f95 do not invoke the preprocessor.

So wouldn't it be this?

  • classic_flang.F95 -> classic_flang_preprocessed.F95
  • classic_flang.f95 -> classic_flang_not_preprocessed.f95
  • classic_flang.F -> classic_flang_preprocessed.F
  • classic_flang.f -> classic_flang_not_preprocessed.f

@banach-space
Copy link
Contributor

Unless I'm letting myself get confused by your request, I think it's the other way around: .F and .F95 invoke preprocessing. .f and .f95 do not invoke the preprocessor.

So wouldn't it be this?

* `classic_flang.F95` -> `classic_flang_preprocessed.F95`

* `classic_flang.f95` -> `classic_flang_not_preprocessed.f95`

* `classic_flang.F` -> `classic_flang_preprocessed.F`

* `classic_flang.f` -> `classic_flang_not_preprocessed.f`

Good point - does classic_flang_preprocessed.F95 mean that the file:

  • has been preprocessed or,
  • should be preprocessed?
    The naming that I suggested is ambiguous, sorry!

Perhaps class_flang_do_preprocess.F95 and classic_flang_do_not_preprocess.f95 instead? Does that make more sense?

@gklimowicz
Copy link
Contributor

Good point - does classic_flang_preprocessed.F95 mean that the file:

  • has been preprocessed or,
  • should be preprocessed?
    The naming that I suggested is ambiguous, sorry!

should be preproprocessed.

Perhaps class_flang_do_preprocess.F95 and classic_flang_do_not_preprocess.f95 instead? Does that make more sense?

It certainly is the right intent. I just hope I don't have to type them! :-)

This commit is motivated by reducing the merge burden by shrinking the
diff between llvm upstream and classic-flang-llvm-project.

Outside of Flang, Fortran code is fed through the Compile phase, and
the appropriate tooling is picked up through ToolChain::SelectTool.

Classic Flang introduced a FortranFrontend, but these days this seems
unnecessary. Fortran can go through the same machinery as everything else.

* Use the Preprocess phase to preprocess Fortran code. This phase is
  always combined with the Compile phase.

* Use the Compile phase to lower Fortran code to LLVM IR, and use the
  Backend phase to compile and optimize the IR. These phases are never
  combined.

* Remove FortranFrontendJobClass.

* Remove FortranFrontend tool (instead it's just the Flang tool, which
  in Classic Flang mode is Classic Flang).

* Update tests which inspect the output of the Classic Flang tooling,
  and ensures that the driver does the right thing for various types of
  inputs.

Based on a patch from Peter Waller <peter.waller@arm.com>.
@bryanpkc
Copy link
Collaborator Author

I have named the test files differently. I decided to let classic-flang.f95 remain short and simple, as I expect more driver-related tests to be added there.

@bryanpkc
Copy link
Collaborator Author

Copy link
Contributor

@banach-space banach-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this @bryanpkc ! LGTM. From what I can tell, the CI failures are unrelated to the changes in this patch.

Copy link
Contributor

@gklimowicz gklimowicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This builds and passes our tostones on OpenPower and LLVM 10.

@bryanpkc
Copy link
Collaborator Author

@kiranchandramohan @shivaramaarao Could you review this PR?

Copy link
Collaborator

@kiranchandramohan kiranchandramohan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@bryanpkc
Copy link
Collaborator Author

bryanpkc commented Nov 3, 2020

@shivaramaarao Could you review this PR please?

Copy link
Collaborator

@shivaramaarao shivaramaarao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. thanks.

@bryanpkc bryanpkc merged commit 710e16e into flang-compiler:release_100 Nov 3, 2020
michalpasztamobica pushed a commit to michalpasztamobica/classic-flang-llvm-project that referenced this pull request Dec 30, 2020
When `Target::GetEntryPointAddress()` calls `exe_module->GetObjectFile()->GetEntryPointAddress()`, and the returned
`entry_addr` is valid, it can immediately be returned.

However, just before that, an `llvm::Error` value has been setup, but in this case it is not consumed before returning, like is done further below in the function.

In https://bugs.freebsd.org/248745 we got a bug report for this, where a very simple test case aborts and dumps core:

```
* thread #1, name = 'testcase', stop reason = breakpoint 1.1
    frame #0: 0x00000000002018d4 testcase`main(argc=1, argv=0x00007fffffffea18) at testcase.c:3:5
   1	int main(int argc, char *argv[])
   2	{
-> 3	    return 0;
   4	}
(lldb) p argc
Program aborted due to an unhandled Error:
Error value was Success. (Note: Success values must still be checked prior to being destroyed).

Thread 1 received signal SIGABRT, Aborted.
thr_kill () at thr_kill.S:3
3	thr_kill.S: No such file or directory.
(gdb) bt
#0  thr_kill () at thr_kill.S:3
#1  0x00000008049a0004 in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52
#2  0x0000000804916229 in abort () at /usr/src/lib/libc/stdlib/abort.c:67
#3  0x000000000451b5f5 in fatalUncheckedError () at /usr/src/contrib/llvm-project/llvm/lib/Support/Error.cpp:112
#4  0x00000000019cf008 in GetEntryPointAddress () at /usr/src/contrib/llvm-project/llvm/include/llvm/Support/Error.h:267
flang-compiler#5  0x0000000001bccbd8 in ConstructorSetup () at /usr/src/contrib/llvm-project/lldb/source/Target/ThreadPlanCallFunction.cpp:67
flang-compiler#6  0x0000000001bcd2c0 in ThreadPlanCallFunction () at /usr/src/contrib/llvm-project/lldb/source/Target/ThreadPlanCallFunction.cpp:114
flang-compiler#7  0x00000000020076d4 in InferiorCallMmap () at /usr/src/contrib/llvm-project/lldb/source/Plugins/Process/Utility/InferiorCallPOSIX.cpp:97
flang-compiler#8  0x0000000001f4be33 in DoAllocateMemory () at /usr/src/contrib/llvm-project/lldb/source/Plugins/Process/FreeBSD/ProcessFreeBSD.cpp:604
flang-compiler#9  0x0000000001fe51b9 in AllocatePage () at /usr/src/contrib/llvm-project/lldb/source/Target/Memory.cpp:347
flang-compiler#10 0x0000000001fe5385 in AllocateMemory () at /usr/src/contrib/llvm-project/lldb/source/Target/Memory.cpp:383
flang-compiler#11 0x0000000001974da2 in AllocateMemory () at /usr/src/contrib/llvm-project/lldb/source/Target/Process.cpp:2301
flang-compiler#12 CanJIT () at /usr/src/contrib/llvm-project/lldb/source/Target/Process.cpp:2331
flang-compiler#13 0x0000000001a1bf3d in Evaluate () at /usr/src/contrib/llvm-project/lldb/source/Expression/UserExpression.cpp:190
flang-compiler#14 0x00000000019ce7a2 in EvaluateExpression () at /usr/src/contrib/llvm-project/lldb/source/Target/Target.cpp:2372
flang-compiler#15 0x0000000001ad784c in EvaluateExpression () at /usr/src/contrib/llvm-project/lldb/source/Commands/CommandObjectExpression.cpp:414
flang-compiler#16 0x0000000001ad86ae in DoExecute () at /usr/src/contrib/llvm-project/lldb/source/Commands/CommandObjectExpression.cpp:646
flang-compiler#17 0x0000000001a5e3ed in Execute () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandObject.cpp:1003
flang-compiler#18 0x0000000001a6c4a3 in HandleCommand () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandInterpreter.cpp:1762
flang-compiler#19 0x0000000001a6f98c in IOHandlerInputComplete () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandInterpreter.cpp:2760
flang-compiler#20 0x0000000001a90b08 in Run () at /usr/src/contrib/llvm-project/lldb/source/Core/IOHandler.cpp:548
flang-compiler#21 0x00000000019a6c6a in ExecuteIOHandlers () at /usr/src/contrib/llvm-project/lldb/source/Core/Debugger.cpp:903
flang-compiler#22 0x0000000001a70337 in RunCommandInterpreter () at /usr/src/contrib/llvm-project/lldb/source/Interpreter/CommandInterpreter.cpp:2946
flang-compiler#23 0x0000000001d9d812 in RunCommandInterpreter () at /usr/src/contrib/llvm-project/lldb/source/API/SBDebugger.cpp:1169
flang-compiler#24 0x0000000001918be8 in MainLoop () at /usr/src/contrib/llvm-project/lldb/tools/driver/Driver.cpp:675
flang-compiler#25 0x000000000191a114 in main () at /usr/src/contrib/llvm-project/lldb/tools/driver/Driver.cpp:890```

Fix the incorrect error catch by only instantiating an `Error` object if it is necessary.

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D86355

(cherry picked from commit 1ce07cd)
bryanpkc pushed a commit to Huawei-CPLLab/classic-flang-llvm-project that referenced this pull request Mar 12, 2022
Andrei Matei reported a llvm11 core dump for his bpf program
   https://bugs.llvm.org/show_bug.cgi?id=48578
The core dump happens in LiveVariables analysis phase.
  flang-compiler#4 0x00007fce54356bb0 __restore_rt
  flang-compiler#5 0x00007fce4d51785e llvm::LiveVariables::HandleVirtRegUse(unsigned int,
      llvm::MachineBasicBlock*, llvm::MachineInstr&)
  flang-compiler#6 0x00007fce4d519abe llvm::LiveVariables::runOnInstr(llvm::MachineInstr&,
      llvm::SmallVectorImpl<unsigned int>&)
  flang-compiler#7 0x00007fce4d519ec6 llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int)
  flang-compiler#8 0x00007fce4d51a4bf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&)
The bug can be reproduced with llvm12 and latest trunk as well.

Futher analysis shows that there is a bug in BPF peephole
TRUNC elimination optimization, which tries to remove
unnecessary TRUNC operations (a <<= 32; a >>= 32).
Specifically, the compiler did wrong transformation for the
following patterns:
   %1 = LDW ...
   %2 = SLL_ri %1, 32
   %3 = SRL_ri %2, 32
   ... %3 ...
   %4 = SRA_ri %2, 32
   ... %4 ...

The current transformation did not check how many uses of %2
and did transformation like
   %1 = LDW ...
   ... %1 ...
   %4 = SRL_ri %2, 32
   ... %4 ...
and pseudo register %2 is used by not defined and
caused LiveVariables analysis core dump.

To fix the issue, when traversing back from SRL_ri to SLL_ri,
check to ensure SLL_ri has only one use. Otherwise, don't
do transformation.

Differential Revision: https://reviews.llvm.org/D97792

(cherry picked from commit 51cdb78)
bryanpkc pushed a commit that referenced this pull request Apr 20, 2022
Andrei Matei reported a llvm11 core dump for his bpf program
   https://bugs.llvm.org/show_bug.cgi?id=48578
The core dump happens in LiveVariables analysis phase.
  #4 0x00007fce54356bb0 __restore_rt
  #5 0x00007fce4d51785e llvm::LiveVariables::HandleVirtRegUse(unsigned int,
      llvm::MachineBasicBlock*, llvm::MachineInstr&)
  #6 0x00007fce4d519abe llvm::LiveVariables::runOnInstr(llvm::MachineInstr&,
      llvm::SmallVectorImpl<unsigned int>&)
  #7 0x00007fce4d519ec6 llvm::LiveVariables::runOnBlock(llvm::MachineBasicBlock*, unsigned int)
  #8 0x00007fce4d51a4bf llvm::LiveVariables::runOnMachineFunction(llvm::MachineFunction&)
The bug can be reproduced with llvm12 and latest trunk as well.

Futher analysis shows that there is a bug in BPF peephole
TRUNC elimination optimization, which tries to remove
unnecessary TRUNC operations (a <<= 32; a >>= 32).
Specifically, the compiler did wrong transformation for the
following patterns:
   %1 = LDW ...
   %2 = SLL_ri %1, 32
   %3 = SRL_ri %2, 32
   ... %3 ...
   %4 = SRA_ri %2, 32
   ... %4 ...

The current transformation did not check how many uses of %2
and did transformation like
   %1 = LDW ...
   ... %1 ...
   %4 = SRL_ri %2, 32
   ... %4 ...
and pseudo register %2 is used by not defined and
caused LiveVariables analysis core dump.

To fix the issue, when traversing back from SRL_ri to SLL_ri,
check to ensure SLL_ri has only one use. Otherwise, don't
do transformation.

Differential Revision: https://reviews.llvm.org/D97792

(cherry picked from commit 51cdb78)
bryanpkc pushed a commit that referenced this pull request Apr 20, 2022
This patch re-introduces the fix in the commit llvm/llvm-project@66b0cebf7f736 by @yrnkrn

> In DwarfEHPrepare, after all passes are run, RewindFunction may be a dangling
>
> pointer to a dead function. To make sure it's valid, doFinalization nullptrs
> RewindFunction just like the constructor and so it will be found on next run.
>
> llvm-svn: 217737

It seems that the fix was not migrated to `DwarfEHPrepareLegacyPass`.

This patch also updates `llvm/test/CodeGen/X86/dwarf-eh-prepare.ll` to include `-run-twice` to exercise the cleanup. Without this patch `llvm-lit -v llvm/test/CodeGen/X86/dwarf-eh-prepare.ll` fails with

```
-- Testing: 1 tests, 1 workers --
FAIL: LLVM :: CodeGen/X86/dwarf-eh-prepare.ll (1 of 1)
******************** TEST 'LLVM :: CodeGen/X86/dwarf-eh-prepare.ll' FAILED ********************
Script:
--
: 'RUN: at line 1';   /home/arakaki/build/llvm-project/main/bin/opt -mtriple=x86_64-linux-gnu -dwarfehprepare -simplifycfg-require-and-preserve-domtree=1 -run-twice < /home/arakaki/repos/watch/llvm-project/llvm/test/CodeGen/X86/dwarf-eh-prepare.ll -S | /home/arakaki/build/llvm-project/main/bin/FileCheck /home/arakaki/repos/watch/llvm-project/llvm/test/CodeGen/X86/dwarf-eh-prepare.ll
--
Exit Code: 2

Command Output (stderr):
--
Referencing function in another module!
  call void @_Unwind_Resume(i8* %ehptr) #1
; ModuleID = '<stdin>'
void (i8*)* @_Unwind_Resume
; ModuleID = '<stdin>'
in function simple_cleanup_catch
LLVM ERROR: Broken function found, compilation aborted!
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.      Program arguments: /home/arakaki/build/llvm-project/main/bin/opt -mtriple=x86_64-linux-gnu -dwarfehprepare -simplifycfg-require-and-preserve-domtree=1 -run-twice -S
1.      Running pass 'Function Pass Manager' on module '<stdin>'.
2.      Running pass 'Module Verifier' on function '@simple_cleanup_catch'
 #0 0x000056121b570a2c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/Unix/Signals.inc:569:0
 #1 0x000056121b56eb64 llvm::sys::RunSignalHandlers() /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/Signals.cpp:97:0
 #2 0x000056121b56f28e SignalHandler(int) /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/Unix/Signals.inc:397:0
 #3 0x00007fc7e9b22980 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12980)
 #4 0x00007fc7e87d3fb7 raise /build/glibc-S7xCS9/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
 #5 0x00007fc7e87d5921 abort /build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81:0
 #6 0x000056121b4e1386 llvm::raw_svector_ostream::raw_svector_ostream(llvm::SmallVectorImpl<char>&) /home/arakaki/repos/watch/llvm-project/llvm/include/llvm/Support/raw_ostream.h:674:0
 #7 0x000056121b4e1386 llvm::report_fatal_error(llvm::Twine const&, bool) /home/arakaki/repos/watch/llvm-project/llvm/lib/Support/ErrorHandling.cpp:114:0
 #8 0x000056121b4e1528 (/home/arakaki/build/llvm-project/main/bin/opt+0x29e3528)
 #9 0x000056121adfd03f llvm::raw_ostream::operator<<(llvm::StringRef) /home/arakaki/repos/watch/llvm-project/llvm/include/llvm/Support/raw_ostream.h:218:0
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/arakaki/build/llvm-project/main/bin/FileCheck /home/arakaki/repos/watch/llvm-project/llvm/test/CodeGen/X86/dwarf-eh-prepare.ll

--

********************
********************
Failed Tests (1):
  LLVM :: CodeGen/X86/dwarf-eh-prepare.ll

Testing Time: 0.22s
  Failed: 1
```

Reviewed By: loladiro

Differential Revision: https://reviews.llvm.org/D110979

(cherry picked from commit e8806d7)
kaadam pushed a commit to kaadam/classic-flang-llvm-project that referenced this pull request Jan 4, 2023
We experienced some deadlocks when we used multiple threads for logging
using `scan-builds` intercept-build tool when we used multiple threads by
e.g. logging `make -j16`

```
(gdb) bt
#0  0x00007f2bb3aff110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
flang-compiler#1  0x00007f2bb3af70a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
flang-compiler#2  0x00007f2bb3d152e4 in ?? ()
flang-compiler#3  0x00007ffcc5f0cc80 in ?? ()
flang-compiler#4  0x00007f2bb3d2bf5b in ?? () from /lib64/ld-linux-x86-64.so.2
flang-compiler#5  0x00007f2bb3b5da27 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
flang-compiler#6  0x00007f2bb3b5dbe0 in exit () from /lib/x86_64-linux-gnu/libc.so.6
flang-compiler#7  0x00007f2bb3d144ee in ?? ()
flang-compiler#8  0x746e692f706d742f in ?? ()
flang-compiler#9  0x692d747065637265 in ?? ()
flang-compiler#10 0x2f653631326b3034 in ?? ()
flang-compiler#11 0x646d632e35353532 in ?? ()
flang-compiler#12 0x0000000000000000 in ?? ()
```

I think the gcc's exit call caused the injected `libear.so` to be unloaded
by the `ld`, which in turn called the `void on_unload() __attribute__((destructor))`.
That tried to acquire an already locked mutex which was left locked in the
`bear_report_call()` call, that probably encountered some error and
returned early when it forgot to unlock the mutex.

All of these are speculation since from the backtrace I could not verify
if frames 2 and 3 are in fact corresponding to the `libear.so` module.
But I think it's a fairly safe bet.

So, hereby I'm releasing the held mutex on *all paths*, even if some failure
happens.

PS: I would use lock_guards, but it's C.

Reviewed-by: NoQ

Differential Revision: https://reviews.llvm.org/D118439

(cherry picked from commit d919d02)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants