-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merge the middle part of the compiler #920
Conversation
This is the set of all patches made to date to implement the middle part
of the flang compiler. These patches contain
- a definition of a Fortran dialect of MLIR (FIR)
- preliminary lowering of the front-end's parse tree, expression, and
symbol table data structures to FIR/MLIR, including the lowering of
Fortran expressions, implicit calls, some intrinsic operations, etc.
- translation of most of the FIR primitive operations to LLVM-IR
- a few optimization passes
- two test tools: bbc for testing lowering from Fortran to FIR and tco
for testing FIR to LLVM-IR
- a set of preliminary tests, including an expression test generator
|
Should I be able to build this? Note that I'm using Windows Subsystem for Linux, which is similar to Ubuntu. I'm using Clang 8.0.0. I get errors when I try to run CMake: -- Performing Test CXX_SUPPORTS_FDATA_SECTIONS - Success -- Found PythonInterp: /usr/bin/python (found version "2.7.17") CMake Error at include/optimizer/CMakeLists.txt:6 (mlir_tablegen): -- Configuring incomplete, errors occurred! |
|
Same errors are there in the CI as well. Is there a change in the build instructions? |
lib/decimal/binary-to-decimal.cc
Outdated
| @@ -312,6 +312,15 @@ void BigRadixFloatingPointNumber<PREC, | |||
| } | |||
| } | |||
|
|
|||
| #define INSTANTIATE_WALDO(X) \ | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's this, here and in decimal-to-binary.cc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are hacks to get these files to build correctly on my Mac.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What was broken? Were you getting unresolved external names?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was a long time ago. But as far as I remember, yes, the builds started failing after this code was introduced with link-time failures.
FWIW, I keep my Mac updated to the "latest and greatest" Apple tools, so there have been occasions where I stumble into issues.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we need this to avoid errors could we name it something more descriptive and add a comment saying why it is here? I don't understand how the name INSTANTIATE_WALDO relates to what this macro does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be a work-around to a compiler bug with template instantiation in a specific environment. Please confirm that the bug persists; if it does, other work-arounds can be tried.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears to have been a transitory issue. This code will be removed.
|
I was able to build using these instructions. |
This merge introduces true dependencies between the F18 project and MLIR, LLVM. In-tree and out-of-tree builds were tested. Varun posted a link to the directions. https://github.com/schweitzpgi/f18/tree/mono#monorepo-now-contains-mlir |
|
@schweitzpgi Does this PR need flang-compiler/f18-llvm-project or does it compile against the master branch of llvm/llvm-project? |
It needs the patches found in flang-compiler/f18-llvm-project on the mono branch. |
|
I'm very excited that Flang may soon generate some code, but I am strongly against this being pushed as-is to F18. This pull request contains one patch comprising 23000 lines of code. This patch is far too big. Contributing FIR in this way will hide all the reasoning behind the changes and makes review incredibly difficult. It will also make bisection impossible for this large part of the compiler. I also question why had Peter create a script to maintain the git history of F18 when it goes into the LLVM monorepo only to have this patch at the top of said history with 23000 LOC in it? If we accept this patch into Flang before the patches in flang-compiler/f18-llvm-project are accepted into the monorepo, then llvm-project/flang will not build with the other components of llvm even following Flang's own out-of-tree build instructions. Given the (very reasonable) scrutiny F18 is under in the LLVM community right now for not looking and smelling like an LLVM project this is going to really hurt our credibility. I am also unsure of how complete the implementation is and of how well tested it is. The commit message describes the tests as 'preliminary' so how well can we expect this code to work? There are 26 tests in this patch, which to me indicates it is not well covered by testing. I think that this PR needs to be broken down into a logical sequence of patches that builds up the FIR capability from scratch. This way, we can review this important code in digestable chunks and ensure that we are testing well the functionality as we build it up. Arm are very happy to conduct these reviews. |
| @@ -8,95 +8,384 @@ | |||
|
|
|||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to get these CMake changes as a separate PR? They seem largely orthogonal to merging the middle-end and would be easier to review separately
include/optimizer/.clang-format
Outdated
| @@ -0,0 +1,2 @@ | |||
| BasedOnStyle: LLVM | |||
| AlwaysBreakTemplateDeclarations: Yes | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is missing a newline at the end
| #include "mlir/Support/LLVM.h" | ||
|
|
||
| namespace mlir { | ||
| class Block; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class is already declared in Block.h
|
|
||
| namespace mlir { | ||
| class Block; | ||
| class DominanceInfo; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class is already declared in Dominance.h
| /// not include blocks where any phi insertion would be dead. | ||
| /// | ||
| /// Note: This set *must* live for the entire lifetime of the IDF calculator. | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: stray newline
| unsigned hashOps; | ||
| if (op->isCommutative()) { | ||
| std::vector<void *> vec; | ||
| for (auto i = op->operand_begin(), e = op->operand_end(); i != e; ++i) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| for (auto i = op->operand_begin(), e = op->operand_end(); i != e; ++i) | |
| for (auto i : op->getOperands()) |
tools/bbc/.clang-format
Outdated
| @@ -0,0 +1,2 @@ | |||
| BasedOnStyle: LLVM | |||
| AlwaysBreakTemplateDeclarations: Yes | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: missing new line at end of file
tools/tco/.clang-format
Outdated
| @@ -0,0 +1,2 @@ | |||
| BasedOnStyle: LLVM | |||
| AlwaysBreakTemplateDeclarations: Yes | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: missing new line at end of file
tools/tco/tco.cpp
Outdated
| llvm::InitLLVM y(argc, argv); | ||
| (void)y; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we're using C++17 we can just mark this variable ``[[maybe_unused]]` for clarity
tools/bbc/bbc.cpp
Outdated
| InitLLVM y(argc, argv); | ||
| (void)y; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we're using C++17 we can just mark this variable [[maybe_unused]] for clarity
|
Just to briefly add to what @RichBarton-Arm said above, I have done my best to review this but I don't feel that I have been close to as thorough as such a significant part of the codebase deserves. It would be much easier to review to a good standard if this were split into multiple PRs. |
- add newlines to .clang-format - remove old school (void) casts and replace with new school [[maybe_unused]] - remove unneeded headers - mlir-tablegen now supports #ifndef - remove a newline - remove a few lines - make code more succinct
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for making these changes available. I have a few comments.
Are there tests for lowering to LLVM-IR dialect?
The LLVM-IR in the first comment/description of this PR refers to the dialect right?
As others have said it would be easier to have a look if changes are organized as a sequence of commits.
|
|
||
| if (!owningRef) { | ||
| errs() << "Error can't load file " << ClInput << '\n'; | ||
| return 2; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can these return error values be values from an enum or preprocessor vals?
I see that similar values are used in the mlir toy examples.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a rough draft of a test tool. It needs more work to make it useful for lit, etc.
lib/optimizer/.clang-format
Outdated
| @@ -0,0 +1,2 @@ | |||
| BasedOnStyle: LLVM | |||
| AlwaysBreakTemplateDeclarations: Yes | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Newline
| using Op::Op; | ||
| using Op::print; | ||
|
|
||
| static llvm::StringRef getOperationName() { return "fir.global"; } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Already declared using llvm::StringRef above. Also in several places below for both StringRef and ArrayRef.
| namespace Br = Fortran::lower; | ||
| namespace Co = Fortran::common; | ||
| namespace Ev = Fortran::evaluate; | ||
| namespace L = llvm; | ||
| namespace M = mlir; | ||
| namespace Pa = Fortran::parser; | ||
| namespace Se = Fortran::semantics; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can these namespace abbreviations be in a single file and included wherever necessary?
| #define TODO() \ | ||
| assert(false); \ | ||
| return {} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There seems to be three TODOs with slightly different meanings. Can a prefix or suffix be added to signify the difference?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The TODO macros are ephemeral. I'm not sure there is great value in standardizing them.
| auto a = operands[0]; | ||
| auto loc = boxaddr.getLoc(); | ||
| auto ty = convertType(boxaddr.getType()); | ||
| if (auto argty = boxaddr.val()->getType().dyn_cast<BoxType>()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
camelCase?
| L::ArrayRef<OperandTy> destOperands, | ||
| M::ConversionPatternRewriter &rewriter) const override { | ||
| selectMatchAndRewrite<fir::SelectOp>(lowering, op, operands, destinations, | ||
| destOpera |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FIXME or TODO?
Could you add some details?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We (obviously) haven't spent enough time in documenting our interfaces in the project. It is on the list of things to be done.
lib/optimizer/FIROps.cpp
Outdated
|
|
||
| // LoopOp | ||
|
|
||
| void LoopOp::build(mlir::Builder *builder, mlir::OperationState &result, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mlir:: -> M:: for consistency.
| ty.dyn_cast<ReferenceType>() || ty.dyn_cast<TypeDescType>()); | ||
| } | ||
|
|
||
| bool verifySameLists(L::ArrayRef<RecordType::TypePair> a1, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't the == on ArrayRef work?
https://github.com/llvm/llvm-project/blob/6c2151bf4c829958891e65a4cc396daa6d308eb0/llvm/include/llvm/ADT/ArrayRef.h#L533
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's fine for how this test is written. We may need a more sophisticated algorithm here. It's not entirely clear at the moment.
lib/optimizer/StdConverter.cpp
Outdated
| L::ArrayRef<M::Block *> destinations, | ||
| L::ArrayRef<OperandTy> destOperands, | ||
| M::ConversionPatternRewriter &rewriter) const override { | ||
| auto selecttype = M::cast<SelectTypeOp>(op); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
camelCase?
Thanks for taking a look at the project, Kiran.
There are some. However, the project is very much in active development and still being written.
Yes. FIR is lowered to the LLVM-IR dialect of MLIR. And, not exactly. The meaning of the statement was that (most) patches have already been upstreamed to the MLIR project so that the LLVM-IR dialect will generate a file that contains LLVM IR. (I'm working on the last one now.) |
I was able to build this PR out-of-tree with a fresh llvm from flang-compiler/f18-llvm-project. I was not able to build llvm with shared libraries owing to compilation problems when building mlir. I also disabled ZLIB when building llvm because I am unable to share the resulting llvm install with zlib between redhat and ubuntu systems. |
The commits are available in https://github.com/schweitzpgi/f18 By the slice or by the pie, it's all the same pizza. |
+1 this patch should not be merged until it builds against the monorepo. |
Thanks, Varun. I tried following those instructions. I'm a newbie to all of this, and there were things that puzzled or blocked me. The title says "Monorepo now contains MLIR", but I don't know what that means or why it's important. Also, there are two options, an "in-tree" build, and an "out-of-tree" build. I'm not sure what the differences are or why I should choose one over the other. I guessed that I wanted an "in-tree" build. But when I tried both of the "git clone" commands, I got messages saying "Permission denied". |
include/flang/Version.h
Outdated
| /// Retrieves the repository revision number (or identifier) from which | ||
| /// LLVM was built. | ||
| /// | ||
| /// If Clang and LLVM are in the same repository, this returns the same |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it Clang or Flang. Why would we need Clang here?
include/flang/Version.h
Outdated
| std::string getLLVMRevision(); | ||
|
|
||
| /// Retrieves the full repository version that is an amalgamation of | ||
| /// the information in getClangRepositoryPath() and getClangRevision(). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess the comments where I see Clang should be Flang now. Correct?
include/flang/Version.inc.in
Outdated
| @@ -0,0 +1,5 @@ | |||
| #define FLANG_VERSION @CLANG_VERSION@ | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why Flang is the same as Clang?
| if (e.subs) { | ||
| return false; | ||
| } | ||
| switch (e.cfg) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This does not need a default case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not when all the values of the enum are explicitly covered by the cases.
| (void)rc; // for release build | ||
| return; | ||
| } | ||
| for (auto *s : *iter->second) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should *iter be tested for null first?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be incorrect to add a nullptr to the map.
| } | ||
|
|
||
| #undef TODO | ||
| #define TODO() \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is there a reason for the back-slash ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a C preprocessor line continuation. TODO() expands to the following two lines.
tools/tco/tco.cpp
Outdated
| #include "llvm/Support/SourceMgr.h" | ||
| #include "llvm/Support/raw_ostream.h" | ||
|
|
||
| using namespace llvm; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've notice that the first files you use llvm::, and by the end namespace started to be used.
Is there a reason for that? If not, why not use using namespace llvm for all?
tools/tco/CMakeLists.txt
Outdated
| @@ -0,0 +1,43 @@ | |||
| #===-- tools/tco/CMakeLists.txt --------------------------------------------===# | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a need for header here? I may be wrong, but I believe CMakeLists.txt does not need the header.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Appears the header rewrites got overly exuberant.
| } | ||
|
|
||
| int main(int argc, char **argv) { | ||
| [[maybe_unused]] llvm::InitLLVM y(argc, argv); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need that when we have namespace llvm?
|
|
||
| std::optional<Type> Operation::GetResultTypeForArguments( | ||
| Type lhs, Type rhs) const { | ||
| // See Fortran 2018 table 10.2 and section 10.1.9.3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe in the future would be nice to write the reason and reference the standard for more info.
Instead of only adding the reference to the standard
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to build (with 4 compiler warnings) this PR using the build instructions provided in the page mentioned by @vjayathirtha-nv. If we are to push this patch before the llvm submission then the Readme file should be updated with the new build instructions.
Was not able to find a way to run the existing tests or the new tests that come as part of the PR. ninja test, ninja check, ninja check-all did not work. There is no test directory hence the old way to run tests ($HOME/cmake-3.11.0-rc4/build/bin/ctest --force-new-ctest-process) also did not work. Is this expected only to work after the lit changes?
I guess some of the changes for FIR will actually be moving to the mlir directories. What is the plan for this? Is this move planned only after the project is mature?
A few more comments inline.
| @@ -0,0 +1,136 @@ | |||
| //===-- include/fir/Attribute.h ---------------------------------*- C++ -*-===// | |||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mismatch with real file path. I guess this comment applies to all headers in this directory. Probably a renaming happened sometime.
| // | ||
| //===----------------------------------------------------------------------===// | ||
|
|
||
| #ifndef OPTIMIZER_FIRATTRIBUTE_H |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should the filename be FirAttribute.h or should this define be ATTRIBUTE_H?
| // Should we also auto-detect that the called function is pure if its | ||
| // arguments are not references? For now, rely on a "pure" attribute. | ||
| if (auto call = dyn_cast<fir::CallOp>(op)) | ||
| return bool(call.getAttr("pure")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the creation of a bool necessary here? (Note: it is not created in the function above)
| FIR_TYPEDESC, | ||
| }; | ||
|
|
||
| bool isa_fir_type(mlir::Type); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Name for the parameter for consistency with the ones below?
| } | ||
| std::int64_t result; | ||
| uniq.substr(init, i).getAsInteger(10, result); | ||
| return result; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compilation warning here for result maybe-uninitialized.
| KindTy kind; | ||
| if (parseCode(code, srcPtr) || parseInt(kind, srcPtr)) | ||
| return badMapString(srcPtr); | ||
| if (code == 'a' || code == 'i' || code == 'l') { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compilation warning here for code maybe-uninitialized.
| LLVMTypeID id; | ||
| if (parseColon(srcPtr) || parseTypeID(id, srcPtr)) | ||
| return badMapString(srcPtr); | ||
| floatMap[code][kind] = id; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compilation warning here for id maybe-uninitialized.
| }; | ||
|
|
||
| /// return true if all `Value`s in `operands` are `ConstantOp`s | ||
| bool allConstants(OperandTy operands) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
compilation warning here for unused function. (I guess it is not used in Release builds)
Answer some comments from flang-compiler#920: - Use std::unique_ptr for IntrinsicLibrary::Implementation instead of a simple "owning pointer" comment. - Remove two "#if 0" dead code pieces
:) As I understand at lof of the changes in this PR was already reviewed in two separate PRs (links given below). If the new changes were separate commits then we only need to look at the new changes. Also, this new PR which is a single commit gives the impression that this a single piece of unreviewed code to a person who has not been following the f18 project. |
|
Here's some performance information about my build. Note that I'm using a Windows laptop with 6 cores and 32 GB of RAM. I'm running Windows 10 using Windows Subsystem for Linux with Ubuntu installed. I used version 8.3 of the GNU compiler. I was also hard-wired into NVIDIA's network at the office. It took me between 9 and 10 minutes to clone the LLVM repository. The actual build (using Ninja) took 57 minutes during which time all of the processors on my machine were showing close to full utilization. Also, a rebuild after changing one file takes me 2.5 minutes. For my old build of just f18, this took about 1 minute. |
Conflicts: include/flang/CMakeLists.txt tools/tco/tco.cpp
|
This merge will not be taking place at this time. |
The Middle End
Includes contributions from @jeanPerier, @vdonaldson, @gklimowicz and @sscalpone