-
Notifications
You must be signed in to change notification settings - Fork 12k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[flang][Frontend] Implement printing defined macros via -dM #87627
Conversation
@llvm/pr-subscribers-clang-driver @llvm/pr-subscribers-clang Author: Krzysztof Parzyszek (kparzysz) ChangesThis should work the same way as in clang. Full diff: https://github.com/llvm/llvm-project/pull/87627.diff 14 Files Affected:
diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.td
index c3e90a70925b78..b1ed29cb1cbc44 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -1446,7 +1446,7 @@ def dD : Flag<["-"], "dD">, Group<d_Group>, Visibility<[ClangOption, CC1Option]>
def dI : Flag<["-"], "dI">, Group<d_Group>, Visibility<[ClangOption, CC1Option]>,
HelpText<"Print include directives in -E mode in addition to normal output">,
MarshallingInfoFlag<PreprocessorOutputOpts<"ShowIncludeDirectives">>;
-def dM : Flag<["-"], "dM">, Group<d_Group>, Visibility<[ClangOption, CC1Option]>,
+def dM : Flag<["-"], "dM">, Group<d_Group>, Visibility<[ClangOption, CC1Option, FlangOption, FC1Option]>,
HelpText<"Print macro definitions in -E mode instead of normal output">;
def dead__strip : Flag<["-"], "dead_strip">;
def dependency_file : Separate<["-"], "dependency-file">,
diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp
index 70daa699e3a949..bfd07addfca811 100644
--- a/clang/lib/Driver/ToolChains/Flang.cpp
+++ b/clang/lib/Driver/ToolChains/Flang.cpp
@@ -688,7 +688,10 @@ void Flang::ConstructJob(Compilation &C, const JobAction &JA,
CmdArgs.push_back(Args.MakeArgString(TripleStr));
if (isa<PreprocessJobAction>(JA)) {
- CmdArgs.push_back("-E");
+ CmdArgs.push_back("-E");
+ if (Args.getLastArg(options::OPT_dM)) {
+ CmdArgs.push_back("-dM");
+ }
} else if (isa<CompileJobAction>(JA) || isa<BackendJobAction>(JA)) {
if (JA.getType() == types::TY_Nothing) {
CmdArgs.push_back("-fsyntax-only");
diff --git a/flang/include/flang/Frontend/PreprocessorOptions.h b/flang/include/flang/Frontend/PreprocessorOptions.h
index b2e9ac0e963b73..13a91ee9a184f8 100644
--- a/flang/include/flang/Frontend/PreprocessorOptions.h
+++ b/flang/include/flang/Frontend/PreprocessorOptions.h
@@ -56,6 +56,9 @@ struct PreprocessorOptions {
// -fno-reformat: Emit cooked character stream as -E output
bool noReformat{false};
+ // -dM: Show macro definitions with -dM -E
+ bool showMacros{false};
+
void addMacroDef(llvm::StringRef name) {
macros.emplace_back(std::string(name), false);
}
diff --git a/flang/include/flang/Parser/parsing.h b/flang/include/flang/Parser/parsing.h
index e80d8f724ac8f4..14891c44dacafd 100644
--- a/flang/include/flang/Parser/parsing.h
+++ b/flang/include/flang/Parser/parsing.h
@@ -16,6 +16,7 @@
#include "provenance.h"
#include "flang/Common/Fortran-features.h"
#include "llvm/Support/raw_ostream.h"
+#include <memory>
#include <optional>
#include <string>
#include <utility>
@@ -23,6 +24,8 @@
namespace Fortran::parser {
+class Preprocessor;
+
struct Options {
Options() {}
@@ -59,6 +62,7 @@ class Parsing {
const SourceFile *Prescan(const std::string &path, Options);
void EmitPreprocessedSource(
llvm::raw_ostream &, bool lineDirectives = true) const;
+ void EmitPreprocessorMacros(llvm::raw_ostream &) const;
void DumpCookedChars(llvm::raw_ostream &) const;
void DumpProvenance(llvm::raw_ostream &) const;
void DumpParsingLog(llvm::raw_ostream &) const;
@@ -83,6 +87,7 @@ class Parsing {
const char *finalRestingPlace_{nullptr};
std::optional<Program> parseTree_;
ParsingLog log_;
+ std::unique_ptr<Preprocessor> preprocessor_;
};
} // namespace Fortran::parser
#endif // FORTRAN_PARSER_PARSING_H_
diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp
index c830c7af2462c9..8ce6ab7baf4812 100644
--- a/flang/lib/Frontend/CompilerInvocation.cpp
+++ b/flang/lib/Frontend/CompilerInvocation.cpp
@@ -772,6 +772,7 @@ static void parsePreprocessorArgs(Fortran::frontend::PreprocessorOptions &opts,
opts.noReformat = args.hasArg(clang::driver::options::OPT_fno_reformat);
opts.noLineDirectives = args.hasArg(clang::driver::options::OPT_P);
+ opts.showMacros = args.hasArg(clang::driver::options::OPT_dM);
}
/// Parses all semantic related arguments and populates the variables
diff --git a/flang/lib/Frontend/FrontendActions.cpp b/flang/lib/Frontend/FrontendActions.cpp
index 849b3c8e4dc027..8f251997ed401b 100644
--- a/flang/lib/Frontend/FrontendActions.cpp
+++ b/flang/lib/Frontend/FrontendActions.cpp
@@ -399,7 +399,9 @@ void PrintPreprocessedAction::executeAction() {
// Format or dump the prescanner's output
CompilerInstance &ci = this->getInstance();
- if (ci.getInvocation().getPreprocessorOpts().noReformat) {
+ if (ci.getInvocation().getPreprocessorOpts().showMacros) {
+ ci.getParsing().EmitPreprocessorMacros(outForPP);
+ } else if (ci.getInvocation().getPreprocessorOpts().noReformat) {
ci.getParsing().DumpCookedChars(outForPP);
} else {
ci.getParsing().EmitPreprocessedSource(
diff --git a/flang/lib/Parser/parsing.cpp b/flang/lib/Parser/parsing.cpp
index a55d33bf6b91d6..ec008be1fcea9d 100644
--- a/flang/lib/Parser/parsing.cpp
+++ b/flang/lib/Parser/parsing.cpp
@@ -60,20 +60,20 @@ const SourceFile *Parsing::Prescan(const std::string &path, Options options) {
}
}
- Preprocessor preprocessor{allSources};
+ preprocessor_ = std::make_unique<Preprocessor>(allSources);
if (!options.predefinitions.empty()) {
- preprocessor.DefineStandardMacros();
+ preprocessor_->DefineStandardMacros();
for (const auto &predef : options.predefinitions) {
if (predef.second) {
- preprocessor.Define(predef.first, *predef.second);
+ preprocessor_->Define(predef.first, *predef.second);
} else {
- preprocessor.Undefine(predef.first);
+ preprocessor_->Undefine(predef.first);
}
}
}
currentCooked_ = &allCooked_.NewCookedSource();
Prescanner prescanner{
- messages_, *currentCooked_, preprocessor, options.features};
+ messages_, *currentCooked_, *preprocessor_, options.features};
prescanner.set_fixedForm(options.isFixedForm)
.set_fixedFormColumnLimit(options.fixedFormColumns)
.AddCompilerDirectiveSentinel("dir$");
@@ -87,7 +87,7 @@ const SourceFile *Parsing::Prescan(const std::string &path, Options options) {
if (options.features.IsEnabled(LanguageFeature::CUDA)) {
prescanner.AddCompilerDirectiveSentinel("$cuf");
prescanner.AddCompilerDirectiveSentinel("@cuf");
- preprocessor.Define("_CUDA", "1");
+ preprocessor_->Define("_CUDA", "1");
}
ProvenanceRange range{allSources.AddIncludedFile(
*sourceFile, ProvenanceRange{}, options.isModuleFile)};
@@ -107,6 +107,12 @@ const SourceFile *Parsing::Prescan(const std::string &path, Options options) {
return sourceFile;
}
+void Parsing::EmitPreprocessorMacros(llvm::raw_ostream &out) const {
+ if (preprocessor_) {
+ preprocessor_->PrintMacros(out);
+ }
+}
+
void Parsing::EmitPreprocessedSource(
llvm::raw_ostream &out, bool lineDirectives) const {
const std::string *sourcePath{nullptr};
diff --git a/flang/lib/Parser/preprocessor.cpp b/flang/lib/Parser/preprocessor.cpp
index 515b8f62daf9ad..cea64a7f10a820 100644
--- a/flang/lib/Parser/preprocessor.cpp
+++ b/flang/lib/Parser/preprocessor.cpp
@@ -11,6 +11,9 @@
#include "flang/Common/idioms.h"
#include "flang/Parser/characters.h"
#include "flang/Parser/message.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/ADT/StringRef.h"
#include "llvm/Support/raw_ostream.h"
#include <algorithm>
#include <cinttypes>
@@ -46,6 +49,110 @@ bool Definition::set_isDisabled(bool disable) {
return was;
}
+void Definition::Print(llvm::raw_ostream &out,
+ llvm::StringRef macroName) const {
+ if (isDisabled_) {
+ return;
+ }
+ if (!isFunctionLike_) {
+ // If it's not a function-like macro, then just print the replacement.
+ out << ' ' << replacement_.ToString();
+ return;
+ }
+
+ // The sequence of characters from which argument names will be created.
+ static llvm::StringRef charSeq{"ABCDEFGHIJKLMNOPQRSTUVWXYZ"};
+
+ auto couldCollide = [&](llvm::StringRef str) {
+ return !str.empty() && llvm::all_of(str, [&](char c) {
+ return charSeq.find(c) != llvm::StringRef::npos;
+ });
+ };
+
+ // For function-like macros we need to invent valid argument names (they
+ // are represented as ~A, ~B, ...). These invented names cannot collide
+ // with any other tokens in the macro definitions.
+ llvm::SmallSet<std::string, 10> usedNames;
+ for (size_t i{0}, e = replacement_.SizeInTokens(); i != e; ++i) {
+ std::string tok{replacement_.TokenAt(i).ToString()};
+ if (tok.empty()) {
+ continue;
+ }
+ // The generated names will only use characters from `charSeq`, so
+ // collect names that could collide, and ignore others.
+ if (couldCollide(tok)) {
+ usedNames.insert(tok);
+ }
+ }
+ if (couldCollide(macroName)) {
+ usedNames.insert(macroName.str());
+ }
+
+ // Given a string that is either empty, or composed from characters
+ // from `charSeq`, create the next string in the lexicographical
+ // order.
+ auto getNextString = [&](llvm::StringRef str) {
+ if (str.empty()) {
+ return charSeq.take_front().str();
+ }
+ if (str.back() == charSeq.back()) {
+ return (llvm::Twine(str) + charSeq.take_front()).str();
+ }
+ size_t idx{charSeq.find(str.back())};
+ return (llvm::Twine(str.drop_back()) + charSeq.substr(idx + 1, 1)).str();
+ };
+
+ // Generate consecutive arg names, until we get one that works
+ // (i.e. doesn't collide with existing names). Give up after 4096
+ // attempts.
+ auto genArgName = [&](std::string name) {
+ for (size_t x{0}; x != 4096; ++x) {
+ name = getNextString(name);
+ if (!usedNames.contains(name))
+ return name;
+ }
+ return std::string();
+ };
+
+ std::string nextName;
+ llvm::SmallVector<std::string> argNames;
+ for (size_t i{0}; i != argumentCount_; ++i) {
+ nextName = genArgName(nextName);
+ if (nextName.empty()) {
+ out << " // unable to print";
+ return;
+ }
+ argNames.push_back(nextName);
+ }
+
+ // Finally, print the macro.
+ out << '(';
+ for (size_t i{0}; i != argumentCount_; ++i) {
+ if (i != 0) {
+ out << ", ";
+ }
+ out << argNames[i];
+ }
+ if (isVariadic_) {
+ out << ", ...";
+ }
+ out << ") ";
+
+ for (size_t i{0}, e{replacement_.SizeInTokens()}; i != e; ++i) {
+ std::string tok{replacement_.TokenAt(i).ToString()};
+ if (tok.size() >= 2 && tok[0] == '~') {
+ // This should be an argument name. The `Tokenize` function only
+ // generates a single character.
+ size_t idx{static_cast<size_t>(tok[1] - 'A')};
+ if (idx < argumentCount_) {
+ out << argNames[idx];
+ continue;
+ }
+ }
+ out << tok;
+ }
+}
+
static bool IsLegalIdentifierStart(const CharBlock &cpl) {
return cpl.size() > 0 && IsLegalIdentifierStart(cpl[0]);
}
@@ -713,6 +820,27 @@ void Preprocessor::Directive(const TokenSequence &dir, Prescanner &prescanner) {
}
}
+void Preprocessor::PrintMacros(llvm::raw_ostream &out) const {
+ // Sort the entries by macro name.
+ llvm::SmallVector<decltype(definitions_)::const_iterator> entries;
+ for (auto it{definitions_.begin()}, e{definitions_.end()}; it != e; ++it) {
+ entries.push_back(it);
+ }
+ llvm::sort(entries, [](const auto it1, const auto it2) {
+ return it1->first.ToString() < it2->first.ToString();
+ });
+
+ for (auto &&it : entries) {
+ const auto &[name, def]{*it};
+ if (def.isDisabled()) {
+ continue;
+ }
+ out << "#define " << name;
+ def.Print(out, name.ToString());
+ out << '\n';
+ }
+}
+
CharBlock Preprocessor::SaveTokenAsName(const CharBlock &t) {
names_.push_back(t.ToString());
return {names_.back().data(), names_.back().size()};
diff --git a/flang/lib/Parser/preprocessor.h b/flang/lib/Parser/preprocessor.h
index b61f1577727beb..b4177766f81c4b 100644
--- a/flang/lib/Parser/preprocessor.h
+++ b/flang/lib/Parser/preprocessor.h
@@ -18,6 +18,8 @@
#include "token-sequence.h"
#include "flang/Parser/char-block.h"
#include "flang/Parser/provenance.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/raw_ostream.h"
#include <cstddef>
#include <list>
#include <stack>
@@ -49,6 +51,8 @@ class Definition {
TokenSequence Apply(const std::vector<TokenSequence> &args, Prescanner &);
+ void Print(llvm::raw_ostream &out, llvm::StringRef macroName = "") const;
+
private:
static TokenSequence Tokenize(const std::vector<std::string> &argNames,
const TokenSequence &token, std::size_t firstToken, std::size_t tokens);
@@ -89,6 +93,8 @@ class Preprocessor {
// Implements a preprocessor directive.
void Directive(const TokenSequence &, Prescanner &);
+ void PrintMacros(llvm::raw_ostream &out) const;
+
private:
enum class IsElseActive { No, Yes };
enum class CanDeadElseAppear { No, Yes };
diff --git a/flang/test/Driver/driver-help-hidden.f90 b/flang/test/Driver/driver-help-hidden.f90
index bf3660d57cbb4f..fd2b0e41e38c56 100644
--- a/flang/test/Driver/driver-help-hidden.f90
+++ b/flang/test/Driver/driver-help-hidden.f90
@@ -21,6 +21,7 @@
! CHECK-NEXT: -ccc-print-phases Dump list of actions to perform
! CHECK-NEXT: -cpp Enable predefined and command line preprocessor macros
! CHECK-NEXT: -c Only run preprocess, compile, and assemble steps
+! CHECK-NEXT: -dM Print macro definitions in -E mode instead of normal output
! CHECK-NEXT: -dumpmachine Display the compiler's target processor
! CHECK-NEXT: -dumpversion Display the version of the compiler
! CHECK-NEXT: -D <macro>=<value> Define <macro> to <value> (or 1 if <value> omitted)
diff --git a/flang/test/Driver/driver-help.f90 b/flang/test/Driver/driver-help.f90
index b4280a454e3128..368cab97d8547f 100644
--- a/flang/test/Driver/driver-help.f90
+++ b/flang/test/Driver/driver-help.f90
@@ -17,6 +17,7 @@
! HELP-NEXT: -### Print (but do not run) the commands to run for this compilation
! HELP-NEXT: -cpp Enable predefined and command line preprocessor macros
! HELP-NEXT: -c Only run preprocess, compile, and assemble steps
+! HELP-NEXT: -dM Print macro definitions in -E mode instead of normal output
! HELP-NEXT: -dumpmachine Display the compiler's target processor
! HELP-NEXT: -dumpversion Display the version of the compiler
! HELP-NEXT: -D <macro>=<value> Define <macro> to <value> (or 1 if <value> omitted)
@@ -152,6 +153,7 @@
! HELP-FC1-NEXT:OPTIONS:
! HELP-FC1-NEXT: -cpp Enable predefined and command line preprocessor macros
! HELP-FC1-NEXT: --dependent-lib=<value> Add dependent library
+! HELP-FC1-NEXT: -dM Print macro definitions in -E mode instead of normal output
! HELP-FC1-NEXT: -D <macro>=<value> Define <macro> to <value> (or 1 if <value> omitted)
! HELP-FC1-NEXT: -emit-fir Build the parse tree, then lower it to FIR
! HELP-FC1-NEXT: -emit-hlfir Build the parse tree, then lower it to HLFIR
diff --git a/flang/test/Preprocessing/show-macros1.F90 b/flang/test/Preprocessing/show-macros1.F90
new file mode 100644
index 00000000000000..8e3d59a7849f70
--- /dev/null
+++ b/flang/test/Preprocessing/show-macros1.F90
@@ -0,0 +1,14 @@
+! RUN: %flang -dM -E -o - %s | FileCheck %s
+
+! Check the default macros. Omit certain ones such as __LINE__
+! or __FILE__, or target-specific ones, like __x86_64__.
+
+! Macros are printed in the alphabetical order.
+
+! CHECK: #define __DATE__
+! CHECK: #define __TIME__
+! CHECK: #define __flang__
+! CHECK: #define __flang_major__
+! CHECK: #define __flang_minor__
+! CHECK: #define __flang_patchlevel__
+
diff --git a/flang/test/Preprocessing/show-macros2.F90 b/flang/test/Preprocessing/show-macros2.F90
new file mode 100644
index 00000000000000..baf52ba8161f11
--- /dev/null
+++ b/flang/test/Preprocessing/show-macros2.F90
@@ -0,0 +1,6 @@
+! RUN: %flang -DFOO -DBAR=FOO -dM -E -o - %s | FileCheck %s
+
+! Check command line definitions
+
+! CHECK: #define BAR FOO
+! CHECK: #define FOO 1
diff --git a/flang/test/Preprocessing/show-macros3.F90 b/flang/test/Preprocessing/show-macros3.F90
new file mode 100644
index 00000000000000..4b07fcf2f505db
--- /dev/null
+++ b/flang/test/Preprocessing/show-macros3.F90
@@ -0,0 +1,10 @@
+! RUN: %flang -dM -E -o - %s | FileCheck %s
+
+! Variadic macro
+#define FOO1(X, Y, ...) bar(bar(X, Y), __VA_ARGS__)
+! CHECK: #define FOO1(A, B, ...) bar(bar(A, B), __VA_ARGS__)
+
+! Macro parameter names are synthesized, starting from 'A', B', etc.
+! Make sure the generated names do not collide with existing identifiers.
+#define FOO2(X, Y) (A + X + C + Y)
+! CHECK: #define FOO2(B, D) (A + B + C + D)
|
We have a customer requesting information about macros defined by the compiler, and this feature would address the issue. |
✅ With the latest revision this PR passed the C/C++ code formatter. |
This should work the same way as in clang.
5d1f8ff
to
f4917dc
Compare
cc: @bcornille |
The Windows build failure is due to an unrelated issue:
|
flang/lib/Parser/preprocessor.cpp
Outdated
size_t idx{static_cast<size_t>(tok[1] - 'A')}; | ||
if (idx < argumentCount_) { | ||
out << argNames[idx]; | ||
continue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use else
clauses and avoid the continue
; it's clearer to read.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Refactored the ifs.
flang/lib/Parser/preprocessor.cpp
Outdated
if (tok.empty()) { | ||
continue; | ||
} | ||
// The generated names will only use characters from `charSeq`, so |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can avoid all of this complexity if you modify the preprocessor to retain the original actual argument names in the macro bodies.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but I didn't want to make that change on my own. Do you want me to go ahead with that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, if you like, or I can do it quickly. There's at least two obvious approaches, and the one that I like would be to replace the instances of arguments like foo
with ~A~foo
in place of the current ~A
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you want to store the argument list (with names) as well? If not, we'd have to extract the names from the body, and sort them it based on the ~?~
character.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did something a bit different: I'm storing the argument names instead of making any changes to the tokenization. The reason is that if there is a macro with an unused argument, we still need to give it some name. Storing all of the actual names would completely eliminate the need to do that.
preprocessor.h is moved to make definition of Preprocessor available in parsing.h. token-sequence is moved because preprocessor.h depends on it.
The Windows build has the same failure as before (it's a known build breakage):
|
@klausler Are you ok with this? |
How is this code used outside of LLVM? Why do people want to use without LLVM? Just curious. |
There are people who want to construct tools to analyze or restructure Fortran programs outside of the context of compilation, and need to have access to a good parse tree or symbol table. Unlike most other languages, Fortran is weirdly hard to lex & parse, and tool writers would rather use an existing parser than do a partial job with something custom. So as a general rule, I avoid using llvm utilities in parsing and semantics (esp. parsing) when the facilities in |
I can see language features in IDE like vscode being a good candidate. |
Is building LLVMSupport and maybe a few other components a barrier, or do they just not want to do that? |
@kparzysz Is it supposed to also print the predefined macros? Or only the user-defined macros? |
This should work the same way as in clang.