cpp -traditional -dM file.h misses most macros

[kernigh]

Run clang's preprocessor with both -traditional and -dM, and it fails to dump most macros,

$ cat exam.h
#define MY1
/* line 2 */
#define MY3

Bad output from OpenBSD's packages of llvm-20.1.8p6 and llvm-21.1.8p4: MY3 is missing,

$ clang-cpp-20 -traditional -dM exam.h|grep MY 
#define MY1 

Good output from OpenBSD's packages of llvm-19.1.7p14 and gcc-15.2.0p5,

$ clang-cpp-19 -traditional -dM exam.h|grep MY 
#define MY1 
#define MY3 

In general, -traditional -dM stops at any line that isn't a preprocessor directive, like /* line 2 */ or int i; or an empty line. It never sees macros under that line.

I'm not sure, but I can guess the cause. -dM sets ShowCPP = false, ShowMacros = True, which causes clang/lib/Frontend/PrintPreprocessedOutput.cpp to call DoPrintMacros() and then LexTokensUntilEOF() in clang/lib/Lex/Preprocessor.cpp, which stops at a tok::unknown,

void Preprocessor::LexTokensUntilEOF(std::vector<Token> *Tokens) {
  while (1) {
    Token Tok;
    Lex(Tok);
    if (Tok.isOneOf(tok::unknown, tok::eof, tok::eod,
                    tok::annot_repl_input_end))
      break;
    if (Tokens != nullptr)
      Tokens->push_back(Tok);
  }
}

but -traditional sets TraditionalCPP = true, which causes clang/lib/Lex/Lexer.cpp to SetKeepWhitespaceMode(true); and make tok::unknown from whitespace,

  // If the client wants us to return whitespace, return it now.
  if (isKeepWhitespaceMode()) {
    FormTokenWithChars(Result, CurPtr, tok::unknown);

So -traditional and -dM don't work together; the tok::unknown whitespace from -traditional acts like an end of file in -dM, which misses most macros.

[llvmorg-github-actions]

@llvm/issue-subscribers-clang-frontend

Author: George Koehler (kernigh)

Details

Run clang's preprocessor with both `-traditional` and `-dM`, and it fails to dump most macros,

$ cat exam.h
#define MY1
/* line 2 */
#define MY3

Bad output from OpenBSD's packages of llvm-20.1.8p6 and llvm-21.1.8p4: MY3 is missing,

$ clang-cpp-20 -traditional -dM exam.h|grep MY 
#define MY1 

Good output from OpenBSD's packages of llvm-19.1.7p14 and gcc-15.2.0p5,

$ clang-cpp-19 -traditional -dM exam.h|grep MY 
#define MY1 
#define MY3 

In general, -traditional -dM stops at any line that isn't a preprocessor directive, like /* line 2 */ or int i; or an empty line. It never sees macros under that line.

I'm not sure, but I can guess the cause. -dM sets ShowCPP = false, ShowMacros = True, which causes clang/lib/Frontend/PrintPreprocessedOutput.cpp to call DoPrintMacros() and then LexTokensUntilEOF() in clang/lib/Lex/Preprocessor.cpp, which stops at a tok::unknown,

void Preprocessor::LexTokensUntilEOF(std::vector<Token> *Tokens) {
  while (1) {
    Token Tok;
    Lex(Tok);
    if (Tok.isOneOf(tok::unknown, tok::eof, tok::eod,
                    tok::annot_repl_input_end))
      break;
    if (Tokens != nullptr)
      Tokens->push_back(Tok);
  }
}

but -traditional sets TraditionalCPP = true, which causes Lexer.cpp to SetKeepWhitespaceMode(true); and make lib/Lex/tok::unknown from whitespace,

  // If the client wants us to return whitespace, return it now.
  if (isKeepWhitespaceMode()) {
    FormTokenWithChars(Result, CurPtr, tok::unknown);

So -traditional and -dM don't work together; the tok::unknown whitespace from -traditional acts like an end of file in -dM, which misses most macros.

[shafik]

CC @cor3ntin @tahonermann