[flang] Don't retain FIXED/FREE compiler directives

[klausler]

Some old code in the prescanner, antedating the current -E output mechanisms, retains the !DIR$ FIXED and !DIR$ FREE directives in the input, and will even generate them to append to the scanned source from source and include files to restore the fixed/free source form distinction. But these directives have not been needed since the -E output generator began generating source form insensitive output, and they can confuse the parser's error recovery when the appended directives follow the END statement. Change their handling so that they're read and respected by the prescanner but no longer retained in either the -E output or the cooked character stream passed on to the parser.

Fixes a regression reported by @DanielCChen after PR 159834.

[llvmbot]

@llvm/pr-subscribers-flang-parser

Author: Peter Klausler (klausler)

Changes

Some old code in the prescanner, antedating the current -E output mechanisms, retains the !DIR$ FIXED and !DIR$ FREE directives in the input, and will even generate them to append to the scanned source from source and include files to restore the fixed/free source form distinction. But these directives have not been needed since the -E output generator began generating source form insensitive output, and they can confuse the parser's error recovery when the appended directives follow the END statement. Change their handling so that they're read and respected by the prescanner but no longer retained in either the -E output or the cooked character stream passed on to the parser.

Fixes a regression reported by @danielcchen after PR 159834.

---

Full diff: https://github.com/llvm/llvm-project/pull/160780.diff

3 Files Affected:

• (modified) flang/lib/Parser/prescan.cpp (+15-18)
• (modified) flang/lib/Parser/prescan.h (+1-1)
• (added) flang/test/Preprocessing/fixed-free.f (+8)

diff --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 3a9a475c365ee..865c149380d85 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -97,17 +97,7 @@ void Prescanner::Prescan(ProvenanceRange range) {
   while (!IsAtEnd()) {
     Statement();
   }
-  if (inFixedForm_ != beganInFixedForm) {
-    std::string dir{"!dir$ "};
-    if (beganInFixedForm) {
-      dir += "fixed";
-    } else {
-      dir += "free";
-    }
-    dir += '\n';
-    TokenSequence tokens{dir, allSources_.AddCompilerInsertion(dir).start()};
-    tokens.Emit(cooked_);
-  }
+  inFixedForm_ = beganInFixedForm;
 }
 
 void Prescanner::Statement() {
@@ -324,10 +314,11 @@ void Prescanner::Statement() {
       }
       NormalizeCompilerDirectiveCommentMarker(*preprocessed);
       preprocessed->ToLowerCase();
-      SourceFormChange(preprocessed->ToString());
-      CheckAndEmitLine(
-          preprocessed->ClipComment(*this, true /* skip first ! */),
-          newlineProvenance);
+      if (!SourceFormChange(preprocessed->ToString())) {
+        CheckAndEmitLine(
+            preprocessed->ClipComment(*this, true /* skip first ! */),
+            newlineProvenance);
+      }
       break;
     case LineClassification::Kind::Source:
       if (inFixedForm_) {
@@ -370,14 +361,16 @@ void Prescanner::Statement() {
         }
       }
       tokens.ToLowerCase();
-      SourceFormChange(tokens.ToString());
+      if (!SourceFormChange(tokens.ToString())) {
+        CheckAndEmitLine(tokens, newlineProvenance);
+      }
     } else { // Kind::Source
       tokens.ToLowerCase();
       if (inFixedForm_) {
         EnforceStupidEndStatementRules(tokens);
       }
+      CheckAndEmitLine(tokens, newlineProvenance);
     }
-    CheckAndEmitLine(tokens, newlineProvenance);
   }
   directiveSentinel_ = nullptr;
 }
@@ -1774,11 +1767,15 @@ Prescanner::LineClassification Prescanner::ClassifyLine(
   return classification;
 }
 
-void Prescanner::SourceFormChange(std::string &&dir) {
+bool Prescanner::SourceFormChange(std::string &&dir) {
   if (dir == "!dir$ free") {
     inFixedForm_ = false;
+    return true;
   } else if (dir == "!dir$ fixed") {
     inFixedForm_ = true;
+    return true;
+  } else {
+    return false;
   }
 }
 
diff --git a/flang/lib/Parser/prescan.h b/flang/lib/Parser/prescan.h
index c181c03273ccc..fc38adb926530 100644
--- a/flang/lib/Parser/prescan.h
+++ b/flang/lib/Parser/prescan.h
@@ -225,7 +225,7 @@ class Prescanner {
   LineClassification ClassifyLine(const char *) const;
   LineClassification ClassifyLine(
       TokenSequence &, Provenance newlineProvenance) const;
-  void SourceFormChange(std::string &&);
+  bool SourceFormChange(std::string &&);
   bool CompilerDirectiveContinuation(TokenSequence &, const char *sentinel);
   bool SourceLineContinuation(TokenSequence &);
 
diff --git a/flang/test/Preprocessing/fixed-free.f b/flang/test/Preprocessing/fixed-free.f
new file mode 100644
index 0000000000000..95f63a4d71e4c
--- /dev/null
+++ b/flang/test/Preprocessing/fixed-free.f
@@ -0,0 +1,8 @@
+!RUN: %flang -E %s 2>&1 | FileCheck %s
+!RUN: %flang -fc1 -fsyntax-only %s 2>&1 | FileCheck --allow-empty %s
+!CHECK-NOT: dir$
+!CHECK-NOT: error:
+!dir$ fixed
+        continue
+!dir$ free
+        end

[DanielCChen]

LGTM.
All the regressions are fixed.
Thanks!