[clang] Implement Change scope of lambda trailing-return-type

Implement P2036R3.

Captured variables by copy (explicitely or not), are deduced
correctly at the point we know whether the lambda is mutable,
and ill-formed before that.

Up until now, the entire lambda declaration up to the start
of the body would  be parsed in the parent scope, such that
captures would not be available to look up.

The scoping is changed to have an outer lambda scope,
followed by the lambda prototype and body.

The lambda scope is necessary because there may be a template scope
between the start of the lambda (to which we want to attach
the captured variable) and the prototype scope.

We also need to introduce a declaration context to attach the captured
variable to (and several parts of clang assume captures are handled from
the call operator context), before we know the type of the call operator.

The order of operations is as follow:

* Parse the init capture in the lambda's parent scope
* Introduce a lambda scope
* Create the lambda class and call operator
* Add the init captures to the call operator context and the lambda scope.
  But the variables are not capured yet (because we don't know their type).
  Instead, explicit  captures are stored in a temporary map that
  conserves the order of capture (for the purpose of having a stable order in the ast dumps).

* A flag is set on LambdaScopeInfo to indicate that we have not yet injected the captures.

* The parameters are parsed (in the parent context, as lambda mangling recurses in the parent context,
  we couldn't mangle a lambda that is attached to the context of a lambda whose type is not yet known).

* The lambda qualifiers are parsed, at this point,
  we can switch (for the second time) inside the lambda context,
  unset the flag indicating that we have not parsed the lambda qualifiers,
  record the lambda is mutable and capture the explicit variables.

* We can parse the rest of the lambda type, transform the lambda and call operator's types and also
  transform the call operator to a template function decl where necessary.

At this point, both captures and parameters can be injected in the body's scope.
When trying to capture an implicit variable, if we are before the qualifiers of a lambda,
we need to remember that the variables are still in the parent's context (rather than in the call operator's).

This is a recommit of adff142 after a fix in d8d793f

Reviewed By: aaron.ballman, #clang-language-wg, ChuanqiXu

Differential Revision: https://reviews.llvm.org/D119136

clang/docs/ReleaseNotes.rst

@@ -247,6 +247,9 @@ C++2b Feature Support
 - Implemented `P2128R6: Multidimensional subscript operator <https://wg21.link/P2128R6>`_.
 - Implemented `P0849R8: auto(x): decay-copy in the language <https://wg21.link/P0849R8>`_.
 - Implemented `P2242R3: Non-literal variables (and labels and gotos) in constexpr functions	<https://wg21.link/P2242R3>`_.
+- Implemented `P2036R3: Change scope of lambda trailing-return-type <https://wg21.link/P2036R3>`_.
+  This proposal modifies how variables captured in lambdas can appear in trailing return type
+  expressions and how their types are deduced therein, in all C++ language versions.
 
 CUDA Language Changes in Clang
 ------------------------------

clang/include/clang/AST/DeclCXX.h

@@ -1799,6 +1799,20 @@ class CXXRecordDecl : public RecordDecl {
     return getLambdaData().MethodTyInfo;
   }
 
+  void setLambdaTypeInfo(TypeSourceInfo *TS) {
+    auto *DD = DefinitionData;
+    assert(DD && DD->IsLambda && "setting lambda property of non-lambda class");
+    auto &DL = static_cast<LambdaDefinitionData &>(*DD);
+    DL.MethodTyInfo = TS;
+  }
+
+  void setLambdaIsGeneric(bool IsGeneric) {
+    auto *DD = DefinitionData;
+    assert(DD && DD->IsLambda && "setting lambda property of non-lambda class");
+    auto &DL = static_cast<LambdaDefinitionData &>(*DD);
+    DL.IsGenericLambda = IsGeneric;
+  }
+
   // Determine whether this type is an Interface Like type for
   // __interface inheritance purposes.
   bool isInterfaceLike() const;

clang/include/clang/Basic/DiagnosticSemaKinds.td

@@ -7704,6 +7704,8 @@ let CategoryName = "Lambda Issue" in {
   def err_lambda_impcap : Error<
     "variable %0 cannot be implicitly captured in a lambda with no "
     "capture-default specified">;
+  def err_lambda_used_before_capture: Error<
+    "captured variable %0 cannot appear here">;
   def note_lambda_variable_capture_fixit : Note<
     "capture %0 by %select{value|reference}1">;
   def note_lambda_default_capture_fixit : Note<

clang/include/clang/Sema/Scope.h

@@ -44,11 +44,11 @@ class Scope {
   enum ScopeFlags {
     /// This indicates that the scope corresponds to a function, which
     /// means that labels are set here.
-    FnScope       = 0x01,
+    FnScope = 0x01,
 
     /// This is a while, do, switch, for, etc that can have break
     /// statements embedded into it.
-    BreakScope    = 0x02,
+    BreakScope = 0x02,
 
     /// This is a while, do, for, which can have continue statements
     /// embedded into it.
@@ -140,6 +140,12 @@ class Scope {
     /// parsed. If such a scope is a ContinueScope, it's invalid to jump to the
     /// continue block from here.
     ConditionVarScope = 0x2000000,
+
+    /// This is the scope for a lambda, after the lambda introducer.
+    /// Lambdas need 2 FunctionPrototypeScope scopes (because there is a
+    /// template scope in between), the outer scope does not increase the
+    /// depth of recursion.
+    LambdaScope = 0x4000000,
   };
 
 private:

clang/include/clang/Sema/ScopeInfo.h

@@ -831,6 +831,28 @@ class LambdaScopeInfo final :
   /// The lambda's compiler-generated \c operator().
   CXXMethodDecl *CallOperator = nullptr;
 
+  struct DelayedCapture {
+    VarDecl *Var;
+    SourceLocation Loc;
+    LambdaCaptureKind Kind;
+  };
+
+  /// Holds the captures until we parsed the qualifiers, as the cv qualified
+  /// type of captures can only be computed at that point, and the captures
+  /// should not be visible before.
+  /// The index represents the position in the original capture list.
+  /// We use a map as not all index represents captures (defaults), or are
+  /// captured (some captures are invalid).
+  llvm::DenseMap<unsigned, DelayedCapture> DelayedCaptures;
+
+  /// Whether the current scope when parsing the lambda
+  /// is after the call operator qualifiers,
+  /// which is the point at which the captures are usable
+  /// per [expr.prim.id.unqual]/p3.2 and [expr.prim.lambda.capture]/6.
+  /// This is set to false by default as the lambda can be reconstructed during
+  /// instantiation
+  bool BeforeLambdaQualifiersScope = false;
+
   /// Source range covering the lambda introducer [...].
   SourceRange IntroducerRange;
 

clang/include/clang/Sema/Sema.h

@@ -6852,11 +6852,9 @@ class Sema final {
   ///
   ///  CodeGen handles emission of lambda captures, ignoring these dummy
   ///  variables appropriately.
-  VarDecl *createLambdaInitCaptureVarDecl(SourceLocation Loc,
-                                          QualType InitCaptureType,
-                                          SourceLocation EllipsisLoc,
-                                          IdentifierInfo *Id,
-                                          unsigned InitStyle, Expr *Init);
+  VarDecl *createLambdaInitCaptureVarDecl(
+      SourceLocation Loc, QualType InitCaptureType, SourceLocation EllipsisLoc,
+      IdentifierInfo *Id, unsigned InitStyle, Expr *Init, DeclContext *DeclCtx);
 
   /// Add an init-capture to a lambda scope.
   void addInitCapture(sema::LambdaScopeInfo *LSI, VarDecl *Var);
@@ -6865,21 +6863,28 @@ class Sema final {
   /// given lambda.
   void finishLambdaExplicitCaptures(sema::LambdaScopeInfo *LSI);
 
-  /// \brief This is called after parsing the explicit template parameter list
+  /// Deduce a block or lambda's return type based on the return
+  /// statements present in the body.
+  void deduceClosureReturnType(sema::CapturingScopeInfo &CSI);
+
+  /// Once the Lambdas capture are known, we can
+  /// start to create the closure, call operator method,
+  /// and keep track of the captures.
+  /// We do the capture lookup here, but they are not actually captured
+  /// until after we know what the qualifiers of the call operator are.
+  void ActOnLambdaIntroducer(LambdaIntroducer &Intro, Scope *CurContext);
+
+  /// This is called after parsing the explicit template parameter list
   /// on a lambda (if it exists) in C++2a.
-  void ActOnLambdaExplicitTemplateParameterList(SourceLocation LAngleLoc,
+  void ActOnLambdaExplicitTemplateParameterList(LambdaIntroducer &Intro,
+                                                SourceLocation LAngleLoc,
                                                 ArrayRef<NamedDecl *> TParams,
                                                 SourceLocation RAngleLoc,
                                                 ExprResult RequiresClause);
 
-  /// Introduce the lambda parameters into scope.
-  void addLambdaParameters(
-      ArrayRef<LambdaIntroducer::LambdaCapture> Captures,
-      CXXMethodDecl *CallOperator, Scope *CurScope);
-
-  /// Deduce a block or lambda's return type based on the return
-  /// statements present in the body.
-  void deduceClosureReturnType(sema::CapturingScopeInfo &CSI);
+  void ActOnLambdaClosureQualifiers(LambdaIntroducer &Intro,
+                                    SourceLocation MutableLoc,
+                                    SourceLocation EndLoc, const DeclSpec &DS);
 
   /// ActOnStartOfLambdaDefinition - This is called just before we start
   /// parsing the body of a lambda; it analyzes the explicit captures and

clang/lib/Parse/ParseExprCXX.cpp

@@ -1250,7 +1250,13 @@ ExprResult Parser::ParseLambdaExpressionAfterIntroducer(
   DeclSpec DS(AttrFactory);
   Declarator D(DS, DeclaratorContext::LambdaExpr);
   TemplateParameterDepthRAII CurTemplateDepthTracker(TemplateParameterDepth);
+
+  ParseScope LambdaScope(this, Scope::LambdaScope | Scope::DeclScope |
+                                   Scope::FunctionDeclarationScope |
+                                   Scope::FunctionPrototypeScope);
+
   Actions.PushLambdaScope();
+  Actions.ActOnLambdaIntroducer(Intro, getCurScope());
 
   ParsedAttributes Attr(AttrFactory);
   if (getLangOpts().CUDA) {
@@ -1300,7 +1306,7 @@ ExprResult Parser::ParseLambdaExpressionAfterIntroducer(
       }
 
       Actions.ActOnLambdaExplicitTemplateParameterList(
-          LAngleLoc, TemplateParams, RAngleLoc, RequiresClause);
+          Intro, LAngleLoc, TemplateParams, RAngleLoc, RequiresClause);
       ++CurTemplateDepthTracker;
     }
   }
@@ -1318,28 +1324,31 @@ ExprResult Parser::ParseLambdaExpressionAfterIntroducer(
 
   TypeResult TrailingReturnType;
   SourceLocation TrailingReturnTypeLoc;
+  SourceLocation LParenLoc, RParenLoc;
+  SourceLocation DeclEndLoc;
+  bool HasParentheses = false;
+  bool HasSpecifiers = false;
+  SourceLocation MutableLoc;
+
+  auto ParseConstexprAndMutableSpecifiers = [&] {
+    // GNU-style attributes must be parsed before the mutable specifier to
+    // be compatible with GCC. MSVC-style attributes must be parsed before
+    // the mutable specifier to be compatible with MSVC.
+    MaybeParseAttributes(PAKM_GNU | PAKM_Declspec, Attr);
+    // Parse mutable-opt and/or constexpr-opt or consteval-opt, and update
+    // the DeclEndLoc.
+    SourceLocation ConstexprLoc;
+    SourceLocation ConstevalLoc;
+    tryConsumeLambdaSpecifierToken(*this, MutableLoc, ConstexprLoc,
+                                   ConstevalLoc, DeclEndLoc);
+
+    addConstexprToLambdaDeclSpecifier(*this, ConstexprLoc, DS);
+    addConstevalToLambdaDeclSpecifier(*this, ConstevalLoc, DS);
+  };
 
   auto ParseLambdaSpecifiers =
-      [&](SourceLocation LParenLoc, SourceLocation RParenLoc,
-          MutableArrayRef<DeclaratorChunk::ParamInfo> ParamInfo,
+      [&](MutableArrayRef<DeclaratorChunk::ParamInfo> ParamInfo,
           SourceLocation EllipsisLoc) {
-        SourceLocation DeclEndLoc = RParenLoc;
-
-        // GNU-style attributes must be parsed before the mutable specifier to
-        // be compatible with GCC. MSVC-style attributes must be parsed before
-        // the mutable specifier to be compatible with MSVC.
-        MaybeParseAttributes(PAKM_GNU | PAKM_Declspec, Attr);
-
-        // Parse mutable-opt and/or constexpr-opt or consteval-opt, and update
-        // the DeclEndLoc.
-        SourceLocation MutableLoc;
-        SourceLocation ConstexprLoc;
-        SourceLocation ConstevalLoc;
-        tryConsumeLambdaSpecifierToken(*this, MutableLoc, ConstexprLoc,
-                                       ConstevalLoc, DeclEndLoc);
-
-        addConstexprToLambdaDeclSpecifier(*this, ConstexprLoc, DS);
-        addConstevalToLambdaDeclSpecifier(*this, ConstevalLoc, DS);
         // Parse exception-specification[opt].
         ExceptionSpecificationType ESpecType = EST_None;
         SourceRange ESpecRange;
@@ -1393,20 +1402,22 @@ ExprResult Parser::ParseLambdaExpressionAfterIntroducer(
                 /*DeclsInPrototype=*/None, LParenLoc, FunLocalRangeEnd, D,
                 TrailingReturnType, TrailingReturnTypeLoc, &DS),
             std::move(Attr), DeclEndLoc);
+
+        if (HasParentheses && Tok.is(tok::kw_requires))
+          ParseTrailingRequiresClause(D);
       };
 
-  if (Tok.is(tok::l_paren)) {
-    ParseScope PrototypeScope(this, Scope::FunctionPrototypeScope |
-                                        Scope::FunctionDeclarationScope |
-                                        Scope::DeclScope);
+  ParseScope Prototype(this, Scope::FunctionPrototypeScope |
+                                 Scope::FunctionDeclarationScope |
+                                 Scope::DeclScope);
 
+  // Parse parameter-declaration-clause.
+  SmallVector<DeclaratorChunk::ParamInfo, 16> ParamInfo;
+  SourceLocation EllipsisLoc;
+  if (Tok.is(tok::l_paren)) {
     BalancedDelimiterTracker T(*this, tok::l_paren);
     T.consumeOpen();
-    SourceLocation LParenLoc = T.getOpenLocation();
-
-    // Parse parameter-declaration-clause.
-    SmallVector<DeclaratorChunk::ParamInfo, 16> ParamInfo;
-    SourceLocation EllipsisLoc;
+    LParenLoc = T.getOpenLocation();
 
     if (Tok.isNot(tok::r_paren)) {
       Actions.RecordParsingTemplateParameterDepth(
@@ -1424,36 +1435,38 @@ ExprResult Parser::ParseLambdaExpressionAfterIntroducer(
     }
 
     T.consumeClose();
+    DeclEndLoc = RParenLoc = T.getCloseLocation();
+    HasParentheses = true;
+  }
 
-    // Parse lambda-specifiers.
-    ParseLambdaSpecifiers(LParenLoc, /*DeclEndLoc=*/T.getCloseLocation(),
-                          ParamInfo, EllipsisLoc);
-
-    // Parse requires-clause[opt].
-    if (Tok.is(tok::kw_requires))
-      ParseTrailingRequiresClause(D);
-  } else if (Tok.isOneOf(tok::kw_mutable, tok::arrow, tok::kw___attribute,
-                         tok::kw_constexpr, tok::kw_consteval,
-                         tok::kw___private, tok::kw___global, tok::kw___local,
-                         tok::kw___constant, tok::kw___generic,
-                         tok::kw_requires, tok::kw_noexcept) ||
-             (Tok.is(tok::l_square) && NextToken().is(tok::l_square))) {
-    if (!getLangOpts().CPlusPlus2b)
+  if (Tok.isOneOf(tok::kw_mutable, tok::arrow, tok::kw___attribute,
+                  tok::kw_constexpr, tok::kw_consteval, tok::kw___private,
+                  tok::kw___global, tok::kw___local, tok::kw___constant,
+                  tok::kw___generic, tok::kw_requires, tok::kw_noexcept) ||
+      (Tok.is(tok::l_square) && NextToken().is(tok::l_square))) {
+    HasSpecifiers = true;
+    if (!HasParentheses && !getLangOpts().CPlusPlus2b) {
       // It's common to forget that one needs '()' before 'mutable', an
       // attribute specifier, the result type, or the requires clause. Deal with
       // this.
       Diag(Tok, diag::ext_lambda_missing_parens)
           << FixItHint::CreateInsertion(Tok.getLocation(), "() ");
+    }
+  }
 
-    SourceLocation NoLoc;
-    // Parse lambda-specifiers.
-    std::vector<DeclaratorChunk::ParamInfo> EmptyParamInfo;
-    ParseLambdaSpecifiers(/*LParenLoc=*/NoLoc, /*RParenLoc=*/NoLoc,
-                          EmptyParamInfo, /*EllipsisLoc=*/NoLoc);
+  if (HasParentheses || HasSpecifiers) {
+    ParseConstexprAndMutableSpecifiers();
   }
 
+  Actions.ActOnLambdaClosureQualifiers(Intro, MutableLoc, DeclEndLoc, DS);
+
+  if (HasSpecifiers || HasParentheses)
+    ParseLambdaSpecifiers(ParamInfo, EllipsisLoc);
+
   WarnIfHasCUDATargetAttr();
 
+  Prototype.Exit();
+
   // FIXME: Rename BlockScope -> ClosureScope if we decide to continue using
   // it.
   unsigned ScopeFlags = Scope::BlockScope | Scope::FnScope | Scope::DeclScope |
@@ -1472,6 +1485,7 @@ ExprResult Parser::ParseLambdaExpressionAfterIntroducer(
   StmtResult Stmt(ParseCompoundStatementBody());
   BodyScope.Exit();
   TemplateParamScope.Exit();
+  LambdaScope.Exit();
 
   if (!Stmt.isInvalid() && !TrailingReturnType.isInvalid())
     return Actions.ActOnLambdaExpr(LambdaBeginLoc, Stmt.get(), getCurScope());

clang/lib/Sema/Scope.cpp

@@ -67,8 +67,10 @@ void Scope::setFlags(Scope *parent, unsigned flags) {
   if (flags & BlockScope)         BlockParent = this;
   if (flags & TemplateParamScope) TemplateParamParent = this;
 
-  // If this is a prototype scope, record that.
-  if (flags & FunctionPrototypeScope) PrototypeDepth++;
+  // If this is a prototype scope, record that. Lambdas have an extra prototype
+  // scope that doesn't add any depth.
+  if (flags & FunctionPrototypeScope && !(flags & LambdaScope))
+    PrototypeDepth++;
 
   if (flags & DeclScope) {
     if (flags & FunctionPrototypeScope)

clang/lib/Sema/Sema.cpp

@@ -2297,7 +2297,8 @@ FunctionScopeInfo *Sema::getEnclosingFunction() const {
 LambdaScopeInfo *Sema::getEnclosingLambda() const {
   for (auto *Scope : llvm::reverse(FunctionScopes)) {
     if (auto *LSI = dyn_cast<sema::LambdaScopeInfo>(Scope)) {
-      if (LSI->Lambda && !LSI->Lambda->Encloses(CurContext)) {
+      if (LSI->Lambda && !LSI->Lambda->Encloses(CurContext) &&
+          !LSI->BeforeLambdaQualifiersScope) {
         // We have switched contexts due to template instantiation.
         // FIXME: We should swap out the FunctionScopes during code synthesis
         // so that we don't need to check for this.
@@ -2323,8 +2324,9 @@ LambdaScopeInfo *Sema::getCurLambda(bool IgnoreNonLambdaCapturingScope) {
       return nullptr;
   }
   auto *CurLSI = dyn_cast<LambdaScopeInfo>(*I);
-  if (CurLSI && CurLSI->Lambda &&
-      !CurLSI->Lambda->Encloses(CurContext)) {
+  if (CurLSI && CurLSI->Lambda && CurLSI->CallOperator &&
+      !CurLSI->Lambda->Encloses(CurContext) &&
+      !CurLSI->BeforeLambdaQualifiersScope) {
     // We have switched contexts due to template instantiation.
     assert(!CodeSynthesisContexts.empty());
     return nullptr;

clang/lib/Sema/SemaCXXScopeSpec.cpp

@@ -292,6 +292,11 @@ bool Sema::ActOnCXXGlobalScopeSpecifier(SourceLocation CCLoc,
 bool Sema::ActOnSuperScopeSpecifier(SourceLocation SuperLoc,
                                     SourceLocation ColonColonLoc,
                                     CXXScopeSpec &SS) {
+  if (getCurLambda()) {
+    Diag(SuperLoc, diag::err_super_in_lambda_unsupported);
+    return true;
+  }
+
   CXXRecordDecl *RD = nullptr;
   for (Scope *S = getCurScope(); S; S = S->getParent()) {
     if (S->isFunctionScope()) {
@@ -308,9 +313,6 @@ bool Sema::ActOnSuperScopeSpecifier(SourceLocation SuperLoc,
   if (!RD) {
     Diag(SuperLoc, diag::err_invalid_super_scope);
     return true;
-  } else if (RD->isLambda()) {
-    Diag(SuperLoc, diag::err_super_in_lambda_unsupported);
-    return true;
   } else if (RD->getNumBases() == 0) {
     Diag(SuperLoc, diag::err_no_base_classes) << RD->getName();
     return true;