Clang frontend C++ crash when initializing a large vector #63562

brutalsavage · 2023-06-27T19:20:22Z

To quickly reproduce: https://gcc.godbolt.org/z/ssGz6Yna8 (assertion-trunk)

#include <vector>

// a different number such as 10995116277000 doesn't crash
std::vector<int> v(1099511627777, 0);

Compiling the above code crashes clang clang++ -x c++ --std=c++20 , crashes locally using clang-17.0 (a10019a), also on trunk with assertion (see godbolt link)

Note that not all large numbers can lead to compiler crash. A number 10995116277000 which is larger compiles successfully. Additionally, this crash doesn't seem to happen without --std=c++20 or above as shown in the godbolt link.

The text was updated successfully, but these errors were encountered:

brutalsavage · 2023-06-27T19:21:25Z

Assertion:

clang++: /root/llvm-project/clang/lib/AST/ExprConstant.cpp:3437: 
void expandArray(clang::APValue&, unsigned int): 
Assertion `Index < Size' failed.

Backtrace:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /opt/compiler-explorer/clang-assertions-trunk/bin/clang++ -gdwarf-4 -g -o /app/output.s -mllvm --x86-asm-syntax=intel -S --gcc-toolchain=/opt/compiler-explorer/gcc-snapshot -fcolor-diagnostics -fno-crash-diagnostics --std=c++20 <source>
1.	<source>:4:37: current parser token ';'
 #0 0x0000560cf2211f2f llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x3c52f2f)
 #1 0x0000560cf220fc9c llvm::sys::CleanupOnSignal(unsigned long) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x3c50c9c)
 #2 0x0000560cf215a398 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
 #3 0x00007fe8bd1eb420 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x14420)
 #4 0x00007fe8bccb800b raise (/lib/x86_64-linux-gnu/libc.so.6+0x4300b)
 #5 0x00007fe8bcc97859 abort (/lib/x86_64-linux-gnu/libc.so.6+0x22859)
 #6 0x00007fe8bcc97729 (/lib/x86_64-linux-gnu/libc.so.6+0x22729)
 #7 0x00007fe8bcca8fd6 (/lib/x86_64-linux-gnu/libc.so.6+0x33fd6)
 #8 0x0000560cf5af35f8 (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x75345f8)
 #9 0x0000560cf5b70f01 (anonymous namespace)::PointerExprEvaluator::VisitCXXNewExpr(clang::CXXNewExpr const*) ExprConstant.cpp:0:0
#10 0x0000560cf5b3194a clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::PointerExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#11 0x0000560cf5b32ea6 EvaluatePointer(clang::Expr const*, (anonymous namespace)::LValue&, (anonymous namespace)::EvalInfo&, bool) ExprConstant.cpp:0:0
#12 0x0000560cf5b194a8 Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#13 0x0000560cf5b3e47e EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#14 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#15 0x0000560cf5b41e01 HandleFunctionCall(clang::SourceLocation, clang::FunctionDecl const*, (anonymous namespace)::LValue const*, llvm::ArrayRef<clang::Expr const*>, (anonymous namespace)::CallRef, clang::Stmt const*, (anonymous namespace)::EvalInfo&, clang::APValue&, (anonymous namespace)::LValue const*) ExprConstant.cpp:0:0
#16 0x0000560cf5b303ec (anonymous namespace)::PointerExprEvaluator::VisitCallExpr(clang::CallExpr const*) ExprConstant.cpp:0:0
#17 0x0000560cf5b315c1 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::PointerExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#18 0x0000560cf5b32ea6 EvaluatePointer(clang::Expr const*, (anonymous namespace)::LValue&, (anonymous namespace)::EvalInfo&, bool) ExprConstant.cpp:0:0
#19 0x0000560cf5b194a8 Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#20 0x0000560cf5b19d4f EvaluateIgnoredValue((anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#21 0x0000560cf5b3e299 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#22 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#23 0x0000560cf5b3dca5 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#24 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#25 0x0000560cf5b41e01 HandleFunctionCall(clang::SourceLocation, clang::FunctionDecl const*, (anonymous namespace)::LValue const*, llvm::ArrayRef<clang::Expr const*>, (anonymous namespace)::CallRef, clang::Stmt const*, (anonymous namespace)::EvalInfo&, clang::APValue&, (anonymous namespace)::LValue const*) ExprConstant.cpp:0:0
#26 0x0000560cf5b74c6f (anonymous namespace)::VoidExprEvaluator::VisitCallExpr(clang::CallExpr const*) ExprConstant.cpp:0:0
#27 0x0000560cf5b76136 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::VoidExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#28 0x0000560cf5b19acb Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#29 0x0000560cf5b19d4f EvaluateIgnoredValue((anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#30 0x0000560cf5b3e299 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#31 0x0000560cf5b4218e EvaluateLoopBody((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) ExprConstant.cpp:0:0
#32 0x0000560cf5b3dda9 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#33 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#34 0x0000560cf5b3ce8b EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#35 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#36 0x0000560cf5b41e01 HandleFunctionCall(clang::SourceLocation, clang::FunctionDecl const*, (anonymous namespace)::LValue const*, llvm::ArrayRef<clang::Expr const*>, (anonymous namespace)::CallRef, clang::Stmt const*, (anonymous namespace)::EvalInfo&, clang::APValue&, (anonymous namespace)::LValue const*) ExprConstant.cpp:0:0
#37 0x0000560cf5b303ec (anonymous namespace)::PointerExprEvaluator::VisitCallExpr(clang::CallExpr const*) ExprConstant.cpp:0:0
#38 0x0000560cf5b315c1 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::PointerExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#39 0x0000560cf5b32ea6 EvaluatePointer(clang::Expr const*, (anonymous namespace)::LValue&, (anonymous namespace)::EvalInfo&, bool) ExprConstant.cpp:0:0
#40 0x0000560cf5b194a8 Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#41 0x0000560cf5b3e47e EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#42 0x0000560cf5b3dca5 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#43 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#44 0x0000560cf5b41e01 HandleFunctionCall(clang::SourceLocation, clang::FunctionDecl const*, (anonymous namespace)::LValue const*, llvm::ArrayRef<clang::Expr const*>, (anonymous namespace)::CallRef, clang::Stmt const*, (anonymous namespace)::EvalInfo&, clang::APValue&, (anonymous namespace)::LValue const*) ExprConstant.cpp:0:0
#45 0x0000560cf5b303ec (anonymous namespace)::PointerExprEvaluator::VisitCallExpr(clang::CallExpr const*) ExprConstant.cpp:0:0
#46 0x0000560cf5b315c1 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::PointerExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#47 0x0000560cf5b32ea6 EvaluatePointer(clang::Expr const*, (anonymous namespace)::LValue&, (anonymous namespace)::EvalInfo&, bool) ExprConstant.cpp:0:0
#48 0x0000560cf5b194a8 Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#49 0x0000560cf5b43cff clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::LValueExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#50 0x0000560cf5b44c7d EvaluateLValue(clang::Expr const*, (anonymous namespace)::LValue&, (anonymous namespace)::EvalInfo&, bool) ExprConstant.cpp:0:0
#51 0x0000560cf5b19104 Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#52 0x0000560cf5b19d4f EvaluateIgnoredValue((anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#53 0x0000560cf5b3e299 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#54 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#55 0x0000560cf5b41e01 HandleFunctionCall(clang::SourceLocation, clang::FunctionDecl const*, (anonymous namespace)::LValue const*, llvm::ArrayRef<clang::Expr const*>, (anonymous namespace)::CallRef, clang::Stmt const*, (anonymous namespace)::EvalInfo&, clang::APValue&, (anonymous namespace)::LValue const*) ExprConstant.cpp:0:0
#56 0x0000560cf5b74c6f (anonymous namespace)::VoidExprEvaluator::VisitCallExpr(clang::CallExpr const*) ExprConstant.cpp:0:0
#57 0x0000560cf5b76110 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::VoidExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#58 0x0000560cf5b19acb Evaluate(clang::APValue&, (anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#59 0x0000560cf5b19d4f EvaluateIgnoredValue((anonymous namespace)::EvalInfo&, clang::Expr const*) ExprConstant.cpp:0:0
#60 0x0000560cf5b3e299 EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#61 0x0000560cf5b3d6ff EvaluateStmt((anonymous namespace)::StmtResult&, (anonymous namespace)::EvalInfo&, clang::Stmt const*, clang::SwitchCase const*) (.part.0) ExprConstant.cpp:0:0
#62 0x0000560cf5b67e4f HandleConstructorCall(clang::Expr const*, (anonymous namespace)::LValue const&, (anonymous namespace)::CallRef, clang::CXXConstructorDecl const*, (anonymous namespace)::EvalInfo&, clang::APValue&) ExprConstant.cpp:0:0
#63 0x0000560cf5b68217 HandleConstructorCall(clang::Expr const*, (anonymous namespace)::LValue const&, llvm::ArrayRef<clang::Expr const*>, clang::CXXConstructorDecl const*, (anonymous namespace)::EvalInfo&, clang::APValue&) ExprConstant.cpp:0:0
#64 0x0000560cf5b68453 (anonymous namespace)::RecordExprEvaluator::VisitCXXConstructExpr(clang::CXXConstructExpr const*, clang::QualType) ExprConstant.cpp:0:0
#65 0x0000560cf5b6c8b5 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::RecordExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#66 0x0000560cf5b6cfd2 clang::StmtVisitorBase<llvm::make_const_ptr, (anonymous namespace)::RecordExprEvaluator, bool>::Visit(clang::Stmt const*) ExprConstant.cpp:0:0
#67 0x0000560cf5b6e594 EvaluateRecord(clang::Expr const*, (anonymous namespace)::LValue const&, clang::APValue&, (anonymous namespace)::EvalInfo&) ExprConstant.cpp:0:0
#68 0x0000560cf5b38225 EvaluateInPlace(clang::APValue&, (anonymous namespace)::EvalInfo&, (anonymous namespace)::LValue const&, clang::Expr const*, bool) ExprConstant.cpp:0:0
#69 0x0000560cf5b73029 clang::Expr::EvaluateAsInitializer(clang::APValue&, clang::ASTContext const&, clang::VarDecl const*, llvm::SmallVectorImpl<std::pair<clang::SourceLocation, clang::PartialDiagnostic>>&, bool) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x75b4029)
#70 0x0000560cf5a43f02 clang::VarDecl::evaluateValueImpl(llvm::SmallVectorImpl<std::pair<clang::SourceLocation, clang::PartialDiagnostic>>&, bool) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x7484f02)
#71 0x0000560cf5a44230 clang::VarDecl::checkForConstantInitialization(llvm::SmallVectorImpl<std::pair<clang::SourceLocation, clang::PartialDiagnostic>>&) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x7485230)
#72 0x0000560cf4ce215d clang::Sema::CheckCompleteVariableDeclaration(clang::VarDecl*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x672315d)
#73 0x0000560cf4cf830a clang::Sema::AddInitializerToDecl(clang::Decl*, clang::Expr*, bool) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x673930a)
#74 0x0000560cf49b37d6 clang::Parser::ParseDeclarationAfterDeclaratorAndAttributes(clang::Declarator&, clang::Parser::ParsedTemplateInfo const&, clang::Parser::ForRangeInit*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x63f47d6)
#75 0x0000560cf49c05c8 clang::Parser::ParseDeclGroup(clang::ParsingDeclSpec&, clang::DeclaratorContext, clang::ParsedAttributes&, clang::SourceLocation*, clang::Parser::ForRangeInit*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x64015c8)
#76 0x0000560cf498d4b1 clang::Parser::ParseDeclOrFunctionDefInternal(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec&, clang::AccessSpecifier) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x63ce4b1)
#77 0x0000560cf498dd6f clang::Parser::ParseDeclarationOrFunctionDefinition(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*, clang::AccessSpecifier) (.part.0) Parser.cpp:0:0
#78 0x0000560cf4994711 clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x63d5711)
#79 0x0000560cf4995086 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x63d6086)
#80 0x0000560cf4988e2a clang::ParseAST(clang::Sema&, bool, bool) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x63c9e2a)
#81 0x0000560cf348b9a8 clang::CodeGenAction::ExecuteAction() (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x4ecc9a8)
#82 0x0000560cf2cd5b79 clang::FrontendAction::Execute() (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x4716b79)
#83 0x0000560cf2c5b4c6 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x469c4c6)
#84 0x0000560cf2dba2a6 clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x47fb2a6)
#85 0x0000560cef6b40ed cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x10f50ed)
#86 0x0000560cef6afdea ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) driver.cpp:0:0
#87 0x0000560cf2aba23d void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const::'lambda'()>(long) Job.cpp:0:0
#88 0x0000560cf215a8a0 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x3b9b8a0)
#89 0x0000560cf2aba85f clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const (.part.0) Job.cpp:0:0
#90 0x0000560cf2a819ec clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x44c29ec)
#91 0x0000560cf2a8247d clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&, bool) const (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x44c347d)
#92 0x0000560cf2a8a55d clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x44cb55d)
#93 0x0000560cef6b234a clang_main(int, char**, llvm::ToolContext const&) (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x10f334a)
#94 0x0000560cef5b85c5 main (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0xff95c5)
#95 0x00007fe8bcc99083 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24083)
#96 0x0000560cef6aabce _start (/opt/compiler-explorer/clang-assertions-trunk/bin/clang+++0x10ebbce)
clang++: error: clang frontend command failed with exit code 134 (use -v to see invocation)
Compiler returned: 134

llvmbot · 2023-06-27T19:32:09Z

@llvm/issue-subscribers-clang-frontend

shafik · 2023-06-27T23:15:52Z

I was going to say the problem in findSubobject(...) is the type of Index is

    uint64_t Index = Sub.Entries[I].getAsArrayIndex();

but the signature of expandArray(...) takes unsigned

static void expandArray(APValue &Array, unsigned Index)

but after testing this it still fails, so when we calculate the Index we are evaluating it as unsigned someplace else as well.

shafik · 2023-06-28T00:37:36Z

So in HandleOperatorNewCall(...) we get here:

  QualType AllocType = Info.Ctx.getConstantArrayType(ElemType, Size, nullptr,
                                                     ArrayType::Normal, 0);
  APValue *Val = Info.createHeapAlloc(E, AllocType, Result);
  *Val = APValue(APValue::UninitArray(), 0, Size.getZExtValue());
  Result.addArray(Info, E, cast<ConstantArrayType>(AllocType));

The result of Size.getZExtValue() is correctly 1099511627777 but APValue constructor takes unsigned

APValue(UninitArray, unsigned InitElts, unsigned Size)

I think this is by design since we actually need to allocate this array.

I think maybe we need to diagnose in HandleOperatorNewCall(...) ?

CC @zygoloid @AaronBallman

shafik · 2023-06-28T02:19:10Z

Later on when we are in findCompleteObject(...) we end up here:

 } else if (DynamicAllocLValue DA = LVal.Base.dyn_cast<DynamicAllocLValue>()) {
    std::optional<DynAlloc *> Alloc = Info.lookupDynamicAlloc(DA);
    if (!Alloc) {
      Info.FFDiag(E, diag::note_constexpr_access_deleted_object) << AK;
      return CompleteObject();
    }
    return CompleteObject(LVal.Base, &(*Alloc)->Value,
                          LVal.Base.getDynamicAllocType());
  } else {

DynamicAllocLValue only stores unsigned and DynAlloc uses APValue.

So size being unsigned seems pretty baked in.

AaronBallman · 2023-06-28T12:56:50Z

Yeah, I think this is going to be tricky to solve correctly (start threading through larger types, which may negatively impact performance for the common case of more reasonably sized allocations) and a reasonable stopgap solution would be to diagnose rather than try to perform the allocation.

Curiously, there's no stated implementation limit for memory allocations in a constant evaluation.

cor3ntin · 2023-06-29T08:00:25Z

Maybe generous reading:

Allocations can fail http://eel.is/c++draft/basic.stc.dynamic#allocation-2.sentence-4
If it does, in this case, new would throw, which would make this not a constant expression. So I think there is already allowance for emitting a diag in this case (and subsequently not constant fold).

cor3ntin · 2023-07-02T08:52:14Z

This turns out to be... complicated.
This specific issue is easy enough, to fix, I'll probably make a patch.

But let's assume we allocate an array of size smaller than UINT_MAX
We are going to end up there:

llvm-project/clang/lib/AST/APValue.cpp

Line 295 in ff286d7

Elts(new APValue[NumElts + (NumElts != Size ? 1 : 0)]),

This is likely to fail for large arrays - APValue seems to be 3x sizeof(void*).
And there no way that i can see to handle allocation failure, it's going to...crash.

There is a FIXME just above to say we should not use malloc but I'm not familiar enough with LLVM facilities to try to see if we could use some sort of allocator that fails gracefully.

But that's not all, in non-constant expression, new expression with a large constant crash in the presence
of an initializers #63637, seemingly also because clang trades in unsigned sizes.

cor3ntin · 2023-07-03T08:23:41Z

More spit balling.
I'm wondering whether this is fixable at all.

We should fix APValue::Arr construction by making sure we can handle allocation failure - we need to be slightly careful because there is a new error handler setup at startup so we'll need a separate allocation function for that.

But assuming we do that, we can still end up in a situation where we can allocate successfully but later we get a segfault because of overcommitment.

And this seems both problematic and unavoidable. Of course, during constant evaluation running out of resources failing in funny ways is not completely unexpected.

But during constant folding, failing for something that would otherwise be fine at runtime is... not ideal.
Should we put a low limit (maybe something like a few megabytes? Would that need a flag?) on allocation sizes during constant folding so that we limit how often we get into a situation where valid programs can't compile?

I thought about querying the available memory in the system but that also does seem like a non-solution to me in a scenario where many compile jobs runs at the same times such that physical memory availability could change a lot between the moment the array is initialized and the moment it is used.

I also thought about alternative solution like using sparse vectors to store APValues but that is probably not useful in real code that actually tries to use the array.

AaronBallman · 2023-07-03T13:56:56Z

I'm wondering whether this is fixable at all.

I think it is, but it could very well be a slog. I don't think anyone ever expected WG21 to go this far with what "constant expression" means.

But assuming we do that, we can still end up in a situation where we can allocate successfully but later we get a segfault because of overcommitment.

We might want to consider an "extremely large allocation allocator" that uses lower-level OS facilities for performing the allocations so that we can catch these faults and use them to mark the constant expression as invalid at that point?

But during constant folding, failing for something that would otherwise be fine at runtime is... not ideal.
Should we put a low limit (maybe something like a few megabytes? Would that need a flag?) on allocation sizes during constant folding so that we limit how often we get into a situation where valid programs can't compile?

If the host system faults at it, there's no reason to presume the target system wouldn't as well. So either outcome is not really ideal. Limiting the amount of memory we can allocate could be interesting, but the limit needs to be on total allocations rather than per-allocation, because there's not much of a difference between one 10GB allocation and 10 one-GB allocations. We already have the notion of "constexpr steps" for limiting complexity, we could consider adding "constexpr memory" for the same purposes, but it'd be more user-friendly for that limit to be somewhere below the actual system resources. e.g., we don't want to take 100% of the system resources (the compiler can't easily report diagnostics at that point and other things on the system may start to fail), but we don't want to use 10MB of someone's machine with 1TB of RAM either.

I thought about querying the available memory in the system but that also does seem like a non-solution to me in a scenario where many compile jobs runs at the same times such that physical memory availability could change a lot between the moment the array is initialized and the moment it is used.

Agreed, I don't think this would be viable. I think we'd basically need to catch the faults ourselves as they occur if we wanted to use the full host system resources for this.

cor3ntin · 2023-07-03T14:18:46Z

We might want to consider an "extremely large allocation allocator" that uses lower-level OS facilities for performing the allocations so that we can catch these faults and use them to mark the constant expression as invalid at that point?

Maybe... but that sounds like a pretty big project.

If the host system faults at it, there's no reason to presume the target system wouldn't as well. So either outcome is not really ideal.

I don't really agree with that.

For example, on my system I can allocate a huge vector of char, if only an application is running, but I can't necessarily allocate an identically sized array of APValue - which is 24 times-ish as big, while running many parallel compile jobs.
And realistically, I expect HPC applications in particular to make allocations many times bigger than what a dev is going to run on their system.

We should not get into a situation when the compiler tries so hard to constant fold arbitrary expression that it crashes.

Limiting the amount of memory we can allocate could be interesting, but the limit needs to be on total allocations rather than per-allocation, because there's not much of a difference between one 10GB allocation and 10 one-GB allocations.

Well, depending on fragmentation that may not be true, but i generally agree that we should count it all.

We already have the notion of "constexpr steps" for limiting complexity, we could consider adding "constexpr memory" for the same purposes, but it'd be more user-friendly for that limit to be somewhere below the actual system resources. e.g., we don't want to take 100% of the system resources (the compiler can't easily report diagnostics at that point and other things on the system may start to fail), but we don't want to use 10MB of someone's machine with 1TB of RAM either.

And that sounds tricky to get right. Again, i think the common use case is to run as many compile jobs as there are core, or thread. And as clang compiles a TU it will probably eat a few GB for AST nodes.

I think that as a first approach limiting to maybe 10 or 100 MB constant folding, and letting constant expression evaluation use as much as it asks until it crashes, is probably reasonable.

AaronBallman · 2023-07-04T13:03:59Z

We might want to consider an "extremely large allocation allocator" that uses lower-level OS facilities for performing the allocations so that we can catch these faults and use them to mark the constant expression as invalid at that point?

Maybe... but that sounds like a pretty big project.

Agreed.

If the host system faults at it, there's no reason to presume the target system wouldn't as well. So either outcome is not really ideal.

I don't really agree with that.

For example, on my system I can allocate a huge vector of char, if only an application is running, but I can't necessarily allocate an identically sized array of APValue - which is 24 times-ish as big, while running many parallel compile jobs. And realistically, I expect HPC applications in particular to make allocations many times bigger than what a dev is going to run on their system.

We should not get into a situation when the compiler tries so hard to constant fold arbitrary expression that it crashes.

Agreed that we should not be constant folding until we crash, but still disagreed on the host vs target distinction. There's no relationship between host and target in general, so it's always going to be situational whether host or target has more resources available or not.

Limiting the amount of memory we can allocate could be interesting, but the limit needs to be on total allocations rather than per-allocation, because there's not much of a difference between one 10GB allocation and 10 one-GB allocations.

Well, depending on fragmentation that may not be true, but i generally agree that we should count it all.

True, I'm handwaving there. :-)

We already have the notion of "constexpr steps" for limiting complexity, we could consider adding "constexpr memory" for the same purposes, but it'd be more user-friendly for that limit to be somewhere below the actual system resources. e.g., we don't want to take 100% of the system resources (the compiler can't easily report diagnostics at that point and other things on the system may start to fail), but we don't want to use 10MB of someone's machine with 1TB of RAM either.

And that sounds tricky to get right. Again, i think the common use case is to run as many compile jobs as there are core, or thread. And as clang compiles a TU it will probably eat a few GB for AST nodes.

I think that as a first approach limiting to maybe 10 or 100 MB constant folding, and letting constant expression evaluation use as much as it asks until it crashes, is probably reasonable.

I think it makes more sense to limit both constant folding and constant expression evaluation. Crashing is never a reasonable behavior for the compiler. To me, the big question is: do you limit in a way that's automatic (based on memory on the system) or do you limit with a hard-coded/user-controlled value? I think user-controlled is probably the right approach, but I think we may want to set the default based on system resources. It'd be kind of silly to pick 10MB for everyone given that some folks have significantly larger systems, but it'd be pretty mean to pick 10GB for everyone given that some folks have significantly smaller systems. Perhaps (Amount of RAM / # of Cores) - Wiggle Room could be a somewhat reasonable default?

cor3ntin · 2023-07-04T14:09:42Z

Agreed that we should not be constant folding until we crash, but still disagreed on the host vs target distinction. There's no relationship between host and target in general, so it's always going to be situational whether host or target has more resources available or not.

Even though, if constant evaluation fails it fails, the only thing not crashing does improve for users are diagnostics.
But if what is in effect a runtime allocation crashes at compile time, we are preventing the runtime the opportunity to run (and fail, or not).

I think there is a QOI question: crashing on constant evaluation isn't great, and hard to solve. Crashing on constant folding is making perfectly well-formed program fail to compile, and therefore an important bug.

What are the users expectations in terms of how much the compiler should try to constant fold?
I don't expect such large objects to be kind on compile times if they were to be supported.

And again... constant folding is allocating APValues which are usually much bigger than the type they represent.

I did some research, as you said, the only way to deal with that perfectly seem to to build an allocator on top of mmap, which... well, do we have the resources to do that?
There may be a somewhat simple solution to allocate all arrays of appvalue greater than some threshold on a mmap-ed buffer, which we'd release after. that would let us test for over-commitment. Trying to recycle pages seem... above my paygrade. (there are probably usable solution like https://github.com/microsoft/mimalloc, but that also seems overkill)

Or can we first go with a limit for constant folding (to fix the bug) and improve later?

Perhaps (Amount of RAM / # of Cores) - Wiggle Room could be a somewhat reasonable default?

I usually count about 2/3 GB per clang process. So with 64GB I run 32 jobs in parallel - at best. That calculation doesn't account for the fact that some header included in all TUs might try to allocate 20GB worth of APValues. so that would not work for me, i suspect most folks do have similar-ish expectations.

AaronBallman · 2023-07-05T12:37:51Z

Agreed that we should not be constant folding until we crash, but still disagreed on the host vs target distinction. There's no relationship between host and target in general, so it's always going to be situational whether host or target has more resources available or not.

Even though, if constant evaluation fails it fails, the only thing not crashing does improve for users are diagnostics. But if what is in effect a runtime allocation crashes at compile time, we are preventing the runtime the opportunity to run (and fail, or not).

That's a matter of QoI -- we've got implementation limits we can use to ensure we don't crash at compile time. Yes, this prevents the runtime from having its chance to run -- it's a tradeoff between user experiences.

I think there is a QOI question: crashing on constant evaluation isn't great, and hard to solve. Crashing on constant folding is making perfectly well-formed program fail to compile, and therefore an important bug.

"Perfectly well-formed program" is a bit of a stretch -- in all of these cases, I think we're hitting implementation resource limits, it's just a matter of which limits we hit (translation time limits or runtime limits). That said, I don't think either should crash.

What are the users expectations in terms of how much the compiler should try to constant fold? I don't expect such large objects to be kind on compile times if they were to be supported.

I think users expect constant folding to "just work". ;-) More helpfully, I think we should constant fold up until we realize we're doing more harm than good, then fall back to letting the runtime have a shot at it. Ideally, we'd do this without requiring the user to tell us the point at which we're doing too much harm -- though it would be reasonable for us to have a flag to say "try harder than you've been trying" that users could optionally use.

And again... constant folding is allocating APValues which are usually much bigger than the type they represent.

I did some research, as you said, the only way to deal with that perfectly seem to to build an allocator on top of mmap, which... well, do we have the resources to do that? There may be a somewhat simple solution to allocate all arrays of appvalue greater than some threshold on a mmap-ed buffer, which we'd release after. that would let us test for over-commitment. Trying to recycle pages seem... above my paygrade. (there are probably usable solution like https://github.com/microsoft/mimalloc, but that also seems overkill)

Or can we first go with a limit for constant folding (to fix the bug) and improve later?

I think it's reasonable to limit constant folding now and then improve the allocation mechanisms later.

Perhaps (Amount of RAM / # of Cores) - Wiggle Room could be a somewhat reasonable default?

I usually count about 2/3 GB per clang process. So with 64GB I run 32 jobs in parallel - at best. That calculation doesn't account for the fact that some header included in all TUs might try to allocate 20GB worth of APValues. so that would not work for me, i suspect most folks do have similar-ish expectations.

Fair!

cor3ntin · 2023-07-05T16:00:28Z

I did more research, I think i have a potential solution to guarantee diagnostics, although i don't have a prototype yet:

When allocating an array of app value, we could:

Use a custom new that
bypasses the get_new_handler set by llvm / call malloc directly
use mlock to try to commit the page - if that fails we can recover nicely with diags on posix systems in which it is available

And upon deleting we can unlock the pages.

The main drawbacks that i can see

mlock will fail on some memory limit that is less than half of the system memory and can be lower based on system configurations
The locked pages won't swap out, but i think that's a non-issue as APValue are, hopefully short lived. They may end up in a template parameter i suppose but even though... if your system swaps during compilation you probably have bigger issues.

Of course this only viable on posix but on windows over-commit is a non-issue so i think it works everywhere.

AaronBallman · 2023-07-07T13:02:27Z

This could be a reasonable approach, it seems worth exploring if you're interested.

The locked pages won't swap out, but i think that's a non-issue as APValue are, hopefully short lived. They may end up in a template parameter i suppose but even though... if your system swaps during compilation you probably have bigger issues.

However, I think this might be more problematic than you'd expect. In general, I think APValue will live as long as the AST lives because it's used for things like non-type template arguments, attribute arguments, constant expressions, etc. I think we'd need some actual performance numbers to evaluate the tradeoffs. :-(

shafik · 2023-07-07T20:15:26Z

I think it might be useful to diagnose these limitation for the time being and perhaps link to this discussion in a FIXME. I don't see a quick fix here but eliminating crashes seems like a pragmatic plus and at least documents that we know this is an issue.

cor3ntin · 2023-07-10T15:58:13Z

After chatting with @AaronBallman, we think the short term fix is to

Fix the overflow by failing allocation larger than UINT_MAX
Prevent constant folding for Very Large Arrays, so that large allocations would not fail in that case - we would let constant expression evaluation crash - for the time being

The question remain what is a reasonable limit. My intuition is that past a certain point, it doesn't really make sense to try to constant fold as it would drastically impact compile times and is likely to fail.

@tbaederr we were also wondering how the new interpreter behave in that case, and wanted to make sure you are aware of the issue

tbaederr · 2023-07-11T06:55:39Z

I thought the current interpreter has some notion of "steps" it executes and a limit for those; does that limit not trigger when constructors of too large arrays are being executed?

(I was planning to add a similar limit for bytecode operations executed, which would trigger in this case).

cor3ntin · 2023-07-11T11:49:59Z

I thought the current interpreter has some notion of "steps" it executes and a limit for those; does that limit not trigger when constructors of too large arrays are being executed?

No, but this is an interesting idea, should we consider the initialization of each element to be a step, and use that?
It would have some benefits in that adding an option to control the memory isn't easy from an UX perspective (users have no sense of how much memory an APVAlue uses).

On the other hand, by linking the two we would certainly set a lower bound than we need to.
On the other other hand, any subsequent traversal or manipulation of the array does count as steps, so even if we can allow larger arrays to be created, we could not... use them. So maybe that's something to explore.

shafik · 2023-07-15T02:09:50Z

The topic of step limits came up and this bug seems to be show some inconsistency with how we handle limits in some cases: #49064 (comment)

maybe worth digging into more.

cor3ntin · 2023-07-21T19:06:59Z

@brutalsavage was that found by fuzzing btw?

brutalsavage · 2023-07-27T02:58:14Z

@cor3ntin Sorry for the late reply, it was found by fuzzing.

This is a temporary fix (for clang 17) that caps the size of any array we try to constant evaluate: There are 2 limits: * We cap to UINT_MAX the size of ant constant evaluated array, because the constant evaluator does not support size_t. * We cap to `-fconstexpr-steps` elements the size of each individual array and dynamic array allocations. This works out because the number of constexpr steps already limits how many array elements can be initialized, which makes this new limit conservatively generous. This ensure that the compiler does not crash when attempting to constant-fold valid programs. If the limit is reached by a given array, constant evaluation will fail, and the program will be ill-formed, until a bigger limit is given. Or, constant folding will fail and the array will be evaluated at runtime. Fixes llvm#63562 Reviewed By: efriedma Differential Revision: https://reviews.llvm.org/D155955

github-actions bot added the new issue label Jun 27, 2023

EugeneZelenko added clang:frontend Language frontend issues, e.g. anything involving "Sema" crash Prefer [crash-on-valid] or [crash-on-invalid] and removed new issue labels Jun 27, 2023

Endilll added the confirmed Verified by a second party label Jun 28, 2023

shafik mentioned this issue Jul 10, 2023

Clang frontend crash on constexpr libc++ vector::insert #63761

Closed

cor3ntin closed this as completed in 45ab2b4 Jul 28, 2023

zyn0217 mentioned this issue Oct 28, 2023

Slow AST builds in code involving large array (e.g. 1000000 elements) clangd/clangd#1808

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clang frontend C++ crash when initializing a large vector #63562

Clang frontend C++ crash when initializing a large vector #63562

brutalsavage commented Jun 27, 2023

brutalsavage commented Jun 27, 2023

llvmbot commented Jun 27, 2023

shafik commented Jun 27, 2023

shafik commented Jun 28, 2023

shafik commented Jun 28, 2023 •

edited

Loading

AaronBallman commented Jun 28, 2023

cor3ntin commented Jun 29, 2023

cor3ntin commented Jul 2, 2023

cor3ntin commented Jul 3, 2023

AaronBallman commented Jul 3, 2023

cor3ntin commented Jul 3, 2023 •

edited

Loading

AaronBallman commented Jul 4, 2023

cor3ntin commented Jul 4, 2023

AaronBallman commented Jul 5, 2023

cor3ntin commented Jul 5, 2023

AaronBallman commented Jul 7, 2023

shafik commented Jul 7, 2023

cor3ntin commented Jul 10, 2023

tbaederr commented Jul 11, 2023

cor3ntin commented Jul 11, 2023

shafik commented Jul 15, 2023

cor3ntin commented Jul 21, 2023

brutalsavage commented Jul 27, 2023

Clang frontend C++ crash when initializing a large vector #63562

Clang frontend C++ crash when initializing a large vector #63562

Comments

brutalsavage commented Jun 27, 2023

brutalsavage commented Jun 27, 2023

llvmbot commented Jun 27, 2023

shafik commented Jun 27, 2023

shafik commented Jun 28, 2023

shafik commented Jun 28, 2023 • edited Loading

AaronBallman commented Jun 28, 2023

cor3ntin commented Jun 29, 2023

cor3ntin commented Jul 2, 2023

cor3ntin commented Jul 3, 2023

AaronBallman commented Jul 3, 2023

cor3ntin commented Jul 3, 2023 • edited Loading

AaronBallman commented Jul 4, 2023

cor3ntin commented Jul 4, 2023

AaronBallman commented Jul 5, 2023

cor3ntin commented Jul 5, 2023

AaronBallman commented Jul 7, 2023

shafik commented Jul 7, 2023

cor3ntin commented Jul 10, 2023

tbaederr commented Jul 11, 2023

cor3ntin commented Jul 11, 2023

shafik commented Jul 15, 2023

cor3ntin commented Jul 21, 2023

brutalsavage commented Jul 27, 2023

shafik commented Jun 28, 2023 •

edited

Loading

cor3ntin commented Jul 3, 2023 •

edited

Loading