clangd crashes when setting up compilation database via initialization options

[ckandeler]

Our client (Qt Creator) currently creates a compilation database file and passes its path to clangd via the --compile-commands-dir command line switch. We'd like to skip creating that file and pass the commands to clangd directly via initializationOptions.compilationDatabaseChanges. In addition, because we still want to keep control over the index location, we also use initializationOptions.compilationDatabasePath, which points to a minimal JSON file containing an empty array. It is my understanding that these get merged into a single compilation database.
However, after doing this, clangd often crashes, apparently while setting up the background indexing progress message. The stack trace always looks like this:

#0 0x00007f5ad2633e82 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /sda/home/christian/dev/llvm/llvm/lib/Support/Unix/Signals.inc:569:3
 #1 0x00007f5ad2631f64 llvm::sys::RunSignalHandlers() /sda/home/christian/dev/llvm/llvm/lib/Support/Signals.cpp:97:20
 #2 0x00007f5ad26320e6 SignalHandler(int) /sda/home/christian/dev/llvm/llvm/lib/Support/Unix/Signals.inc:407:1
 #3 0x00007f5ad7a5c870 __restore_rt sigaction.c:0:0
 #4 0x00005567ff8b81ab llvm::detail::UniqueFunctionBase<void, clang::clangd::WorkDoneProgressCreateParams const&, llvm::unique_function<void (llvm::Expected<std::nullptr_t>)> >::getTrivialCallback() const <:180:82
 #5 0x00005567ff8b81ab llvm::detail::UniqueFunctionBase<void, clang::clangd::WorkDoneProgressCreateParams const&, llvm::unique_function<void (llvm::Expected<std::nullptr_t>)> >::getCallPtr() const /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:189:52
 #6 0x00005567ff8b81ab llvm::unique_function<void (clang::clangd::WorkDoneProgressCreateParams const&, llvm::unique_function<void (llvm::Expected<std::nullptr_t>)>)>::operator()(clang::clangd::WorkDoneProgressCreateParams const&, llvm::unique_function<void (llvm::Expected<std::nullptr_t>)>) /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:384:28
 #7 0x00005567ff8b81ab clang::clangd::ClangdLSPServer::onBackgroundIndexProgress(clang::clangd::BackgroundQueue::Stats const&) /sda/home/christian/dev/llvm/clang-tools-extra/clangd/ClangdLSPServer.cpp:1638:27
 #8 0x00005567ff8e5155 std::_Function_handler<void (clang::clangd::BackgroundQueue::Stats), clang::clangd::ClangdServer::ClangdServer(clang::clangd::GlobalCompilationDatabase const&, clang::clangd::ThreadsafeFS const&, clang::clangd::ClangdServer::Options const&, clang::clangd::ClangdServer::Callbacks*)::'lambda0'(clang::clangd::BackgroundQueue::Stats)>::_M_invoke(std::_Any_data const&, clang::clangd::BackgroundQueue::Stats&&) /usr/include/c++/11.1.0/bits/std_function.h:292:44
 #9 0x00005567ffbb3801 std::unique_lock<std::mutex>::~unique_lock() /usr/include/c++/11.1.0/bits/unique_lock.h:103:2
#10 0x00005567ffbb3801 clang::clangd::BackgroundQueue::work(std::function<void ()>) /sda/home/christian/dev/llvm/clang-tools-extra/clangd/index/BackgroundQueue.cpp:37:5
#11 0x00005567ffba53f1 std::_Function_base::~_Function_base() /usr/include/c++/11.1.0/bits/std_function.h:244:11
#12 0x00005567ffba53f1 std::function<void ()>::~function() /usr/include/c++/11.1.0/bits/std_function.h:328:11
#13 0x00005567ffba53f1 operator() /sda/home/christian/dev/llvm/clang-tools-extra/clangd/index/Background.cpp:114:37
#14 0x00005567ffba53f1 void llvm::detail::UniqueFunctionBase<void>::CallImpl<clang::clangd::BackgroundIndex::BackgroundIndex(clang::clangd::ThreadsafeFS const&, clang::clangd::GlobalCompilationDatabase const&, llvm::unique_function<clang::clangd::BackgroundIndexStorage* (llvm::StringRef)>, clang::clangd::BackgroundIndex::Options)::'lambda0'()>(void*) /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:222:16
#15 0x00005567ffc1cbe2 llvm::PointerIntPair<llvm::PointerUnion<llvm::detail::UniqueFunctionBase<void>::TrivialCallback*, llvm::detail::UniqueFunctionBase<void>::NonTrivialCallbacks*>, 1u, bool, llvm::PointerLikeTypeTraits<llvm::PointerUnion<llvm::detail::UniqueFunctionBase<void>::TrivialCallback*, llvm::detail::UniqueFunctionBase<void>::NonTrivialCallbacks*> >, llvm::PointerIntPairInfo<llvm::PointerUnion<llvm::detail::UniqueFunctionBase<void>::TrivialCallback*, llvm::detail::UniqueFunctionBase<void>::NonTrivialCallbacks*>, 1u, llvm::PointerLikeTypeTraits<llvm::PointerUnion<llvm::detail::UniqueFunctionBase<void>::TrivialCallback*, llvm::detail::UniqueFunctionBase<void>::NonTrivialCallbacks*> > > >::getPointer() const /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/PointerIntPair.h:59:58
#16 0x00005567ffc1cbe2 llvm::detail::UniqueFunctionBase<void>::~UniqueFunctionBase() /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:284:42
#17 0x00005567ffc1cbe2 llvm::detail::UniqueFunctionBase<void>::operator=(llvm::detail::UniqueFunctionBase<void>&&) /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:335:30
#18 0x00005567ffc1cbe2 llvm::detail::UniqueFunctionBase<void>::operator=(llvm::detail::UniqueFunctionBase<void>&&) /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:328:23
#19 0x00005567ffc1cbe2 llvm::unique_function<void ()>::operator=(llvm::unique_function<void ()>&&) /sda/home/christian/dev/llvm/llvm/include/llvm/ADT/FunctionExtras.h:372:20
#20 0x00005567ffc1cbe2 operator() /sda/home/christian/dev/llvm/clang-tools-extra/clangd/support/Threading.cpp:95:14
#21 0x00005567ffc1cbe2 Apply<clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function<void()>)::<lambda()> > /sda/home/christian/dev/llvm/llvm/include/llvm/Support/thread.h:42:35
#22 0x00005567ffc1cbe2 GenericThreadProxy<std::tuple<clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function<void()>)::<lambda()> > > /sda/home/christian/dev/llvm/llvm/include/llvm/Support/thread.h:50:10
#23 0x00005567ffc1cbe2 void* llvm::thread::ThreadProxy<std::tuple<clang::clangd::AsyncTaskRunner::runAsync(llvm::Twine const&, llvm::unique_function<void ()>)::'lambda0'()> >(void*) /sda/home/christian/dev/llvm/llvm/include/llvm/Support/thread.h:60:36
#24 0x00007f5ad7a52259 start_thread pthread_create.c:0:0
#25 0x00007f5ad10c95e3 clone (/usr/lib/libc.so.6+0xfe5e3)

stderr output from clangd --verbose:

I[11:40:33.122] clangd version 14.0.0 (https://github.com/llvm/llvm-project.git 734ec0b21c5a73d7372ad1f9157d85b519547aa4)
I[11:40:33.122] Features: linux
I[11:40:33.122] PID: 1978599
I[11:40:33.122] Working directory: /sda/home/christian/dev/qtcreator/super-master/Debug_System_Q_2df610fdb9294b86/install-root/usr/local/bin
I[11:40:33.122] argv[0]: /sda/home/christian/dev/llvm-master-install-release/bin/clangd
I[11:40:33.122] argv[1]: --background-index
I[11:40:33.122] argv[2]: --header-insertion=never
I[11:40:33.122] argv[3]: --limit-results=0
I[11:40:33.122] argv[4]: --limit-references=0
I[11:40:33.122] argv[5]: --clang-tidy=0
I[11:40:33.122] Starting LSP over stdin/stdout
I[11:40:33.122] <-- initialize("{a3b99ed6-a4f2-47b0-9605-a65cb809d8f8}")
I[11:40:33.122] Loaded compilation database from /tmp/QtCreator-bEBwOF/build-highlighting-Desktop-Debug/.qtc_clangd/compile_commands.json
I[11:40:33.123] --> reply:initialize("{a3b99ed6-a4f2-47b0-9605-a65cb809d8f8}") 0 ms
I[11:40:33.123] <-- initialized
I[11:40:33.124] <-- textDocument/didOpen
I[11:40:33.125] <-- textDocument/didClose
I[11:40:33.125] ASTWorker building file /tmp/QtCreator-bEBwOF/qtcreator-tests-NxwdvK/highlighting.cpp version 0 with command 
[/tmp/QtCreator-bEBwOF/build-highlighting-Desktop-Debug/.qtc_clangd]
/usr/lib/icecream/bin/clang -I /tmp/QtCreator-bEBwOF/clang-uiheader-MZNUUX -nostdinc -nostdinc++ -pipe -g -Wall -Wextra -fPIC -fsyntax-only -m64 --target=x86_64-pc-linux-gnu -std=gnu++17 -DQ_CREATOR_RUN -DQT_QML_DEBUG -I /tmp/QtCreator-bEBwOF/qtcreator-tests-NxwdvK -I /tmp/QtCreator-bEBwOF/build-highlighting-Desktop-Debug -I /usr/lib/qt/mkspecs/linux-g++ -isystem /usr/include/c++/11.1.0 -isystem /usr/include/c++/11.1.0/x86_64-pc-linux-gnu -isystem /usr/include/c++/11.1.0/backward -isystem /usr/local/include -isystem /usr/lib/clang/12.0.1/include -isystem /usr/include -fmessage-length=0 -fdiagnostics-show-note-include-stack -fretain-comments-from-system-headers -fmacro-backtrace-limit=0 -ferror-limit=1000 -x c++ -Wno-unknown-pragmas -Wno-unknown-warning-option -Wno-documentation-unknown-command -resource-dir=/sda/home/christian/dev/llvm-master-install-release/lib/clang/14.0.0 -- /tmp/QtCreator-bEBwOF/qtcreator-tests-NxwdvK/highlighting.cpp

(followed by the stack trace above)
The concrete file content does not seem to matter.

[kadircet]

can you invoke clangd with --log=verbose and provide the logs again ? (unclear how helpful they'll be, but still.)

As you mentioned it looks like this is happening as soon as clangd tries to invoke the unique_function stored at CreateWorkdoneProgress. I've got a feeling that this has nothing to do with compilation database passed during initialization (it just makes the memory alignment perfect for a crash). Any chance you can get a coredump from the crashing process?

It would also be worth trying to build clangd with a different compiler (if the crash goes away, it's probably a UB/bug inside the unique_function/compiler).

[ckandeler]

verbose-log.txt
stack_trace_from_core_file.txt

I've attached the verbose log and the full stack trace from the core dump. Let me know if you want me to examine the core dump in some other way.

[ckandeler]

Rebuilding with clang (instead of gcc) did not help, and neither did using libc++.

[kadircet]

It'd be great if you could print the state of ClangdLSPServer::CreateWorkDoneProgress at crash and after https://github.com/llvm/llvm-project/tree/main/clang-tools-extra/clangd/ClangdLSPServer.cpp#L1491

[ckandeler]

pretty_printed_state_at_onBackgroundIndexProgress.txt

Here's the pretty-printed state of onBackgroundIndexProgress() right before the crashing call to CreateWorkDoneProgress() (as shown in Qt Creator's debug window). Regarding the state after binding: Since live debugging will probably a bit cumbersome (e.g. because I don't know in advance whether this instance will crash), I suppose inserting logging statements is the way to go here. Can you provide a small patch or code snippet that prints the values you are interested in?

[sam-mccall]

Oh dear, this one is silly and my fault...

On startup we bind "initialize" only, and delay binding the rest of LSP until initialization.
However in onInitialize() we call applyConfiguration() before bindMethods().
If you configure a compile-commands in the initialization options, we immediately start indexing and try to tell the client. If we get there before bindMethods(), CreateWorkDoneProgress is null and we crash. I can't see any problem with just reorderding the calls there.

An aside: I assume you do want background indexing to happen for the files you specify in initializationOptions? And the list might be big? I think applyConfiguration is pretty slow for a big list of changes, we apply them one at a time, doing a bit of work and locking to enqueue each one. Worse, we do it on the main thread. This might slow down startup. We can fix it though.

[ckandeler]

An aside: I assume you do want background indexing to happen for the files you specify in initializationOptions? And the list might be big?

Yes on both counts. Once the crash is gone, I can check whether we observe a noticeably delayed startup. We'd really like to get rid of the JSON file redirection on the client side.