cquery crash: LanguageClient-neovim

[vargheseg]

cquery: 32d1c70

Facing issue similar to #185. Suspect method: initializemessage framed by the client. Raising this ticket for feed back if this assumption is correct.

(For example, initialize message has workspace:null. From specs think workspace should be optional but if present should be a WorkspaceClientCapabilities object and not null?)

LanguageClient.log

2018-01-06T23:38:47.708110617+01:00 INFO languageclient::languageclient - => {"jsonrpc":"2.0","method":"initialize","params":{"capabilities":{"experimental":null,"textDocument":{"codeAction":null,"codeLens":null,"completion":{"completionItem":{"commitCharactersSupport":null,"documentationFormat":null,"snippetSupport":false},"dynamicRegistration":null},"definition":null,"documentHighlight":null,"documentLink":null,"documentSymbol":null,"formatting":null,"hover":null,"onTypeFormatting":null,"rangeFormatting":null,"references":null,"rename":null,"signatureHelp":null,"synchronization":null},"workspace":null},"initializationOptions":{"cacheDirectory":"/tmp/cquery","compilationDatabaseDirectory":"/home/vargheseg/Projects/DeviceAccess","resourceDirectory":"/home/vargheseg/bin/cquery/lib/clang+llvm-4.0.0-x86_64-linux-gnu-ubuntu-14.04/resource-dir/"},"processId":20355,"rootPath":"/home/vargheseg/Projects/DeviceAccess","rootUri":"file:///home/vargheseg/Projects/DeviceAccess","trace":"off"},"id":7}

LanguageServer.log

cquery: ../../third_party/rapidjson/include/rapidjson/document.h:1233: MemberIterator rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::FindMember(const GenericValue<Encoding, SourceAllocator> &) [Encoding = rapidjson::UTF8<char>, Allocator = rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, SourceAllocator = rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>]: Assertion `IsObject()' failed.
0 0x00007fea6e5743c8 llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/usr/lib/llvm-4.0/lib/../lib/libLLVM-4.0.so.1+0x6d43c8)
1 0x00007fea6e572586 llvm::sys::RunSignalHandlers() (/usr/lib/llvm-4.0/lib/../lib/libLLVM-4.0.so.1+0x6d2586)
2 0x00007fea6e5726bb (/usr/lib/llvm-4.0/lib/../lib/libLLVM-4.0.so.1+0x6d26bb)
3 0x00007fea734cc150 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x13150)
4 0x00007fea70e650bb gsignal /build/glibc-CxtIbX/glibc-2.26/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
5 0x00007fea70e66f5d abort /build/glibc-CxtIbX/glibc-2.26/stdlib/abort.c:92:0
6 0x00007fea70e5cf17 __assert_fail_base /build/glibc-CxtIbX/glibc-2.26/assert/assert.c:92:0
7 0x00007fea70e5cfc2 (/lib/x86_64-linux-gnu/libc.so.6+0x2efc2)
8 0x0000000000487df6 rapidjson::GenericMemberIterator<false, rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::FindMember<rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&) /home/vargheseg/Projects/cquery/cquery/build/release/../../third_party/rapidjson/include/rapidjson/document.h:1234:9
9 0x0000000000487df6 rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::FindMember(char const*) /home/vargheseg/Projects/cquery/cquery/build/release/../../third_party/rapidjson/include/rapidjson/document.h:1213:0
10 0x0000000000487df6 void ReflectMember<std::experimental::optional<bool> >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, char const*, std::experimental::optional<bool>&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/serializer.h:196:0
11 0x00000000004886aa void Reflect<rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, lsTextDocumentClientCapabilities::lsSynchronization&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/language_server_api.h:665:1
12 0x00000000004886aa void ReflectMember<lsTextDocumentClientCapabilities::lsSynchronization>(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, char const*, lsTextDocumentClientCapabilities::lsSynchronization&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/serializer.h:199:0
13 0x00000000004884b7 void Reflect<rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, lsTextDocumentClientCapabilities&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/language_server_api.h:682:1
14 0x0000000000488478 void Reflect<lsTextDocumentClientCapabilities>(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, std::experimental::optional<lsTextDocumentClientCapabilities>&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/serializer.h:0:3
15 0x0000000000487a21 void ReflectMember<std::experimental::optional<lsTextDocumentClientCapabilities> >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, char const*, std::experimental::optional<lsTextDocumentClientCapabilities>&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/serializer.h:201:1
16 0x000000000048680a void ReflectMember<lsClientCapabilities>(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, char const*, lsClientCapabilities&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/serializer.h:201:1
17 0x0000000000486085 void Reflect<rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > >(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, lsInitializeParams&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/language_server_api.h:749:1
18 0x0000000000486085 void ReflectMember<lsInitializeParams>(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&, char const*, lsInitializeParams&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/serializer.h:199:0
19 0x000000000048479c std::_Head_base<0ul, BaseIpcMessage*, false>::_Head_base<BaseIpcMessage*&>(BaseIpcMessage*&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/tuple:133:4
20 0x000000000048479c std::_Tuple_impl<0ul, BaseIpcMessage*, std::default_delete<BaseIpcMessage> >::_Tuple_impl<BaseIpcMessage*&, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest>, void>(BaseIpcMessage*&, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/tuple:218:0
21 0x000000000048479c std::tuple<BaseIpcMessage*, std::default_delete<BaseIpcMessage> >::tuple<BaseIpcMessage*&, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest>, true>(BaseIpcMessage*&, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/tuple:972:0
22 0x000000000048479c std::__uniq_ptr_impl<BaseIpcMessage, std::default_delete<BaseIpcMessage> >::__uniq_ptr_impl<std::default_delete<(anonymous namespace)::Ipc_InitializeRequest> >(BaseIpcMessage*, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/unique_ptr.h:144:0
23 0x000000000048479c std::unique_ptr<BaseIpcMessage, std::default_delete<BaseIpcMessage> >::unique_ptr<(anonymous namespace)::Ipc_InitializeRequest, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest>, void>(std::unique_ptr<(anonymous namespace)::Ipc_InitializeRequest, std::default_delete<(anonymous namespace)::Ipc_InitializeRequest> >&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/unique_ptr.h:253:0
24 0x000000000048479c std::_Function_handler<std::unique_ptr<BaseIpcMessage, std::default_delete<BaseIpcMessage> > (rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&), MessageRegistryRegister<(anonymous namespace)::Ipc_InitializeRequest>::MessageRegistryRegister()::{lambda(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&)#1}>::_M_invoke(std::_Any_data const&, rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/std_function.h:301:0
25 0x000000000044c8fe std::function<std::unique_ptr<BaseIpcMessage, std::default_delete<BaseIpcMessage> > (rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&)>::operator()(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&) const /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/std_function.h:706:14
26 0x000000000044c8fe MessageRegistry::Parse(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >&) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/language_server_api.cc:178:0
27 0x000000000044c45b rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::Destroy() /home/vargheseg/Projects/cquery/cquery/build/release/../../third_party/rapidjson/include/rapidjson/document.h:2473:9
28 0x000000000044c45b rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>::~GenericDocument() /home/vargheseg/Projects/cquery/cquery/build/release/../../third_party/rapidjson/include/rapidjson/document.h:2151:0
29 0x000000000044c45b MessageRegistry::ReadMessageFromStdin(bool) /home/vargheseg/Projects/cquery/cquery/build/release/../../src/language_server_api.cc:159:0
30 0x00000000004236e7 std::__uniq_ptr_impl<BaseIpcMessage, std::default_delete<BaseIpcMessage> >::_M_ptr() const /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/unique_ptr.h:147:42
31 0x00000000004236e7 std::unique_ptr<BaseIpcMessage, std::default_delete<BaseIpcMessage> >::get() const /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/unique_ptr.h:337:0
32 0x00000000004236e7 std::unique_ptr<BaseIpcMessage, std::default_delete<BaseIpcMessage> >::operator bool() const /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/unique_ptr.h:351:0
33 0x00000000004236e7 LaunchStdinLoop(Config*, std::unordered_map<IpcId, Timer, std::hash<IpcId>, std::equal_to<IpcId>, std::allocator<std::pair<IpcId const, Timer> > >*)::$_0::operator()() const /home/vargheseg/Projects/cquery/cquery/build/release/../../src/command_line.cc:246:0
34 0x00000000004236e7 std::_Function_handler<void (), LaunchStdinLoop(Config*, std::unordered_map<IpcId, Timer, std::hash<IpcId>, std::equal_to<IpcId>, std::allocator<std::pair<IpcId const, Timer> > >*)::$_0>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/7.2.0/../../../../include/c++/7.2.0/bits/std_function.h:316:0
35 0x00007fea718370ff (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbc0ff)
36 0x00007fea734c07fc start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x77fc)
37 0x00007fea70f42b0f clone /build/glibc-CxtIbX/glibc-2.26/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:97:0

[jacobdufault]

From the stack it looks like JSON decoding is failing. Does cquery crash on the commit before 32d1c70? Does 32d1c70 introduce a regression?

[vargheseg]

I went back 10 commits from 32d1c70 and the problem is still there. So I am guessing its probably not a regression on cquery side. I suspect a regression with LanguageClient-neovim.

[jiegec]

This seems to be the same problem as #185.

[jacobdufault]

Can you start cquery with --log-stdin-stdout-to-stderr? This will then dump the exact initialization content that is being passed to cquery.

[vargheseg]

@jiegec I have the same question as you from #185

Not sure about whether null is valid for a optional parameter.

My json experience is limited, but shouldn't an optional parameter be optional. Again I also see the spec using 'null' explicitly in other contexts as a valid value (which makes sense to me) eg:

id: number | string | null;

[vargheseg]

With --log-stdin-stdout-to-stderr flag:

[CIN] |{"jsonrpc":"2.0","method":"initialize","params":{"capabilities":{"experimental":null,"textDocument":{"codeAction":null,"codeLens":null,"completion":{"completionItem":{"commitCharactersSupport":null,"documentationFormat":null,"snippetSupport":false},"dynamicRegistration":null},"definition":null,"documentHighlight":null,"documentLink":null,"documentSymbol":null,"formatting":null,"hover":null,"onTypeFormatting":null,"rangeFormatting":null,"references":null,"rename":null,"signatureHelp":null,"synchronization":null},"workspace":null},"initializationOptions":{"cacheDirectory":"/tmp/cquery","compilationDatabaseDirectory":"/home/vargheseg/Projects/DeviceAccess","resourceDirectory":"/home/vargheseg/bin/cquery/lib/clang+llvm-4.0.0-x86_64-linux-gnu-ubuntu-14.04/resource-dir/"},"processId":28686,"rootPath":"/home/vargheseg/Projects/DeviceAccess","rootUri":"file:///home/vargheseg/Projects/DeviceAccess","trace":"off"},"id":7}|
[COUT] |Content-Length: 161
{"jsonrpc":"2.0","method":"$cquery/progress","params":{"indexRequestCount":0,"doIdMapCount":0,"loadPreviousIndexCount":0,"onIdMappedCount":0,"onIndexedCount":0}}|
cquery: ../../third_party/rapidjson/include/rapidjson/document.h:1233: MemberIterator rapidjson::GenericValue<rapidjson::UTF8, rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator >::FindMember(const GenericValue<Encoding, SourceAllocator> &) [Encoding = rapidjson::UTF8, Allocator = rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator, SourceAllocator = rapidjson::MemoryPoolAllocatorrapidjson::CrtAllocator]: Assertion `IsObject()' failed.

[jiegec]

@vargheseg Yep, in the spec, null is explicitly specified as an option, instead of optional, because this distinction can really convey different meanings in some cases. So for the mentioned issue, we just made it a std::optional to work around the Language Server client bug. IMHO, although something like this should be fixed by Language Server clients, cquery still can be more robust/have better error messages for this.

But we should really think if sometime later we do need to consider the differences between nulls and do-no-existss, what we should do then.

[vargheseg]

But if the client is not confirming to spec, wouldn't it better to be explicit about it and complain. If this is an incorrect usage (though I am not sure if it is) i would think it better to fail gracefully than workaround the message which is not well formed.

[jiegec]

I think we should report this to your LSP client then. I am writing a wiki for this.

Edit: wiki updated: https://github.com/jacobdufault/cquery/wiki/Developer-tips#crashing-in-reflect-when-parsing-json

[vargheseg]

See autozimu/LanguageClient-neovim#265 for feedback from lsp client author.

[autozimu]

Quote from the wiki,

Problem: the LSP client is not strictly conforming to the spec e.g. passing null instead of do-no-exist for optional objects, making cquery crash.

Solution: report this to the corresponding LSP client.

Why is server crashing should be reported to client? Even if the request is not confirming to spec, shouldn't the server complain back to client instead of crash itself?

@jacobdufault

[jacobdufault]

cquery should gracefully exit instead of crashing. There is some error handling but the reflect system generally doesn't deal with it well. So I'm fine with leaving this issue open as a "cquery shouldn't crash but instead give an appropriate error and exit gracefully"

[autozimu]

It shouldn’t exit either. If the request is deemed invalid, it should return error back to client and continue to handle next request or notification.

…

On Sat, Jan 6, 2018 at 20:00 Jacob Dufault ***@***.***> wrote: cquery should gracefully exit instead of crashing. There is some error handling but the reflect system generally doesn't deal with it well. So I'm fine with leaving this issue open as a "cquery shouldn't crash but instead give an appropriate error and exit gracefully" — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#247 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABYt70cKOSSfmPagvgGOs6zsKfWjimWIks5tIEFugaJpZM4RVeSa> .

[jacobdufault]

It shouldn’t exit either. If the request is deemed invalid, it should
return error back to client and continue to handle next request or
notification.

Yea, I agree, but this is the initialize request which may be hard to recover from.

@MaskRay is modifying the reflection system atm - one way I can think of to support better error handling is to pass an additional context/meta parameter to reach reflection function, which if function fails is set to true or similar. Then the caller can check if reflection failed/succeeded.