Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daily crashes on hhvm 3.7.2 (and previous versions, at least 3.7.0) #5528

Closed
0x64616E69656C opened this issue Jun 24, 2015 · 16 comments

Comments

@0x64616E69656C
Copy link

commented Jun 24, 2015

Hi,

hhvm crashes regularly on a high load webapp (/www.XXX/) once or twice a day, quite more often when switching another app (/other.XXX/) sharing the same database and controller/model from php-cgi to hhvm too. There's no way to reproduce this, as it affects pdf files generated by mPDF, normal sites, GET and POST methods on random pages. Since it's monitored and automatically restarted, that's not so bad. Not sure if this relates to bug #2721. But sometimes, like once or twice a week, the hhvm daemon just get unresponsible for 10, 15, 20 minutes until manually restarted without consuming any additional cpu or memory. Clients receive a 502 error after timeout connecting to hhvm backend (nginx 1.9.2 behind nghttp2 1.0.4 as tls offloader).

HHVM 3.7.2~jessie
System: debian-testing/amd64
libc6 2.19-18

let me know if you need more information.

kern.log & stacktrace from the last few days:

[2015-06-18 18:54:56]  hhvm[30919]: segfault at 7f0100000060 ip 00007fafd7d0eb7f sp 00007fafadbf9eb0 error 4 in libc-2.19.so[7fafd7ca5000+19f000]
[2015-06-19 08:44:41]  hhvm[15651]: segfault at 8 ip 00007f203de77385 sp 00007f20247f9840 error 4 in libc-2.19.so[7f203de31000+19f000]
[2015-06-20 12:42:38]  hhvm[5467] general protection ip:7f72672c1b36 sp:7f7241bf9ef0 error:0 in libc-2.19.so[7f7267258000+19f000]
[2015-06-20 19:25:32]  hhvm[21629] general protection ip:144565a sp:7fe697bf89e0 error:0 in hhvm[400000+34e5000]
[2015-06-21 23:07:24]  hhvm[12651]: segfault at 8 ip 00007fcf8a75f385 sp 00007fcf69ff9840 error 4 in libc-2.19.so[7fcf8a719000+19f000]
Host: XXX
ProcessID: 11214
ThreadID: 7fcf69fff700
ThreadPID: 12651
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0

Server: www.XXX
ThreadType: Web Request
Server_SERVER_NAME: www.XXX
URL: (normal URL)

#0  fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#1  HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#2  std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#3  std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#4  HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#5  HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#6  HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#7  HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#8  HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#9  HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#16 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:
Host: XXX
ProcessID: 15611
ThreadID: 7f20247ff700
ThreadPID: 15651
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0

Server: other.XXX
ThreadType: Web Request
Server_SERVER_NAME: other.XXX
URL: (GET method)

#0  fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#1  HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#2  std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#3  std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#4  HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#5  HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#6  HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#7  HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#8  HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#9  HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#16 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:
Host: XXX
ProcessID: 21522
ThreadID: 7fe697bff700
ThreadPID: 21629
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0

Server: www.XXX
ThreadType: Web Request
Server_SERVER_NAME: www.XXX
URL: (PDF file (mPDF Library))

#0  fgets at /lib/x86_64-linux-gnu/libc.so.6:0
#1  BIO_fd_should_retry at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#2  PEM_read_bio at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#3  PEM_X509_INFO_read_bio at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#4  X509_load_cert_crl_file at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#5  X509_load_cert_crl_file at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#6  X509_STORE_load_locations at /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0:0
#7  curl_slist_free_all at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#8  curl_slist_free_all at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#9  curl_formget at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#10 curl_formget at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#11 curl_multi_remove_handle at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#12 curl_multi_perform at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#13 curl_easy_perform at /usr/lib/x86_64-linux-gnu/libcurl.so.4:0
#14 HPHP::HttpClient::request(char const*, char const*, char const*, int, HPHP::StringBuffer&, std::unordered_map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, HPHP::string_hashi, HPHP::stringlike_eqstri<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const*, std::vector<HPHP::String, std::allocator<HPHP::String> >*) at /usr/bin/hhvm:0
#15 HPHP::HttpClient::get(char const*, HPHP::StringBuffer&, std::unordered_map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, HPHP::string_hashi, HPHP::stringlike_eqstri<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const*, std::vector<HPHP::String, std::allocator<HPHP::String> >*) at /usr/bin/hhvm:0
#16 HPHP::UrlFile::open(HPHP::String const&, HPHP::String const&) at /usr/bin/hhvm:0
#17 HPHP::HttpStreamWrapper::open(HPHP::String const&, HPHP::String const&, int, HPHP::SmartPtr<HPHP::StreamContext> const&) at /usr/bin/hhvm:0
#18 HPHP::File::Open(HPHP::String const&, HPHP::String const&, int, HPHP::SmartPtr<HPHP::StreamContext> const&) at /usr/bin/hhvm:0
#19 HPHP::f_fopen(HPHP::String const&, HPHP::String const&, bool, HPHP::Variant const&) at /usr/bin/hhvm:0
#20 HPHP::jit::x64::(anonymous namespace)::BackEnd::enterTCHelper(unsigned char*, HPHP::ActRec*) at back-end-x64.cpp:0
#21 HPHP::jit::MCGenerator::enterTC(unsigned char*, HPHP::ActRec*) at /usr/bin/hhvm:0
#22 HPHP::enterVM(HPHP::ActRec*, HPHP::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at bytecode.cpp:0
#23 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at /usr/bin/hhvm:0
#24 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at /usr/bin/hhvm:0
#25 HPHP::invoke_file(HPHP::String const&, bool, char const*) at builtin-functions.cpp:0
#26 HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) at /usr/bin/hhvm:0
#27 HPHP::hphp_invoke(HPHP::ExecutionContext*, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool, bool, bool) at /usr/bin/hhvm:0
#28 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#29 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#30 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#31 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#32 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#33 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#34 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#35 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#36 clone at /lib/x86_64-linux-gnu/libc.so.6:0
Host: XXX
ProcessID: 30825
ThreadID: 7fafadbff700
ThreadPID: 30919
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0

Server: other.XXX
ThreadType: Web Request
Server_SERVER_NAME: other.XXX
URL: (POST method)

#0  _IO_file_setbuf at /lib/x86_64-linux-gnu/libc.so.6:0
#1  _IO_do_write at /lib/x86_64-linux-gnu/libc.so.6:0
#2  _IO_file_close_it at /lib/x86_64-linux-gnu/libc.so.6:0
#3  fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#4  HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#5  std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#6  std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#7  HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#8  HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#9  HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#10 HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#11 HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#12 HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#13 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#14 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#15 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#16 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#17 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#18 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#19 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:
Host: XXX
ProcessID: 3733
ThreadID: 7f7241bff700
ThreadPID: 5467
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0

Server: www.XXX
ThreadType: Web Request
Server_SERVER_NAME: www.XXX
URL: (GET method)

#0  fclose at /lib/x86_64-linux-gnu/libc.so.6:0
#1  HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
#2  std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
#3  std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
#4  HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
#5  HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
#6  HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
#7  HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
#8  HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
#9  HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
#10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
#11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
#12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
#13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
#14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
#15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
#16 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:
@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Jun 24, 2015

As it just happened for the second time today, an update on the (likely separate) "unresponsive issue". I just noticed that hhvm backend could not be reached for about 15 minutes before I manually restarted it, and it created a stacktrace.log in /tmp, but without any further information. Also no kern.log or hhvm/error.log. So basicly it segfaulted without terminating the running process. I'm not entirely sure if this is more likely related to nghttp2`s stream implementation (an upgrade from 1.0.2 to 1.0.4 to which fixes nghttp2/nghttp2#264 were the only changes made today)?

Host: XXX
ProcessID: 12963
ThreadID: 7fd4f93ff700
ThreadPID: 23829
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0
@paulbiss

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2015

These all look like they involve some sort of file I/O, some of them look potentially ini related (cc @JoelMarcey)

@paulbiss paulbiss added the crash label Jun 24, 2015

@JoelMarcey

This comment has been minimized.

Copy link
Contributor

commented Jun 24, 2015

I think the calls to

# 4  HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
# 5  HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0

are ancillary to the problem here. The ini request shutdown function is called at the end of every request and any ini setting that was set via ini_set() is reset back to the value before the request started.

So, it looks to me like someone called ini_set("log_errors", true) and/or ini_set("error_log", "filename") somewhere and there is a problem writing to the log file.

https://github.com/facebook/hhvm/blob/master/hphp/runtime/base/request-injection-data.cpp#L372

@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Jun 25, 2015

Hi,

well, that's interesting. ini_set is used:

_conf/defines.php:ini_set('session.save_handler', 'files');
_conf/defines.php:ini_set('session.save_path', '/var/lib/php5/sessions');
_conf/defines.php:ini_set('session.auto_start', 0);
_conf/defines.php:ini_set('session.cookie_path', '/');
_conf/defines.php:ini_set('session.gc_maxlifetime', '604800');
_conf/defines.php:ini_set('session.cookie_lifetime', '604800');
_conf/defines.php:ini_set('session.cookie_domain', 'XXX');
_conf/defines.php:ini_set('error_log', DIR_FS_ROOT . '_logs/error.log');
_conf/defines.php:ini_set('display_errors', 0);

error_log is watched with swatch, but, in production, where those crashes happen, error.log is always empty, since errors only occur in dev-env when making typos :-)

The thing is, the app has a frontend and a backend. Frontend running on hhvm just crashes once, maybe twice a day, somedays not at all. I tried pen-testing dev-env, but no luck reproducing this. Setup is exactly the same, the only variable missing is user-(mis)behaviour.
When switching the backend (huge amount of AJAX requests, ariadb writes) from php-cgi (no problems at all) to hhvm, hhvm crashes quite more often. But, it got better with each release of hhvm, < 3.7.0 it wasn't usuable at all with backend, since 3.7.0 I at least can try it for a few days before switching back to php-cgi when users start complaining about 502 ;)

server.ini

pid = /var/run/hhvm/pid
hhvm.enable_xhp = 1
hhvm.jit_a_size = 536870912
hhvm.jit_a_prof_size = 536870912
hhvm.jit_a_max_usage = 536870912
hhvm.server.upload.upload_max_file_size = 52428800
hhvm.server.upload.upload_tmp_dir = /tmp
hhvm.server.user = www-data
hhvm.server.port = 9001
hhvm.server.ip   = 127.0.0.1
hhvm.server.type = fastcgi
hhvm.server.default_document = index.php
hhvm.server.thread_count = 16
hhvm.keep_perf_pid_map = 0
hhvm.perf_pid_map = 0
hhvm.perf_data_map = 0
hhvm.log.use_log_file = true
hhvm.log.file = /var/log/hhvm/error.log
hhvm.log.header = true
hhvm.mysql.socket = /var/run/mysqld/mysqld.sock
hhvm.repo.central.path = /var/run/hhvm/hhvm.hhbc

php.ini

session.save_handler = files
session.save_path = /var/lib/php5/sessions
session.gc_probability = 10
session.gc_maxlifetime = 604800
upload_max_filesize = 52428800
post_max_size = 52428800
max_input_time = 360
max_execution_time = 360
default_socket_timeout = 360
expose_php = Off
cgi.fix_pathinfo = 0
apc.stat = 0
hhvm.log.level = Warning
hhvm.log.always_log_unhandled_exceptions = true
hhvm.log.runtime_error_reporting_level = 8191
hhvm.mysql.typed_results = false
hhvm.libxml.ext_entity_whitelist = file

file-systems related:

/dev/xvde1 on /var type ext4 (rw,relatime,user_xattr,barrier=1,data=ordered)
/dev/xvdc1 on /var/www type ext4 (rw,noatime,nodiratime,errors=remount-ro,user_xattr,barrier=1,data=ordered)

Any ideas for ini tweaks or reproducing crashes? ab/h2load/siege seem quite useless to simulate user behaviour.

//Edit:
I currently have

fs.file-max = 100000

and

www-data         soft   nofile          4096
www-data         hard   nofile          10240

When I think about why backend crashes more likely than frontend, this might be related: in frontend, MariaDB qps is insignificantly low, since most of the data is stored indefinitely in redis until a change in a dataset is made in backend, while backend has a high qps. For a few month now, starting with mariadb 10.0.x, mysqloptimize/mysqlcheck/mysqlanalyze cronjobs mark random tables as crashed, as it cannot create a tmp table (this relates to the latest mysqlatfacebook rant about "sheer incompetence", mysql bug #77439), because it already exists (created by well ordered queries running at the same time). This happens on all machines running hhvm with mariadb/ariadb/mysqli. Never reported that though, because: unreproducible.

@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Jul 5, 2015

Hi again,

I was able to pinpoint this to mPDF Library (mpdf1.com, v6.0, unrelated to previous mPDF issues), which explains why HHVM crashed more often on backend (heavy usage of mpdf). To be fair, this affects < 0,1% of all mPDF requests, but this makes it even harder to understand.
The URL in stacktrace don't seem mean anything, it looks more likely to be related to another client that was trying to connect to HHVM through nginx when/while HHVM crashes.

At first, I set the mPDF timeout from 480s (?!) to 30s, so I could sleep again. Then I went through all related log files. One entry in HHVM error.log showed that mPDF was not able to load a 300dpi (~150k) image from a CDN host (resolver timeout), so I wondered if there are libc/HHVM issues with getaddrinfo(), since I'm on debian testing with hhvm 3.7.2 for jessie, and added the CDN hosts to /etc/hosts (nsswitch preferes files before dns). That didn't happen again, but the issues remains (but never with php-cgi up to 5.6.9)

[Fri Jul  3 11:54:11 2015] [hphp] [12842:7fc5c2bff700:7988:000002] [2ac7f19:db9a2e:e0027e:17fa697:17fa9f5:4b756fc:47ec7c5:4dd8f57:58e90dd:44600000:4600001:159ff46:141caeb:fd440d:2a6a8ca:2aba4f9:2aba37f:c677c1:dcae35:f203ed:f22646:f3811f:f38939:f351d9:2029e77:c5e2a7:7fc5d7c2d0a4:7fc5d713f04d] \nWarning: Failed to open https://cdn.host/path/to/file.jpg (Resolving timed out after 360904 milliseconds) in /path/to/mpdf/mpdf.php on line 10431
[Fri Jul  3 12:01:38 2015] [hphp] [12842:7fc591bff700:512:000003] [dca781:dcaafa:dcb419:f203ed:f22646:f3811f:f38939:f351d9:2029e77:c5e2a7:7fc5d7c2d0a4:7fc5d713f04d] \nFatal error: entire web request took longer than 480 seconds and timed out in /path/to/mpdf/mpdf.php
[Sat Jul  4 15:42:09 2015] [hphp] [29911:7fe3bf7ff700:19:000001] [] \nFatal error: entire web request took longer than 30 seconds and timed out in /path/to/mpdf/classes/cssmgr.php on line 1129

and the stacktrace for above:

Host: xxx
ProcessID: 7989
ThreadID: 7fbec1fff700
ThreadPID: 8426
Name: unknown program
Type: Segmentation fault
Runtime: hhvm
Version: tags/HHVM-3.7.2-0-gfc9f29b2799933d8215faaadfa83de722df64e26
DebuggerCount: 0

Server: xxx
ThreadType: Web Request
Server_SERVER_NAME: xxx
URL: /unrelated/scriptname

# 0  fclose at /lib/x86_64-linux-gnu/libc.so.6:0
# 1  HPHP::Logger::SetNewOutput(_IO_FILE*) at /usr/bin/hhvm:0
# 2  std::_Function_handler<bool (std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), HPHP::RequestInjectionData::threadInit()::{lambda(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)#13}>::_M_invoke(std::_Any_data const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at request-injection-data.cpp:0
# 3  std::_Function_handler<bool (folly::dynamic const&), void HPHP::IniSetting::Bind<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >(HPHP::Extension const*, HPHP::IniSetting::Mode, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, HPHP::IniSetting::SetAndGet<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >*)::{lambda(folly::dynamic const&)#1}>::_M_invoke(std::_Any_data const&, folly::dynamic const&) at /usr/bin/hhvm:0
# 4  HPHP::IniSetting::SetUser(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, folly::dynamic const&, HPHP::IniSetting::FollyDynamic) at /usr/bin/hhvm:0
# 5  HPHP::IniSettingExtension::requestShutdown() at /usr/bin/hhvm:0
# 6  HPHP::ExtensionRegistry::requestShutdown() at /usr/bin/hhvm:0
# 7  HPHP::hphp_context_shutdown() at /usr/bin/hhvm:0
# 8  HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) at /usr/bin/hhvm:0
# 9  HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) at /usr/bin/hhvm:0
# 10 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) at /usr/bin/hhvm:0
# 11 HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) at /usr/bin/hhvm:0
# 12 HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() at /usr/bin/hhvm:0
# 13 HPHP::AsyncFuncImpl::ThreadFunc(void*) at /usr/bin/hhvm:0
# 14 HPHP::start_routine_wrapper(void*) at /usr/bin/hhvm:0
# 15 start_thread at /lib/x86_64-linux-gnu/libpthread.so.0:0
# 16 clone at /lib/x86_64-linux-gnu/libc.so.6:0

PHP Stacktrace:

When I restart HHVM as it becomes unresponsive, and before script times out, the stacktrace will be empty after DebuggerCount.

What I don't understand:

Regardless of yet unknown mPDF code issues, if one client is locking up HHVM with a mPDF request, why are all other requests to HHVM through nginx being stalled (and 503'ed after timeout)? Shouldn't HHVM be threaded, and should not only the one "spawned" thread running the script crashing? Did I miss something in server.ini?

@JoelMarcey

This comment has been minimized.

Copy link
Contributor

commented Jul 8, 2015

@64616E69656C Hi. I am on vacation right now, but a quick glance. You have the server thread count at 16, so that should be fine. I looked at an issue with similar errors as yours....

#4779

There are some timeout settings there, but I am not sure that is the problem, but maybe, I guess.

Doing a google search ... https://www.google.com/search?q=%22entire+web+request+took+longer%22

That might provide some hints.

@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Jul 9, 2015

For now I've settled for a slightly decreased performance and switched the site in question back to php-cgi due to increased agitation and panic attacks from management.

There are 3 issues for me here, which might or might not be related to each other.

  1. "Fatal error: entire web request took longer than X seconds and timed out"
    === sev:low ===
    This is an error I've encountered dozens of times in the past 15 years back to the days of php3 for various reasons, mostly due to high load or just shitty code. In this case though, it affects only mPDF+hhvm. This never happened with php5[3-6] and mPDF 5.0-6.0 in the past 4 or 5 years I've been using this library. The only thing that startles me is the fact, that fastcgi jams connections to all of hhvm's worker processes, not just the one running the script. ini/fastcgi errors on my side?

  2. "HHVM crashes"
    === sev:medium ===
    Although stacktrace shows various URLs, this is most likely related to mPDF too.
    My best guess, this is a curl/TLS issue when fetching resources (jpg/png) from another HTTPS resource (this might apply to issue 1 too).
    Since the process is being monitored, the outage would be max 60 secs, so not many f*cks given.

And then, there is

  1. "HHVM unresponsive"
    === sev:high ===
    In this case, max_input_time, max_execution_time and default_socket_timeout do not apply, hhvm just stalls until it's restarted (or crashes, when unattended?), so nginx just throws the towel after fastcgi_connect_timeout has been reached and returns 502 for each request.
    On my dev box (workers 2), I was however able to somewhat "reproduce" this behaviour when calling mPDF:
(gdb) thread apply all bt

Thread 3 (Thread 0x7f2e0ebff700 (LWP 30610)):
#0  0x00007f2e1d1a8623 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f2e1eb98b08 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#2  0x00007f2e1eb83d5a in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
#3  0x0000000002a00db6 in folly::EventBase::loopBody(int) ()
#4  0x0000000002a01e66 in folly::EventBase::loopForever() ()
#5  0x000000000227f68a in proxygen::WorkerThread::runLoop() ()
#6  0x000000000227f8ef in std::thread::_Impl<std::_Bind_simple<proxygen::WorkerThread::start()::{lambda()#1} ()> >::_M_run() ()
#7  0x00007f2e1da0edf0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f2e1dc960a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9  0x00007f2e1d1a804d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7f2e09fff700 (LWP 30611)):
#0  0x00007f2e1f76826a in malloc () from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
#1  0x00007f2e1d9e9d78 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x0000000001231ae0 in HPHP::jit::FrameState::FrameState(HPHP::jit::FrameState const&) ()
#3  0x0000000001236b90 in std::vector<HPHP::jit::FrameState, std::allocator<HPHP::jit::FrameState> >::operator=(std::vector<HPHP::jit::FrameState, std::allocator<HPHP::jit::FrameState> > const&) ()
#4  0x00000000012376de in HPHP::jit::FrameStateMgr::save(HPHP::jit::Block*) ()
#5  0x00000000013eee96 in HPHP::jit::IRBuilder::appendInstruction(HPHP::jit::IRInstruction*) ()
#6  0x00000000013f0756 in HPHP::jit::IRBuilder::optimizeInst(HPHP::jit::IRInstruction*, HPHP::jit::IRBuilder::CloneFlag, HPHP::jit::Block*) ()
#7  0x00000000014053e0 in HPHP::jit::irgen::detail::genInstruction(HPHP::jit::IRGS&, HPHP::jit::IRInstruction*) ()
#8  0x000000000135cab1 in HPHP::jit::irgen::(anonymous namespace)::profiledGuard(HPHP::jit::IRGS&, HPHP::jit::Type, HPHP::jit::irgen::(anonymous namespace)::ProfGuard, int, HPHP::jit::Block*) [clone .constprop.114] ()
#9  0x000000000135d791 in HPHP::jit::irgen::checkTypeLocal(HPHP::jit::IRGS&, unsigned int, HPHP::jit::Type, int, bool) ()
#10 0x00000000014684c7 in HPHP::jit::(anonymous namespace)::RegionFormer::go() ()
#11 0x000000000146b8bb in HPHP::jit::selectTracelet(HPHP::jit::RegionContext const&, bool, bool) ()
#12 0x0000000001385160 in HPHP::jit::selectRegion(HPHP::jit::RegionContext const&, HPHP::jit::TransKind) ()
#13 0x00000000014281e5 in HPHP::jit::MCGenerator::translateWork(HPHP::jit::TranslArgs const&) ()
#14 0x00000000014298aa in HPHP::jit::MCGenerator::translate(HPHP::jit::TranslArgs const&) ()
#15 0x0000000001429b24 in HPHP::jit::MCGenerator::retranslate(HPHP::jit::TranslArgs const&) ()
#16 0x0000000001429ec9 in HPHP::jit::MCGenerator::createTranslation(HPHP::jit::TranslArgs const&) ()
#17 0x000000000142b18b in HPHP::jit::MCGenerator::bindJmp(unsigned char*, HPHP::SrcKey, HPHP::jit::ServiceRequest, HPHP::jit::TransFlags, bool&) ()
#18 0x0000000001430ac1 in MCGenerator_handleServiceRequest ()
#19 0x00000000015bfbc1 in handleSRHelper ()
#20 0x0000000002a6b866 in enterTCHelper ()
#21 0x00000000015a03f6 in HPHP::jit::x64::(anonymous namespace)::BackEnd::enterTCHelper(unsigned char*, HPHP::ActRec*) ()
#22 0x000000000141cf9b in HPHP::jit::MCGenerator::enterTC(unsigned char*, HPHP::ActRec*) ()
#23 0x0000000000fd48bd in HPHP::enterVM(HPHP::ActRec*, HPHP::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) ()
#24 0x0000000002a6afba in HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) ()
#25 0x0000000002ababe9 in HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) ()
#26 0x0000000002abaa6f in HPHP::invoke_file(HPHP::String const&, bool, char const*) ()
#27 0x0000000000c67b31 in HPHP::include_impl_invoke(HPHP::String const&, bool, char const*) ()
#28 0x0000000000dcb1f5 in HPHP::hphp_invoke(HPHP::ExecutionContext*, std::string const&, bool, HPHP::Array const&, HPHP::VRefParamValue const&, std::string const&, std::string const&, bool&, std::string&, bool, bool, bool) ()
#29 0x0000000000f2089d in HPHP::HttpRequestHandler::executePHPRequest(HPHP::Transport*, HPHP::RequestURI&, HPHP::SourceRootInfo&, bool) ()
#30 0x0000000000f22af6 in HPHP::HttpRequestHandler::handleRequest(HPHP::Transport*) ()
#31 0x0000000000f385cf in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJobImpl(std::shared_ptr<HPHP::FastCGIJob>, bool) ()
#32 0x0000000000f38de9 in HPHP::ServerWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::FastCGITransportTraits>::doJob(std::shared_ptr<HPHP::FastCGIJob>) ()
#33 0x0000000000f35689 in HPHP::JobQueueWorker<std::shared_ptr<HPHP::FastCGIJob>, HPHP::Server*, true, false, HPHP::JobQueueDropVMStack>::start() ()
#34 0x000000000202a567 in HPHP::AsyncFuncImpl::ThreadFunc(void*) ()
#35 0x0000000000c5e617 in HPHP::start_routine_wrapper(void*) ()
#36 0x00007f2e1dc960a4 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#37 0x00007f2e1d1a804d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f2e0f3ff700 (LWP 30591)):
#0  0x00007f2e1dc9a08f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x0000000000f02920 in HPHP::HttpServer::runOrExitProcess() ()
#2  0x0000000000de0eb3 in HPHP::execute_program_impl(int, char**) ()
#3  0x0000000000de1e0f in HPHP::execute_program(int, char**) ()
#4  0x0000000000b6613c in main ()

HTH.

//EDIT: I would very much appreciate deb-src for apt, so one could easyly rebuild the package with afl and more custom debugging options.

@Nikerabbit

This comment has been minimized.

Copy link

commented Jul 16, 2015

FWIW I have been having very similar symptoms. I removed all ini_sets from my code and it seems much more stable now under artificial load.

@JoelMarcey

This comment has been minimized.

Copy link
Contributor

commented Jul 20, 2015

@64616E69656C Hi. I am back from vacation now. HHVM 3.8 is out now. I am curious if you could upgrade to 3.8 and check and see if you have the same issues?

@Nikerabbit It seems odd to me that a call to ini_set in and of itself would be causing instability. What properties were you setting in ini_set that you removed?

@Nikerabbit

This comment has been minimized.

Copy link

commented Jul 20, 2015

These:

ini_set( 'error_log', "$LOGS/error_php" );
ini_set( 'display_errors',          0);
ini_set( 'ignore_repeated_errors',  1);
ini_set( 'log_errors',              1);

ini_set( 'memory_limit',      '175M' );
ini_set( 'max_execution_time', '11s' );
@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Jul 21, 2015

@JoelMarcey I was just about to do this, but then #1647 happend. Customers do like email notifictions (even more than PDFs), so I decided not to re-enable until I got that one figured out :-)

@Nikerabbit

This comment has been minimized.

Copy link

commented Aug 31, 2015

Is 27da833 possibly related to this issue?

@JoelMarcey

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2015

We just patched HHVM 3.9 with 27da833 and looking at the stacktrace above, it indeed looks like the issue that was solved by that diff.

@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Sep 22, 2015

I've switched back to hhvm (3.9.1) on sunday, since then I had no further crashes of this kind. Therefore I'll close this for now.

@0x64616E69656C

This comment has been minimized.

Copy link
Author

commented Oct 16, 2015

after upgrading to 3.10.0 today, it's happening again, now with crashes every few minutes.
stacktrace is much more helpfull though.

[Fri Oct 16 08:23:57 2015] [hphp] [17350:7fcefdfff700:57:000001] [] Core dumped: Segmentation fault

#0  mPDF->SubstituteChars() called at [/var/www/xxx/vendor/mpdf/mpdf/mpdf.php:14118]
#1  mPDF->WriteHTML(<html stuff>) called at [/var/www/[...]/pdf.php:147]
#2  include(/var/www/[...]/pdf.php) called at [/var/www/[]/bootstrap.php:196]
#3  include(/var/www/[...]/bootstrap.php) called at [/var/www/[...]/htdocs/index.php:2]
@jwatzman

This comment has been minimized.

Copy link
Member

commented Oct 16, 2015

The stacktrace above is different than the one this task was originally tracking. Can you file a new github issue for the new stacktrace please? It gets hard to track things if we reuse tasks or otherwise conflate issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.