Crash when cleaning up persistent sockets

[jmarrama]

I get the following stacktrace:

#0 ?? at hhvm/hphp/util/process.h:0
#1 __restore_rt at sigaction.c:0
#2 HPHP::sockopen_impl(HPHP::HostURL const&, HPHP::VRefParamValue const&, HPHP::VRefParamValue const&, double, bool) at hhvm/hphp/runtime/ext/ext_socket.cpp:1090
#3 HPHP::f_pfsockopen(HPHP::String const&, int, HPHP::VRefParamValue const&, HPHP::VRefParamValue const&, double) at /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.1/../../../../include/c++/4.8.1/bits/basic_string.h:0
#4 HPHP::JIT::X64::BackEnd::enterTCHelper(unsigned char*, HPHP::JIT::TReqInfo&) at hhvm/hphp/runtime/vm/jit/back-end-x64.cpp:113
#5 HPHP::JIT::MCGenerator::enterTC(unsigned char*, void*) at hhvm/hphp/runtime/base/rds-header.h:0
#6 HPHP::ExecutionContext::enterVMAtCurPC() at hhvm/hphp/runtime/vm/jit/mc-generator.h:225
#7 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at hhvm/hphp/runtime/vm/bytecode.cpp:1947

#8 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at hhvm/hphp/runtime/vm/vm-regs.h:0
#9 HPHP::ObjectData::o_invoke(HPHP::String const&, HPHP::Variant const&, bool) at hhvm/hphp/runtime/base/type-variant.h:0
#10 HPHP::f_hphp_invoke_method(HPHP::Variant const&, HPHP::String const&, HPHP::String const&, HPHP::Variant const&) at hhvm/hphp/runtime/ext/reflection/ext_reflection.cpp:359
#11 HPHP::JIT::X64::BackEnd::enterTCHelper(unsigned char*, HPHP::JIT::TReqInfo&) at hhvm/hphp/runtime/vm/jit/back-end-x64.cpp:113
#12 HPHP::JIT::MCGenerator::enterTC(unsigned char*, void*) at hhvm/hphp/runtime/base/rds-header.h:0
#13 HPHP::ExecutionContext::enterVMAtCurPC() at hhvm/hphp/runtime/vm/jit/mc-generator.h:225
#14 HPHP::ExecutionContext::enterVM(HPHP::ActRec*, HPHP::ExecutionContext::StackArgsState, HPHP::Resumable*, HPHP::ObjectData*) at hhvm/hphp/runtime/vm/bytecode.cpp:1947
#15 HPHP::ExecutionContext::invokeFunc(HPHP::TypedValue*, HPHP::Func const*, HPHP::Variant const&, HPHP::ObjectData*, HPHP::Class*, HPHP::VarEnv*, HPHP::StringData*, HPHP::ExecutionContext::InvokeFlags) at hhvm/hphp/runtime/vm/vm-regs.h:0
#16 HPHP::ExecutionContext::invokeUnit(HPHP::TypedValue*, HPHP::Unit const*) at hhvm/hphp/runtime/vm/bytecode.cpp:2244
.....

The offending line at ext_socket.cpp:1090 was added in one of my recent pull requests to fix a failing test (see the last commit).

I'm worried that there might be a bad race condition between Sweepable::SweepAll() and calling sock->Close(), or something else weird is happening. FWIW, I observed this problem after clearing HHVM's on-disk HHBC cache.

[jmarrama]

Finally got a core dump - this issue is very occasionally reoccurring. It contained some relevant information:

(gdb) bt
#0  0x00000000017eca1d in HPHP::sockopen_impl (hosturl=..., errnum=..., errstr=..., timeout=timeout@entry=0.10000000000000001, persistent=persistent@entry=true) at hphp/runtime/ext/ext_socket.cpp:1090
#1  0x00000000017ed95b in HPHP::f_pfsockopen (hostname=..., port=7911, errnum=..., errstr=..., timeout=<optimized out>) at hphp/runtime/ext/ext_socket.cpp:1201
#2  0x00000000058f4e3a in ?? ()
#3  0x0000000000001ee7 in ?? ()
#4  0x00007fe6dd8fb7b0 in ?? ()
#5  0x3fb999999999999a in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb) p sock->m_error
$33 = 32
(gdb) p sock->m_closed
$34 = 231
(gdb) p (bool) sock->m_closed
$35 = true
(gdb) p sock->m_fd
$36 = 917677344
(gdb) p sock->m_persistentCount
$37 = 1
(gdb) p sock->m_sweepNode
$38 = {next = 0x0, prev = 0x7fe735ef18e0}
(gdb) p sock->m_type
$39 = 0
(gdb) p sock->m_address
$40 = ""
(gdb) p sock->m_port
$41 = 62927168
(gdb) p key
$42 = "some.internal.server.box.net:7911"
(gdb) p sock
$43 = (HPHP::Socket *) 0x7fe71a779580