Remote connection closed while reverse continuing

[ejpbruel]

I ran into an issue with rr while trying to analyse the following bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1260239

Steps to reproduce:

1. Create a recording with rr, using the steps to reproduce described in the bug above.
2. Replay the recording until Firefox crashes.
3. Set a watchpoint on obj (this step may be unneccessary)
4. Reverse continue.

Note that I ran the above steps on Linux Mint in VMWare Fusion on OS X.

Expected results:
The watchpoint eventually triggers.

Actual results:
rr eventually displays the following message:

rr: /home/ejpbruel/Projects/rr/src/ReplayTimeline.cc:213: void rr::ReplayTimeline::mark_after_singlestep(const rr::ReplayTimeline::Mark&, const rr::ReplayResult&): Assertion `i + 1 < mark_vector.size() && mark_vector[i + 1] == m.ptr' failed.
Remote connection closed

After this, I can no longer debug the program.

[rocallahan]

I'll try to reproduce this. In the meantime, can you run with RR_LOG=GdbConnection and dump the resulting session here? Thanks!

[rocallahan]

I reproduced the Firefox crash in rr. It crashed here:

#0  0x00007f6bd60e01cc in js::jit::HandleException(js::jit::ResumeFromException*) (this=0x7f6ba76fc380)
    at /home/roc/mozilla-inbound/js/src/jit/JitFrames.cpp:2022

There's no obj in scope, so I didn't set a watchpoint. Then I reverse-continued and it reached the start of execution without any problems. :-(

[rocallahan]

That was in an opt build. In a debug build I guess I see what I'm supposed to see:

#0  0x00007f4240c26e38 in (anonymous namespace)::Wrap(JSContext*, JS::HandleObject, JS::HandleObject) (cx=0x7f420d6e3800, existing=0x0, obj=(JSObject * const) 0x7f4209dde0d0 [object Exception])
    at /home/roc/mozilla-inbound/dom/workers/RuntimeService.cpp:847
#1  0x00007f4243345d2f in JSCompartment::wrap(JSContext*, JS::MutableHandle<JSObject*>, JS::Handle<JSObject*>) (this=0x7f420d3c0000, cx=0x7f420d6e3800, obj=(JSObject *) 0x7f4209dde0d0 [object Exception], existingArg=0x0)
    at /home/roc/mozilla-inbound/js/src/jscompartment.cpp:466
#2  0x00007f4243355fbe in JSCompartment::wrap(JSContext*, JS::MutableHandle<JS::Value>, JS::Handle<JSObject*>) (this=0x7f420d3c0000, cx=0x7f420d6e3800, vp=$jsval((JSObject *) 0x7f4209dde0d0 [object Exception]), existing=0x0)
    at /home/roc/mozilla-inbound/js/src/jscompartmentinlines.h:117
#3  0x00007f42433435f2 in JSContext::getPendingException(JS::MutableHandle<JS::Value>) (this=0x7f420d6e3800, rval=$jsval((JSObject *) 0x7f4209dde0d0 [object Exception])) at /home/roc/mozilla-inbound/js/src/jscntxt.cpp:992
#4  0x00007f424333e171 in JS_GetPendingException(JSContext*, JS::MutableHandle<JS::Value>) (cx=0x7f420d6e3800, vp=$jsval((JSObject *) 0x7f4209dde0d0 [object Exception])) at /home/roc/mozilla-inbound/js/src/jsapi.cpp:5839
#5  0x00007f423f15e6b7 in mozilla::dom::AutoJSAPI::PeekException(JS::MutableHandle<JS::Value>) (this=0x7f42133fe118, aVal=$jsval((JSObject *) 0x7f4209dde0d0 [object Exception])) at /home/roc/mozilla-inbound/dom/base/ScriptSettings.cpp:626
#6  0x00007f423f15e6ef in mozilla::dom::AutoJSAPI::StealException(JS::MutableHandle<JS::Value>) (this=0x7f42133fe118, aVal=$jsval((JSObject *) 0x7f4209dde0d0 [object Exception])) at /home/roc/mozilla-inbound/dom/base/ScriptSettings.cpp:635
#7  0x00007f423f15e09a in mozilla::dom::AutoJSAPI::ReportException() (this=0x7f42133fe118)
    at /home/roc/mozilla-inbound/dom/base/ScriptSettings.cpp:577
#8  0x00007f4240cb74eb in mozilla::dom::workers::WorkerRunnable::Run() (this=0x7f4004315e80)
    at /home/roc/mozilla-inbound/dom/workers/WorkerRunnable.cpp:376
#9  0x00007f4240cada6d in mozilla::dom::workers::WorkerPrivate::DoRunLoop(JSContext*) (this=
    0x7f4214fb5800, aCx=0x7f420d6e3800) at /home/roc/mozilla-inbound/dom/workers/WorkerPrivate.cpp:4528
#10 0x00007f4240c2eb6a in (anonymous namespace)::WorkerThreadPrimaryRunnable::Run() (this=0x7f4214f542e0)
    at /home/roc/mozilla-inbound/dom/workers/RuntimeService.cpp:2692
#11 0x00007f423dad8f61 in nsThread::ProcessNextEvent(bool, bool*) (this=0x7f4215102780, aMayWait=true, aResult=0x7f42133fe92f)
    at /home/roc/mozilla-inbound/xpcom/threads/nsThread.cpp:994
#12 0x00007f423db3f5c2 in NS_ProcessNextEvent(nsIThread*, bool) (aThread=0x7f4215102780, aMayWait=true)
    at /home/roc/mozilla-inbound/xpcom/glue/nsThreadUtils.cpp:297
#13 0x00007f423e08a567 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (this=0x7f4213410800, aDelegate=0x7f421d7e1430) at /home/roc/mozilla-inbound/ipc/glue/MessagePump.cpp:369
#14 0x00007f423dfebecb in MessageLoop::RunInternal() (this=0x7f421d7e1430)
    at /home/roc/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:230
#15 0x00007f423dfebe5e in MessageLoop::RunHandler() (this=0x7f421d7e1430)
    at /home/roc/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:223
#16 0x00007f423dfebe37 in MessageLoop::Run() (this=0x7f421d7e1430)
    at /home/roc/mozilla-inbound/ipc/chromium/src/base/message_loop.cc:203
#17 0x00007f423dad742f in nsThread::ThreadFunc(void*) (aArg=0x7f4215102780)
    at /home/roc/mozilla-inbound/xpcom/threads/nsThread.cpp:396
#18 0x00007f4247ccfb0d in _pt_root (arg=0x7f4216ff2ee0) at /home/roc/mozilla-inbound/nsprpub/pr/src/pthreads/ptthread.c:216
#19 0x00007f42476af5ca in start_thread (arg=0x7f42133ff700) at pthread_create.c:333
#20 0x00007f423bdfef6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[rocallahan]

I'm not sure exactly what watchpoint I'm supposed to set?

[rocallahan]

(rr) p obj
$1 = (JSObject * const) 0x7f4209dde0d0 [object Exception]

But I can't do much with that.

(rr) watch -l *(void**) 0x7f4209dde0d0
Hardware watchpoint 1: -location *(void**) 0x7f4209dde0d0
(rr) rc
Continuing.

Thread 2 received signal SIGSEGV, Segmentation fault.
0x00007f4240c26e38 in (anonymous namespace)::Wrap (cx=0x7f420d6e3800, existing=0x0, 
    obj=(JSObject * const) 0x7f4209dde0d0 [object Exception]) at /home/roc/mozilla-inbound/dom/workers/RuntimeService.cpp:847
847     MOZ_CRASH("There should be no edges from the debuggee to the debugger.");
(rr) rc
Continuing.

Thread 2 hit Hardware watchpoint 1: -location *(void**) 0x7f4209dde0d0

Old value = (void *) 0x7f4209dddf10
New value = (void *) 0xfffc4d4d4d4d4d4d
0x00007f4242f1ac9c in js::HeapPtr<js::ObjectGroup*>::init (this=0x7f4209dde0d0, v=0x7f4209dddf10)
    at /home/roc/mozilla-inbound/js/src/gc/Barrier.h:443
443         this->value = v;

Should I have seen the bug by now?

Are you on rr master?

[ejpbruel]

Frustratingly, I can no longer get the crash to reproduce, but I still had an old rr recording around. Here is the result of running that with RR_LOG=GdbConnection set:
https://gist.github.com/ejpbruel/647cf06ad797a751e819a93b86415ab6

When Firefox crashes, I set the watchpoint like this:
watch obj.get()

Then reverse continue. As you can see in the log, the connection is eventually closed.

It's been a while since I last recompiled rr. The version I am using is version 4.3.0.

Let me know if there's anything else I can do to help analyse this.

[rocallahan]

We fixed a leak since 4.3.0 that caused rr to crash with OOM during some reverse-continues. If you watch rr's memory usage and see it growing a lot, that's probably what you're hitting.

Whether that's it or not, 4.3.0 is quite old now so it's probably not worth spending any more time investigating unless you can reproduce on master.

Thanks!!!