rr segfaults when running UML

[dcolascione]

rr a002887 (master as of today) aborts when running a 32-bit x86 user-mode linux binary. munmapappears to fail with EINVAL.

(gdb) where
#0  0x00007ffff66ddfcf in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff66df3fa in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x000055555592e068 in rr::notifying_abort () at /usr/local/google/home/dancol/software/rr/src/util.cc:1259
#3  0x0000555555819196 in rr::FatalOstream::~FatalOstream (this=0x7fffffffc30b, __in_chrg=<optimized out>) at /usr/local/google/home/dancol/software/rr/src/log.cc:343
#4  0x000055555577de25 in rr::AddressSpace::<lambda(const rr::AddressSpace::Mapping&, const rr::MemoryRange&)>::operator()(const rr::AddressSpace::Mapping &, const rr::MemoryRange &) const (__closure=0x7fffffffc500, mm=..., rem=...) at /usr/local/google/home/dancol/software/rr/src/AddressSpace.cc:1138
#5  0x00005555557834d1 in std::_Function_handler<void(const rr::AddressSpace::Mapping&, const rr::MemoryRange&), rr::AddressSpace::unmap_internal(rr::Task*, rr::remote_ptr<void>, ssize_t)::<lambda(const rr::AddressSpace::Mapping&, const rr::MemoryRange&)> >::_M_invoke(const std::_Any_data &, const rr::AddressSpace::Mapping &, const rr::MemoryRange &) (__functor=..., 
    __args#0=..., __args#1=...) at /usr/include/c++/6/functional:1731
#6  0x000055555578b5a5 in std::function<void (rr::AddressSpace::Mapping const&, rr::MemoryRange const&)>::operator()(rr::AddressSpace::Mapping const&, rr::MemoryRange const&) const (
    this=0x7fffffffc500, __args#0=..., __args#1=...) at /usr/include/c++/6/functional:2127
#7  0x0000555555781d07 in rr::AddressSpace::for_each_in_range(rr::remote_ptr<void>, long, std::function<void (rr::AddressSpace::Mapping const&, rr::MemoryRange const&)>, int) (
    this=0x555555d069e0, addr=..., num_bytes=4293898240, f=..., how=0) at /usr/local/google/home/dancol/software/rr/src/AddressSpace.cc:1740
#8  0x000055555577e091 in rr::AddressSpace::unmap_internal (this=0x555555d069e0, addr=..., num_bytes=4293898240) at /usr/local/google/home/dancol/software/rr/src/AddressSpace.cc:1142
#9  0x000055555577d6e5 in rr::AddressSpace::unmap (this=0x555555d069e0, t=0x555555d05fc0, addr=..., num_bytes=4293898240)
    at /usr/local/google/home/dancol/software/rr/src/AddressSpace.cc:1092
#10 0x0000555555907a27 in rr::Task::on_syscall_exit_arch<rr::X86Arch> (this=0x555555d05fc0, syscallno=91, regs=...) at /usr/local/google/home/dancol/software/rr/src/Task.cc:330
#11 0x00005555558fbd83 in rr::Task::<lambda(const rr::Registers&)>::operator()(const rr::Registers &) const (__closure=0x7fffffffcd60, regs=...)
    at /usr/local/google/home/dancol/software/rr/src/Task.cc:557
#12 0x000055555590590b in rr::with_converted_registers<void, rr::Task::on_syscall_exit(int, rr::SupportedArch, const rr::Registers&)::<lambda(const rr::Registers&)> >(const rr::Registers &, rr::SupportedArch, rr::Task::<lambda(const rr::Registers&)>) (regs=..., arch=rr::x86, f=...) at /usr/local/google/home/dancol/software/rr/src/Registers.h:467
#13 0x00005555558fbdf4 in rr::Task::on_syscall_exit (this=0x555555d05fc0, syscallno=91, arch=rr::x86, regs=...) at /usr/local/google/home/dancol/software/rr/src/Task.cc:558
#14 0x000055555589347b in rr::RecordTask::<lambda(const rr::Registers&)>::operator()(const rr::Registers &) const (__closure=0x7fffffffcf20, regs=...)
    at /usr/local/google/home/dancol/software/rr/src/RecordTask.cc:535
#15 0x0000555555899f55 in rr::with_converted_registers<void, rr::RecordTask::on_syscall_exit(int, rr::SupportedArch, const rr::Registers&)::<lambda(const rr::Registers&)> >(const rr::Registers &, rr::SupportedArch, rr::RecordTask::<lambda(const rr::Registers&)>) (regs=..., arch=rr::x86, f=...) at /usr/local/google/home/dancol/software/rr/src/Registers.h:467
#16 0x000055555589353c in rr::RecordTask::on_syscall_exit (this=0x555555d05fc0, syscallno=91, arch=rr::x86, regs=...) at /usr/local/google/home/dancol/software/rr/src/RecordTask.cc:537
#17 0x000055555585f99d in rr::rec_process_syscall (t=0x555555d05fc0) at /usr/local/google/home/dancol/software/rr/src/record_syscall.cc:5376
#18 0x0000555555848aea in rr::RecordSession::syscall_state_changed (this=0x555555cef410, t=0x555555d05fc0, step_state=0x7fffffffd38c)
    at /usr/local/google/home/dancol/software/rr/src/RecordSession.cc:1003
#19 0x000055555584d116 in rr::RecordSession::record_step (this=0x555555cef410) at /usr/local/google/home/dancol/software/rr/src/RecordSession.cc:1979
#20 0x0000555555843662 in rr::record (args=std::vector of length 6, capacity 8 = {...}, flags=...) at /usr/local/google/home/dancol/software/rr/src/RecordCommand.cc:362
#21 0x0000555555843f36 in rr::RecordCommand::run (this=0x555555cd88a0 <rr::RecordCommand::singleton>, args=std::vector of length 6, capacity 8 = {...})
    at /usr/local/google/home/dancol/software/rr/src/RecordCommand.cc:474
#22 0x0000555555944476 in main (argc=8, argv=0x7fffffffd6c8) at /usr/local/google/home/dancol/software/rr/src/main.cc:237

(gdb) frame 4
#4  0x000055555577de25 in rr::AddressSpace::<lambda(const rr::AddressSpace::Mapping&, const rr::MemoryRange&)>::operator()(const rr::AddressSpace::Mapping &, const rr::MemoryRange &) const (__closure=0x7fffffffc500, mm=..., rem=...) at /usr/local/google/home/dancol/software/rr/src/AddressSpace.cc:1138
1138            FATAL() << "Can't munmap";

(gdb) print/x m.local_addr
$1 = 0x7ffff7ff3000
(gdb) print/x rem.start()
$2 = {
  ptr = 0x70001000
}
(gdb) print/x m.map.start()
$3 = {
  ptr = 0x70001000
}
(gdb) print/x rem.size()
$4 = 0x8fffc000

[rocallahan]

It looks like the application is trying to unmap everything in its address space, or something like that.

It looks like rem.size() is incorrect. What's rem.end()?

[deliciouslytyped]

I haven't tried this yet, but it sounds very useful to be able to run usermode linux under rr, and I intend to try it soon. It sounds like a more direct way to do debugging than going through a QEMU layer (which method I am also not familiar with).

I'm not familiar with the UML internals, but a very little bit of the memory behavior is explained here, in case that helps: https://www.kernel.org/doc/html/v5.8/virt/uml/user_mode_linux.html#uml-on-2g-2g-hosts

This is all to say, I'm interested in having this work.