Freezing when >1500 torrents

[Rid]

Version: 0.9.4
Libtorrent: 0.13.4
OS: CentOS 6.5 x64

When a large number (>1500) are loaded rtorrent regularly freezes and becomes unstable when accessed through XMLRPC or via the commandline. We are experiencing this issue on multiple separate systems. Using gdb it appears to be stuck in the following call:
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x0000003e8e609508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003e8e6093d7 in __pthread_mutex_lock (mutex=0x728608) at pthread_mutex_lock.c:61
#3 0x00000000004c79e6 in acquire_global_lock (this=0x72b940, file=, argv=, flags=)

at /usr/local/include/torrent/utils/thread_base.h:183

#4 rpc::ExecFile::execute (this=0x72b940, file=, argv=, flags=) at exec_file.cc:165
#5 0x00000000004c7d66 in rpc::ExecFile::execute_object (this=0x72b940, rawArgs=, flags=3) at exec_file.cc:227
#6 0x0000000000424b74 in operator() (__functor=, __args#0=, __args#1=)

at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552

#7 __call<rpc::rt_triple<int, void*, void*>&, torrent::Object const&, 0, 1, 2> (__functor=, __args#0=, __args#1=)

at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137

#8 operator()<rpc::rt_triple<int, void*, void*>, const torrent::Object> (__functor=, __args#0=, __args#1=)

at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191

#9 std::tr1::_Function_handler<torrent::Object(rpc::rt_triple<int, void*, void*>, const torrent::Object&), std::tr1::_Bind<std::tr1::Mem_fn<torrent::Object (rpc::ExecFile::*)(const torrent::Object&, int)>(rpc::ExecFile, std::tr1::_Placeholder<2>, int)> >::_M_invoke(const std::tr1::Any_data &, rpc::rt_triple<int, void, void*>, const torrent::Object &) (

__functor=<value optimized out>, __args#0=<value optimized out>, __args#1=<value optimized out>)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1654

#10 0x00000000004bf963 in operator() (rawCommand=0x25a93b8, target=, args=)

at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024

#11 _call<std::tr1::function<torrent::Object(rpc::target_type, const torrent::Object&)>, rpc::rt_triple<int, void*, void*>, torrent::Object> (rawCommand=0x25a93b8,

target=<value optimized out>, args=<value optimized out>) at command.h:249

#12 rpc::command_base_call<rpc::rt_triple<int, void*, void*> > (rawCommand=0x25a93b8, target=, args=) at command.cc:59
#13 0x00000000004c336f in rpc::CommandMap::call_command (this=0x72b860, key=0x7fff25dacd00 "execute", arg=..., target=...) at command_map.cc:171
#14 0x00000000004cea87 in rpc::parse_command (target=..., first=0x7fa58c004a70 "", last=) at parse_commands.cc:166
#15 0x00000000004cf937 in rpc::parse_command_multiple (target=..., first=, last=0x7fa58c004a70 "") at parse_commands.cc:176
#16 0x00000000004d0279 in rpc::call_object (command=, target=...) at parse_commands.cc:243
#17 0x00000000004c5404 in rpc::CommandScheduler::call_item (this=0x253b3e0, item=0x7fa58c001710) at command_scheduler.cc:100
#18 0x000000000040f02b in operator() () at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#19 priority_queue_perform () at ../rak/priority_queue_default.h:98
#20 client_perform () at main.cc:176
#21 0x00007fa59d17418a in torrent::thread_base::event_loop (thread=0x2549850) at thread_base.cc:139
#22 0x0000000000412c0f in main (argc=1, argv=0x7fff25dad9e8) at main.cc:857

[skarekrow]

This may be of little help for you, but I read that rtorrent has issues like that if c-ares is not enabled in curl. If you can, I'd try recompiling curl with c-ares and see if the situation improves. I have a much larger count of torrents and am not experiencing any freezes.

On Sep 03, 2014, at 02:42 PM, Grant Millar notifications@github.com wrote:

Version: 0.9.4
Libtorrent: 0.13.4
OS: CentOS 6.5
When a large number (>1500) are loaded rtorrent regularly freezes and becomes unstable when accessed through XMLRPC or via the commandline. We are experiencing this issue on multiple separate systems. Using gdb it appears to be stuck in the following call:
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x0000003e8e609508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003e8e6093d7 in __pthread_mutex_lock (mutex=0x728608) at pthread_mutex_lock.c:61
#3 0x00000000004c79e6 in acquire_global_lock (this=0x72b940, file=, argv=, flags=)
at /usr/local/include/torrent/utils/thread_base.h:183
#4 rpc::ExecFile::execute (this=0x72b940, file=, argv=, flags=) at exec_file.cc:165
#5 0x00000000004c7d66 in rpc::ExecFile::execute_object (this=0x72b940, rawArgs=, flags=3) at exec_file.cc:227
#6 0x0000000000424b74 in operator() (__functor=, __args#0=, __args#1=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#7 __call&, torrent::Object const&, 0, 1, 2> (__functor=, __args#0=, __args#1=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#8 operator(), const torrent::Object> (__functor=, __args#0=, __args#1=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#9 std::tr1::_Function_handler, const torrent::Object&), std::tr1::_Bind(rpc::ExecFile*, std::tr1::_Placeholder, int)> >::_M_invoke(const std::tr1::_Any_data &, rpc::rt_triple, const torrent::Object &) (
__functor=, __args#0=, __args#1=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1654
#10 0x00000000004bf963 in operator() (rawCommand=0x25a93b8, target=, args=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#11 _call, rpc::rt_triple, torrent::Object> (rawCommand=0x25a93b8,
target=, args=) at command.h:249
#12 rpc::command_base_call > (rawCommand=0x25a93b8, target=, args=) at command.cc:59
#13 0x00000000004c336f in rpc::CommandMap::call_command (this=0x72b860, key=0x7fff25dacd00 "execute", arg=..., target=...) at command_map.cc:171
#14 0x00000000004cea87 in rpc::parse_command (target=..., first=0x7fa58c004a70 "", last=) at parse_commands.cc:166
#15 0x00000000004cf937 in rpc::parse_command_multiple (target=..., first=, last=0x7fa58c004a70 "") at parse_commands.cc:176
#16 0x00000000004d0279 in rpc::call_object (command=, target=...) at parse_commands.cc:243
#17 0x00000000004c5404 in rpc::CommandScheduler::call_item (this=0x253b3e0, item=0x7fa58c001710) at command_scheduler.cc:100
#18 0x000000000040f02b in operator() () at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#19 priority_queue_perform () at ../rak/priority_queue_default.h:98
#20 client_perform () at main.cc:176
#21 0x00007fa59d17418a in torrent::thread_base::event_loop (thread=0x2549850) at thread_base.cc:139
#22 0x0000000000412c0f in main (argc=1, argv=0x7fff25dad9e8) at main.cc:857
—
Reply to this email directly or view it on GitHub.

[Rid]

I can confirm curl was compiled with the follow options:

./configure --prefix=/usr --enable-ares --libdir=/usr/lib64

[skarekrow]

Unfortunately I won't be of much help, thankfully you have a gdb output. Hopefully things can get sorted for you. Good luck :)

On Sep 03, 2014, at 02:54 PM, Grant Millar notifications@github.com wrote:

I can confirm curl was compiled with the follow options:
./configure --prefix=/usr --enable-ares --libdir=/usr/lib64
—
Reply to this email directly or view it on GitHub.

[rakshasa]

You seem to be executing some shell command and it waits until it gets back the return value.

Use 'execute.nothrow.bg' to run those in the background.

[Rid]

I've also noticed that it's hanging on the below, I've enabled execute logging to see what is causing the above.

#0 0x0000003e8e2e9163 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f9034a41555 in torrent::PollEPoll::poll (this=0x1025c80, msec=) at poll_epoll.cc:146
#2 0x00007f9034a41dfe in torrent::PollEPoll::do_poll (this=0x1025c80, timeout_usec=, flags=) at poll_epoll.cc:209
#3 0x00007f9034a78241 in torrent::thread_base::event_loop (thread=0x1022850) at thread_base.cc:174
#4 0x0000000000412c0f in main (argc=1, argv=0x7fff6429b668) at main.cc:857

AND

#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x0000003456809508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00000034568093d7 in __pthread_mutex_lock (mutex=0x728608) at pthread_mutex_lock.c:61
#3 0x00007f3971870e13 in acquire_global_lock (this=0x132dc80, timeout_usec=, flags=) at utils/thread_base.h:183
#4 torrent::PollEPoll::do_poll (this=0x132dc80, timeout_usec=, flags=) at poll_epoll.cc:213
#5 0x00007f39718a7241 in torrent::thread_base::event_loop (thread=0x132a850) at thread_base.cc:174
#6 0x0000000000412c0f in main (argc=1, argv=0x7fff2df9be08) at main.cc:857

[rakshasa]

You need to bt all the threads with "thread apply all bt".

[Rid]

Thanks,

Thread 3 (Thread 0x7f3952a01700 (LWP 23501)):
#0 0x000000345680ecdc in __libc_send (fd=, buf=, n=, flags=)
at ../sysdeps/unix/sysv/linux/x86_64/send.c:33
#1 0x00000000004d27a5 in rpc::SCgiTask::event_write (this=0x13a5988) at scgi_task.cc:203
#2 0x00000000004d2bd5 in rpc::SCgiTask::receive_write (this=0x13a5988,
buffer=0x7f394d3499b0 "\r\n\r\n\r\n\r\n\r\n2DFB45C54078523C107E9715C921FEDB7D219FF5\r\n"..., length=3731867) at scgi_task.cc:251
#3 0x00000000004d33cf in operator() (this=, inBuffer=, length=3116, slotWrite=...)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#4 rpc::XmlRpc::process (this=, inBuffer=, length=3116, slotWrite=...) at xmlrpc.cc:540
#5 0x00000000004d1b6f in rpc::SCgi::receive_call (this=, task=, buffer=0x7f394c3aca00 "pI{L9\177", length=3116) at scgi.cc:175
#6 0x00000000004d2949 in rpc::SCgiTask::event_read (this=0x13a5988) at scgi_task.cc:191
#7 0x00007f3971870860 in torrent::PollEPoll::perform (this=0x133cb30) at poll_epoll.cc:185
#8 0x00007f3971870d8e in torrent::PollEPoll::do_poll (this=0x133cb30, timeout_usec=, flags=) at poll_epoll.cc:219
#9 0x00007f39718a7241 in torrent::thread_base::event_loop (thread=0x133bf70) at thread_base.cc:174
#10 0x00000034568079d1 in start_thread (arg=0x7f3952a01700) at pthread_create.c:301
#11 0x00000034564e8b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7f39713ef700 (LWP 18851)):
#0 0x00000034564e9153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f3971870555 in torrent::PollEPoll::poll (this=0x1334d20, msec=) at poll_epoll.cc:146
#2 0x00007f3971870d79 in torrent::PollEPoll::do_poll (this=0x1334d20, timeout_usec=, flags=) at poll_epoll.cc:209
#3 0x00007f39718a7241 in torrent::thread_base::event_loop (thread=0x132aed0) at thread_base.cc:174
#4 0x00000034568079d1 in start_thread (arg=0x7f39713ef700) at pthread_create.c:301
#5 0x00000034564e8b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f39713f17e0 (LWP 18849)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x0000003456809508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00000034568093d7 in __pthread_mutex_lock (mutex=0x728608) at pthread_mutex_lock.c:61
#3 0x00007f3971870e13 in acquire_global_lock (this=0x132dc80, timeout_usec=, flags=) at utils/thread_base.h:183
#4 torrent::PollEPoll::do_poll (this=0x132dc80, timeout_usec=, flags=) at poll_epoll.cc:213
#5 0x00007f39718a7241 in torrent::thread_base::event_loop (thread=0x132a850) at thread_base.cc:174
#6 0x0000000000412c0f in main (argc=1, argv=0x7fff2df9be08) at main.cc:857

[Rid]

The other rtorrent on another machine with the same system (OS version etc similar number of torrents) is stuck here:

Thread 3 (Thread 0x7fc851be9700 (LWP 1142)):
#0 0x0000003e8e60ecdc in __libc_send (fd=, buf=, n=, flags=)
at ../sysdeps/unix/sysv/linux/x86_64/send.c:33
#1 0x00000000004d27a5 in rpc::SCgiTask::event_write (this=0xcb69a8) at scgi_task.cc:203
#2 0x00000000004d2bd5 in rpc::SCgiTask::receive_write (this=0xcb69a8,
buffer=0x7fc850c65010 "\r\n\r\n\r\n\r\n\r\nCA702F870A45F0A1D5643AC88AE906C80C7052EE\r\n"..., length=3091277) at scgi_task.cc:251
#3 0x00000000004d33cf in operator() (this=, inBuffer=, length=3178, slotWrite=...)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#4 rpc::XmlRpc::process (this=, inBuffer=, length=3178, slotWrite=...) at xmlrpc.cc:540
#5 0x00000000004d1b6f in rpc::SCgi::receive_call (this=, task=, buffer=0x7fc84c002300 "x", length=3178) at scgi.cc:175
#6 0x00000000004d2949 in rpc::SCgiTask::event_read (this=0xcb69a8) at scgi_task.cc:191
#7 0x00007fc85b6f1860 in torrent::PollEPoll::perform (this=0xc41b30) at poll_epoll.cc:185
#8 0x00007fc85b6f1d8e in torrent::PollEPoll::do_poll (this=0xc41b30, timeout_usec=, flags=) at poll_epoll.cc:219
#9 0x00007fc85b728241 in torrent::thread_base::event_loop (thread=0xc40f70) at thread_base.cc:174
#10 0x0000003e8e6079d1 in start_thread (arg=0x7fc851be9700) at pthread_create.c:301
#11 0x0000003e8e2e8b6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7fc85b28d700 (LWP 30314)):
#0 0x0000003e8e2e9163 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007fc85b6f1555 in torrent::PollEPoll::poll (this=0xc2dd20, msec=) at poll_epoll.cc:146
#2 0x00007fc85b6f1d79 in torrent::PollEPoll::do_poll (this=0xc2dd20, timeout_usec=, flags=) at poll_epoll.cc:209
#3 0x00007fc85b728241 in torrent::thread_base::event_loop (thread=0xc17ed0) at thread_base.cc:174
#4 0x0000003e8e6079d1 in start_thread (arg=0x7fc85b28d700) at pthread_create.c:301
#5 0x0000003e8e2e8b6d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7fc85b28f7e0 (LWP 30303)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x0000003e8e609508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x0000003e8e6093d7 in __pthread_mutex_lock (mutex=0x728608) at pthread_mutex_lock.c:61
#3 0x00007fc85b6f1e13 in acquire_global_lock (this=0xc1ac80, timeout_usec=, flags=) at utils/thread_base.h:183
#4 torrent::PollEPoll::do_poll (this=0xc1ac80, timeout_usec=, flags=) at poll_epoll.cc:213
#5 0x00007fc85b728241 in torrent::thread_base::event_loop (thread=0xc17850) at thread_base.cc:174
#6 0x0000000000412c0f in main (argc=1, argv=0x7fff3fa03cc8) at main.cc:857

[rakshasa]

https://github.com/rakshasa/rtorrent/blob/branch-0.9/src/rpc/scgi.cc#L172

The issue is that rtorrent is currently taking the global lock during XMLRPC command processing as a whole, that is to say calling the commands in rtorrent's global lock and writing the data out while in the same.

Try this; 65dfc46

[Rid]

@rakshasa thanks for the fix, the freezing of rtorrent is now gone, however the webui front-end (rutorrent) still seems to time out when connecting to the SCGI, it's currently stuck here:

Thread 3 (Thread 0x7f67ae4b2700 (LWP 31854)):
#0 0x00000034564e9153 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#1 0x00007f67ae933555 in torrent::PollEPoll::poll (this=0x15c0d20, msec=) at poll_epoll.cc:146
#2 0x00007f67ae933d79 in torrent::PollEPoll::do_poll (this=0x15c0d20, timeout_usec=, flags=) at poll_epoll.cc:209
#3 0x00007f67ae96a241 in torrent::thread_base::event_loop (thread=0x15b6ed0) at thread_base.cc:174
#4 0x00000034568079d1 in start_thread (arg=0x7f67ae4b2700) at pthread_create.c:301
#5 0x00000034564e8b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 2 (Thread 0x7f6796ed8700 (LWP 32475)):
#0 __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1 0x0000003456809508 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00000034568093d7 in __pthread_mutex_lock (mutex=0x728608) at pthread_mutex_lock.c:61
#3 0x00000000004d2c3c in acquire_global_lock (this=0x1631988,
buffer=0x7f6791457db0 "\r\n\r\n\r\n\r\n\r\n1538BDBADF48F3F4DDC2BE2497EEF9F9892535BA\r\n"..., length=3930769) at /usr/local/include/torrent/utils/thread_base.h:183
#4 rpc::SCgiTask::receive_write (this=0x1631988,
buffer=0x7f6791457db0 "\r\n\r\n\r\n\r\n\r\n1538BDBADF48F3F4DDC2BE2497EEF9F9892535BA\r\n"..., length=3930769) at scgi_task.cc:255
#5 0x00000000004d33df in operator() (this=, inBuffer=, length=3178, slotWrite=...)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#6 rpc::XmlRpc::process (this=, inBuffer=, length=3178, slotWrite=...) at xmlrpc.cc:540
#7 0x00000000004d1b50 in rpc::SCgi::receive_call (this=, task=,
buffer=0x7f6790010a00 "d.multicall\r\nmain\r\nd.get_hash=<"..., length=3178) at scgi.cc:174
#8 0x00000000004d2929 in rpc::SCgiTask::event_read (this=0x1631988) at scgi_task.cc:191
#9 0x00007f67ae933860 in torrent::PollEPoll::perform (this=0x15c8b30) at poll_epoll.cc:185
#10 0x00007f67ae933d8e in torrent::PollEPoll::do_poll (this=0x15c8b30, timeout_usec=, flags=) at poll_epoll.cc:219
#11 0x00007f67ae96a241 in torrent::thread_base::event_loop (thread=0x15c7f70) at thread_base.cc:174
#12 0x00000034568079d1 in start_thread (arg=0x7f6796ed8700) at pthread_create.c:301
#13 0x00000034564e8b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 1 (Thread 0x7f67ae4b47e0 (LWP 31842)):
#0 0x00000034564db055 in __xstat (vers=,
name=0x12cea7d8 "/path/to/some/file.mp3", buf=0x7fff11f5d6f0)
at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:38
#1 0x00007f67ae967966 in stat (download=..., object=) at /usr/include/sys/stat.h:457
#2 update (download=..., object=) at ../../../rak/file_stat.h:51
#3 update (download=..., object=) at ../../../rak/file_stat.h:52
#4 torrent::resume_save_progress (download=..., object=) at resume.cc:218
#5 0x00000000004993e8 in core::DownloadStore::save (this=0x15a9220, d=0x962fde0, flags=1) at download_store.cc:145
#6 0x0000000000492671 in operator() (this=0x15a9240) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_function.h:605
#7 operator() (this=0x15a9240) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward/binders.h:118
#8 count_ifstd::_List_iterator<core::Download*, std::binder1st<std::mem_fun1_t<bool, core::DownloadStore, core::Download*> > > (this=0x15a9240)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/bits/stl_algo.h:4437
#9 core::DownloadList::session_save (this=0x15a9240) at download_list.cc:90
#10 0x0000000000424b18 in operator() (__functor=, __args#0=, __args#1=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#11 __call<rpc::rt_triple<int, void*, void*>&, std::vector<torrent::Object, std::allocatortorrent::Object > const&, 0> (__functor=,
__args#0=, __args#1=) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#12 operator()<rpc::rt_triple<int, void*, void*>, const std::vector<torrent::Object, std::allocatortorrent::Object > > (__functor=,
__args#0=, __args#1=) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#13 operator()<rpc::rt_triple<int, void*, void*>, const std::vector<torrent::Object, std::allocatortorrent::Object > > (__functor=,
__args#0=, __args#1=) at command_helpers.h:182
#14 std::tr1::_Function_handler<torrent::Object(rpc::rt_triple<int, void*, void*>, const torrent::Object::list_type&), object_convert_type<std::tr1::_Bind<std::tr1::Mem_fn<void (core::DownloadList::*)()>(core::DownloadList)>, void> >::M_invoke(const std::tr1::Any_data &, rpc::rt_triple<int, void, void>, const torrent::Object::list_type &) (
__functor=, args#0=, args#1=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1654
#15 0x00000000004c2cec in operator() (rawCommand=0x15d8848, target=..., rawArgs=)
at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#16 call<std::tr1::function<torrent::Object(rpc::rt_triple<int, void, void>, const torrent::Object::list_type&)>, rpc::rt_triple<int, void, void>, std::vector<torrent::Object, std::allocatortorrent::Object > > (rawCommand=0x15d8848, target=..., rawArgs=) at command.h:249
#17 rpc::command_base_call_list<rpc::rt_triple<int, void, void*> > (rawCommand=0x15d8848, target=..., rawArgs=) at command.cc:122
#18 0x00000000004c336f in rpc::CommandMap::call_command (this=0x72b860, key=0x15f0948 "session.save", arg=..., target=...) at command_map.cc:171
#19 0x00000000004d01ef in rpc::call_object (command=, target=) at parse_commands.cc:275
#20 0x00000000004c5404 in rpc::CommandScheduler::call_item (this=0x15a83e0, item=0x15f0a00) at command_scheduler.cc:100
#21 0x000000000040f02b in operator() () at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:2024
#22 priority_queue_perform () at ../rak/priority_queue_default.h:98
#23 client_perform () at main.cc:176
#24 0x00007f67ae96a18a in torrent::thread_base::event_loop (thread=0x15b6850) at thread_base.cc:139
#25 0x0000000000412c0f in main (argc=1, argv=0x7fff11f5e828) at main.cc:857

[rakshasa]

This is an issue with session save blocking, which iirc already has a ticket or if not make a new one.