Python script deadlocked when threads call get_torrent_status

[categorical]

version
libtorrent: v1.2.19
python: 3.8.10
platform: ubuntu 20.04 (5.4.0-169-generic)
compiler: g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

Call get_torrent_status repeatedly in two python threads stalls the program.
I would expect the program does not block.

It appeared to me that threads are calling get_torrent_status but the callback function get called from the session thread.
The calling threads block on libtorrent::aux::torrent_wait.

Script to reproduce

#!/bin/python3
  
import libtorrent as lt
import time
from threading import Thread,Event

sh=lt.session({'listen_interfaces':'0.0.0.0:6000'})
lt.add_magnet_uri(sh,'m',{})

def callback(u):return True

statusarr=[s for s in sh.get_torrent_status(callback)]

def call():
    while 1:
        for s in sh.get_torrent_status(callback):
        #for s in sh.refresh_torrent_status(statusarr):
            pass

arr=[
Thread(target=call),
Thread(target=call),
]
for t in arr:t.start()
for t in arr:t.join()

Server spec

 OS: Ubuntu 20.04 focal
 Kernel: x86_64 Linux 5.4.0-169-generic
 Uptime: 5d 4h 25m
 Packages: 635
 Shell: bash 5.0.17
 Disk: 62G / 127G (51%)
 CPU: Intel Xeon Platinum @ 2x 2.5GHz
 GPU: Cirrus Logic GD 5446
 RAM: 996MiB / 1890MiB

Run the script gives four LWP's

main.py(250572)─┬─{main.py}(250573)
                ├─{main.py}(250574)
                ├─{main.py}(250577)
                └─{main.py}(250578)

strace output of the first

futex(0x937b30, FUTEX_WAKE_PRIVATE, 1)  = 0
futex(0x937b28, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=448386, tv_nsec=545160047}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x937b30, FUTEX_WAKE_PRIVATE, 1)  = 0
futex(0x937b28, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=448386, tv_nsec=550300181}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)

strace of others

futex(0x273bab8, FUTEX_WAIT_PRIVATE, 0, NULL

gdb backtrace of first thread

#0  futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f315b6ec110, 
    clockid=<optimized out>, expected=0, futex_word=0x937b28 <_PyRuntime+1224>)
    at ../sysdeps/nptl/futex-internal.h:320
#1  __pthread_cond_wait_common (abstime=0x7f315b6ec110, clockid=<optimized out>, 
    mutex=0x937b30 <_PyRuntime+1232>, cond=0x937b00 <_PyRuntime+1184>) at pthread_cond_wait.c:520
#2  __pthread_cond_timedwait (cond=0x937b00 <_PyRuntime+1184>, mutex=0x937b30 <_PyRuntime+1232>, 
    abstime=0x7f315b6ec110) at pthread_cond_wait.c:665
#3  0x0000000000684798 in PyCOND_TIMEDWAIT (us=<optimized out>, mut=0x937b30 <_PyRuntime+1232>, 
    cond=0x937b00 <_PyRuntime+1184>) at ../Python/condvar.h:73
#4  take_gil (ceval=0x9378a8 <_PyRuntime+584>, tstate=0x27067b0) at ../Python/ceval_gil.h:206
#5  0x00000000005501da in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at ../Python/ceval.c:1246
#6  0x00000000005d5846 in PyEval_EvalFrameEx (throwflag=0, f=
    Frame 0x7f315b7ad640, for file ./main.py, line 16, in callback (u=<torrent_status at remote 0x7f3154011fb0>)) at ../Python/ceval.c:741
#7  function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, 
    co=<optimized out>) at ../Objects/call.c:284
#8  _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, 
    kwnames=<optimized out>) at ../Objects/call.c:411
#9  0x0000000000551a88 in _PyObject_Vectorcall (callable=<function at remote 0x7f315c7f2ee0>, 
    args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>)
    at ../Include/cpython/abstract.h:127
#10 0x00000000006aed5e in _PyObject_FastCall (nargs=<optimized out>, args=<optimized out>, 
    func=<function at remote 0x7f315c7f2ee0>) at ../Include/cpython/abstract.h:147
#11 _PyObject_CallFunctionVa (is_size_t=<optimized out>, va=<optimized out>, format=<optimized out>, 
    callable=<function at remote 0x7f315c7f2ee0>) at ../Objects/call.c:936
#12 _PyObject_CallFunctionVa (callable=<function at remote 0x7f315c7f2ee0>, format=<optimized out>, 
    va=<optimized out>, is_size_t=<optimized out>) at ../Objects/call.c:901
#13 0x00000000004c032e in PyEval_CallFunction (callable=<optimized out>, format=<optimized out>)
    at ../Objects/call.c:978
#14 0x00007f315bc27359 in boost::python::call<boost::python::api::object, libtorrent::torrent_status>
    (callable=<function at remote 0x7f315c7f2ee0>, a0=...) at /usr/include/boost/python/call.hpp:62
#15 0x00007f315bc1bf20 in boost::python::api::object_operators<boost::python::api::object>::operator()<libtorrent::torrent_status> (this=0x7f315b6ec550, a0=...)
    at /usr/include/boost/python/object_call.hpp:19
#16 0x00007f315bc0adf9 in (anonymous namespace)::wrap_pred (pred=..., st=...) at src/session.cpp:524
#17 0x00007f315bc59016 in std::__invoke_impl<bool, bool (*&)(boost::python::api::object, libtorrent::torrent_status const&), boost::python::api::object&, libtorrent::torrent_status const&> (__f=
    @0x7f3159eb1300: 0x7f315bc0adbe <(anonymous namespace)::wrap_pred(boost::python::api::object, libtorrent::torrent_status const&)>) at /usr/include/c++/9/bits/invoke.h:60
#18 0x00007f315bc5181c in std::__invoke<bool (*&)(boost::python::api::object, libtorrent::torrent_status const&), boost::python::api::object&, libtorrent::torrent_status const&> (__fn=
    @0x7f3159eb1300: 0x7f315bc0adbe <(anonymous namespace)::wrap_pred(boost::python::api::object, libtorrent::torrent_status const&)>) at /usr/include/c++/9/bits/invoke.h:95
#19 0x00007f315bc4c250 in std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)>::__call<bool, libtorrent::torrent_status const&, 0ul, 1ul>(std::tuple<libtorrent::torrent_status const&>&&, std::_Index_tuple<0ul, 1ul>) (
    this=0x7f3159eb1300, __args=...) at /usr/include/c++/9/functional:400
#20 0x00007f315bc454b6 in std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)>::operator()<libtorrent::torrent_status const&, bool>(libtorrent::torrent_status const&) (this=0x7f3159eb1300) at /usr/include/c++/9/functional:484
#21 0x00007f315bc3f425 in std::__invoke_impl<bool, std::_Bind<bool (*(boost::python::api::object, std::
--Type <RET> for more, q to quit, c to continue without paging--
_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)>&, libtorrent::torrent_status const&>(std::__invoke_other, std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)>&, libtorrent::torrent_status const&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#22 0x00007f315bc385f9 in std::__invoke<std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)>&, libtorrent::torrent_status const&>(std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)>&, libtorrent::torrent_status const&) (__fn=...)
    at /usr/include/c++/9/bits/invoke.h:95
#23 0x00007f315bc317cf in std::reference_wrapper<std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)> >::operator()<libtorrent::torrent_status const&>(libtorrent::torrent_status const&) const (this=0x7f315b6ecbe8)
    at /usr/include/c++/9/bits/refwrap.h:340
#24 0x00007f315bc27550 in std::_Function_handler<bool (libtorrent::torrent_status const&), std::reference_wrapper<std::_Bind<bool (*(boost::python::api::object, std::_Placeholder<1>))(boost::python::api::object, libtorrent::torrent_status const&)> > >::_M_invoke(std::_Any_data const&, libtorrent::torrent_status const&) (__functor=..., __args#0=...) at /usr/include/c++/9/bits/std_function.h:285
#25 0x00007f315bfce691 in std::function<bool (libtorrent::torrent_status const&)>::operator()(libtorrent::torrent_status const&) const (this=0x7f315b6ecbe8, __args#0=...)
    at /usr/include/c++/9/bits/std_function.h:688
#26 0x00007f315bf9ce68 in libtorrent::aux::session_impl::get_torrent_status(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const (this=0x273ba30, ret=0x7f3159eb1310, pred=..., flags=...) at ../../src/session_impl.cpp:4620
#27 0x00007f315bf52a64 in libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}::operator()() (this=0x7f315b6ecbb0) at ../../src/session_handle.cpp:122
#28 0x00007f315bf758ca in boost::asio::asio_handler_invoke<libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}>(libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorren--Type <RET> for more, q to quit, c to continue without paging--
t::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}&, ...) (function=...) at /usr/include/boost/asio/handler_invoke_hook.hpp:69
#29 0x00007f315bf6e167 in boost_asio_handler_invoke_helpers::invoke<libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}, {lambda()#1}>(libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}&, {lambda()#1}&) (function=..., context=...)
    at /usr/include/boost/asio/detail/handler_invoke_helpers.hpp:37
#30 0x00007f315bf83cb5 in boost::asio::detail::handler_work<libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}, boost::asio::system_executor, libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent:--Type <RET> for more, q to quit, c to continue without paging--
:torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}>::complete<{lambda()#1}>({lambda()#1}&, {lambda()#1}&) (
    this=0x7f315b6ecb86, function=..., handler=...)
    at /usr/include/boost/asio/detail/handler_work.hpp:100
#31 0x00007f315bf75bec in boost::asio::detail::completion_handler<libtorrent::session_handle::sync_call<void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&>(void (libtorrent::aux::session_impl::*)(std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void>) const, std::vector<libtorrent::torrent_status, std::allocator<libtorrent::torrent_status> >*&&, std::function<bool (libtorrent::torrent_status const&)> const&, libtorrent::flags::bitfield_flag<unsigned int, libtorrent::status_flags_tag, void> const&) const::{lambda()#1}>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (owner=0x2658840, base=0x7f31440024f0)
    at /usr/include/boost/asio/detail/completion_handler.hpp:70
#32 0x00007f315be90a62 in boost::asio::detail::scheduler_operation::complete (this=0x7f31440024f0, 
    owner=0x2658840, ec=..., bytes_transferred=0)
    at /usr/include/boost/asio/detail/scheduler_operation.hpp:40
#33 0x00007f315be92a36 in boost::asio::detail::scheduler::do_run_one (this=0x2658840, lock=..., 
    this_thread=..., ec=...) at /usr/include/boost/asio/detail/impl/scheduler.ipp:447
#34 0x00007f315be927b0 in boost::asio::detail::scheduler::run (this=0x2658840, ec=...)
    at /usr/include/boost/asio/detail/impl/scheduler.ipp:200
#35 0x00007f315bf44dd0 in boost::asio::io_context::run (this=0x266f7b0)
    at /usr/include/boost/asio/impl/io_context.ipp:63
#36 0x00007f315bf42f49 in libtorrent::session::<lambda()>::operator()(void) const (
    __closure=0x2738608) at ../../src/session.cpp:363
#37 0x00007f315bf44bf5 in std::__invoke_impl<void, libtorrent::session::start(libtorrent::session_handle::session_flags_t, libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> >(std::__invoke_other, libtorrent::session::<lambda()> &&) (__f=...) at /usr/include/c++/9/bits/invoke.h:60
#38 0x00007f315bf44b96 in std::__invoke<libtorrent::session::start(libtorrent::session_handle::session_flags_t, libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> >(libtorrent::session::<lambda()> &&) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95
#39 0x00007f315bf44b34 in std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_handle::session_flags_t, libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x2738608) at /usr/include/c++/9/thread:244
#40 0x00007f315bf44af5 in std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_handle::session_flags_t, libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > >::operator()(void) (this=0x2738608) at /usr/include/c++/9/thread:251
#41 0x00007f315bf44aca in std::thread::_State_impl<std::thread::_Invoker<std::tuple<libtorrent::session::start(libtorrent::session_handle::session_flags_t, libtorrent::session_params&&, libtorrent::io_service*)::<lambda()> > > >::_M_run(void) (this=0x2738600) at /usr/include/c++/9/thread:195
#42 0x00007f315b89fdf4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#43 0x00007f315d0b4609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#44 0x00007f315d1ee353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[arvidn]

the python callback requires that libtorrent locks the GIL, which is questional to begin with. This is probably the reason for the deadlock when using multiple threads.

In your example, you just return True anyway. If you don't specify it, you will have the same behavior, but without the need to lock the GIL

[categorical]

Thanks @arvidn for the reply. Yes there are ways to circumvent this and I did, though "not specify it" is signature mismatch from python side. I used to have production servers all stall every few hours and days and without knowing why had to script to reset them, and now I can confirm this deadlock is indeed the casue.

fyi, regarding the deadlock, what I see is that say there are 3 threads of concern: a) python, b) python and c) session,
then a) calls get_torrent_status and c) calls callback but released gil mid way, if b) acquired gil now it waits on c) and c) waits on gil. I tried some changes and the deadlock goes away from my settings.

v1.2.19

--- a/bindings/python/src/session.cpp
+++ b/bindings/python/src/session.cpp
@@ -27,6 +27,8 @@
 #include <libtorrent/extensions/smart_ban.hpp>
 #include <libtorrent/extensions/ut_metadata.hpp>
 #include <libtorrent/extensions/ut_pex.hpp>
+#include <sstream>
+#include <sys/syscall.h>
 
 namespace boost
 {
@@ -521,6 +523,11 @@ namespace
 
     bool wrap_pred(object pred, torrent_status const& st)
     {
+        long int tid=syscall(SYS_gettid);
+        fprintf(stdout,"thread %-10ld gil %d\n",tid,PyGILState_Check());
+
+        PyEval_InitThreads();
+        lock_gil g; 
         return pred(st);
     }
 
@@ -531,12 +538,14 @@ namespace
         // libtorrent thread the python predicate will be freed from that
         // thread, which won't work
         auto wrapped_pred = std::bind(&wrap_pred, pred, std::placeholders::_1);
+        list ret;
+        Py_BEGIN_ALLOW_THREADS
         std::vector<torrent_status> torrents
             = s.get_torrent_status(std::ref(wrapped_pred), status_flags_t(flags));
 
-        list ret;
         for (std::vector<torrent_status>::iterator i = torrents.begin(); i != torrents.end(); ++i)
             ret.append(*i);
+        Py_END_ALLOW_THREADS
         return ret;
     }

[categorical]

ways to circumvent are use refresh_torrent_status instead or by not calling in threads at all:)