--free-version Deathlock in ws communication #463

piezza · 2017-12-05T23:30:11Z

I have a deathlock after running K for about 2-10 minutes. I am not an experienced C++ developer to analyze this in deep but at least I can give you some gdb backtraces:


(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7ffff7fc5780 (LWP 12339) "K" 0x00007ffff76b6dbd in __GI___pthread_mutex_lock (mutex=0xc7b440 <K::kxMutex>)
    at ../nptl/pthread_mutex_lock.c:80
  4    Thread 0x7ffff72e2700 (LWP 12362) "K" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  5    Thread 0x7ffff62aa700 (LWP 12363) "K" 0x00007ffff73e0593 in select () at ../sysdeps/unix/syscall-template.S:84

The text was updated successfully, but these errors were encountered:

ctubio · 2017-12-05T23:35:57Z

many thanks for the report, hope i can reproduce or something this time :S will investigate soOn''

ctubio · 2017-12-05T23:37:23Z

can i ask what exchange/currency pairs/operating system are you using (some $ uname -a maybe? or a link to download your very same distro?)?

thanks''

piezza · 2017-12-05T23:39:17Z

I am using coinbase and trade BTC/EUR

edit: and I use Ubuntu xenial x64

ctubio · 2017-12-06T02:10:48Z

please let me know if you still have issues after the next commit, is supposed that now websockets cannot be locked continuosly (still i was not able to reproduce the issue)

Free Software Free Society To support commits by ctubio, you can buy-me-a-drink with a small git tip at: 1GitTipgxvKB3zjCLXRcSgDhC9pivkpc7u I promise to drink chocolate milk in the development of the next commit. To request new features or in case this commit breaks something for you, please create a new github issue with all possible details, but never share your API Keys! Signed-off-by: Carles Tubio <ctubio@users.noreply.github.com>

piezza · 2017-12-07T23:24:59Z

The issue is more seldom now, but still there:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 0x7fa51e4ee780 (LWP 8046) "K" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  2    Thread 0x7fa51d807700 (LWP 8070) "K" __lll_lock_wait ()
    at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
  3    Thread 0x7fa517fff700 (LWP 8071) "K" 0x00007fa51d905593 in select ()
    at ../sysdeps/unix/syscall-template.S:84

(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007fa51dbdbdbd in __GI___pthread_mutex_lock (mutex=0xc68440 <K::kxMutex>)
    at ../nptl/pthread_mutex_lock.c:80
#2  0x00000000005764d0 in __gthread_mutex_lock (__mutex=0xc68440 <K::kxMutex>)
    at /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:748
#3  std::mutex::lock (this=0xc68440 <K::kxMutex>) at /usr/include/c++/6/bits/std_mutex.h:103
#4  K::<lambda(uWS::WebSocket<false>*, char*, size_t, uWS::OpCode)>::operator()(uWS::WebSocket<false> *, char *, size_t, uWS::OpCode) (w=0x247a860, message=<optimized out>, length=<optimized out>, opCode=<optimized out>,
    __closure=<optimized out>) at build/K.msg:7003
#5  0x000000000055f688 in std::function<void (uWS::WebSocket<false>*, char*, unsigned long, uWS::OpCode)>::operator()(uWS::WebSocket<false>*, char*, unsigned long, uWS::OpCode) const (__args#3=<optimized out>,
    __args#2=<optimized out>, __args#1=<optimized out>, __args#0=<optimized out>, this=0x24b7958)
    at /usr/include/c++/6/functional:2127
#6  uWS::WebSocket<false>::handleFragment (
    data=data@entry=0x7fa51e4a2021 "{\"type\":\"job\",\"params\":{\"job_id\":\"911154918465763\",\"blob\":\"0606f694a6d105f21a26300f921400f611638055ad96a31ae4d2671d9c7a6547d95fc41ba61fc800000000b58d0696c593d8388344eb9f5c2f6803961c3c2e6c0ea3e7a01969b"..., length=<optimized out>, length@entry=234, remainingBytes=remainingBytes@entry=0, opCode=1,
    fin=<optimized out>, webSocketState=webSocketState@entry=0x247a8a0)
    at build-x86_64-linux-gnu/local/include/uWS/WebSocket.cpp:316
#7  0x00000000005601e6 in uWS::WebSocketProtocol<false, uWS::WebSocket<false> >::consumeMessage<4u, unsigned short>
    (wState=0x247a8a0, length=<synthetischer Zeiger>, src=<synthetischer Zeiger>, payLength=234)
    at build-x86_64-linux-gnu/local/include/uWS/WebSocketProtocol.h:135
#8  uWS::WebSocketProtocol<false, uWS::WebSocket<false> >::consume (wState=0x247a8a0, length=238,
    src=0x7fa51e4a201d "\201~") at build-x86_64-linux-gnu/local/include/uWS/WebSocketProtocol.h:353
#9  uWS::WebSocket<false>::onData (s=s@entry=0x247a860, data=<optimized out>, length=<optimized out>)
    at build-x86_64-linux-gnu/local/include/uWS/WebSocket.cpp:180
#10 0x000000000054de37 in uS::Socket::sslIoHandler<uWS::WebSocket<false> > (p=0x247a860, status=<optimized out>,
    events=<optimized out>) at build-x86_64-linux-gnu/local/include/uWS/Socket.h:207
#11 0x0000000000546083 in Loop::run (this=0x2430710) at build-x86_64-linux-gnu/local/include/uWS/Epoll.cpp:35
#12 0x00000000004f796d in main (argc=<optimized out>, argv=0x7ffd48d5e178) at src/server/K.cxx:76

(gdb) print __mutex.__data.__owner
$1 = 8070

(gdb) thread 2
[Switching to thread 2 (Thread 0x7fa51d807700 (LWP 8070))]
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
135     ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: Datei oder Verzeichnis nicht gefunden.
(gdb) bt
#0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007fa51dbdbe42 in __GI___pthread_mutex_lock (mutex=0x2430000) at ../nptl/pthread_mutex_lock.c:115
#2  0x000000000054d830 in __gthread_mutex_lock (__mutex=0x2430000)
    at /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:748
#3  __gthread_recursive_mutex_lock (__mutex=0x2430000)
    at /usr/include/x86_64-linux-gnu/c++/6/bits/gthr-default.h:810
#4  std::recursive_mutex::lock (this=0x2430000) at /usr/include/c++/6/mutex:105
#5  std::lock_guard<std::recursive_mutex>::lock_guard (__m=..., this=<synthetischer Zeiger>)
    at /usr/include/c++/6/bits/std_mutex.h:162
#6  uWS::Group<false>::broadcast (this=0x24b7890,
    message=0x7fa510000b90 "{\"type\":\"submit\",\"params\":{\"nonce\":\"45e70eb9\",\"result\":\"da8649dfc799879d0a8b1bc905fafc98986928b0fa35fbcc5e20629011fd6e00\",\"job_id\":\"530295938113704\"}}", length=150,
    opCode=opCode@entry=uWS::TEXT) at build-x86_64-linux-gnu/local/include/uWS/Group.cpp:230
#7  0x0000000000574206 in K::kxx (kxM=0xc68440 <K::kxMutex>) at build/K.msg:6943
#8  K::<lambda()>::operator() (__closure=<optimized out>) at build/K.msg:6965
#9  std::_Bind_simple<K::kA(K::Gw*, uWS::Hub*)::<lambda()>()>::_M_invoke<> (this=<optimized out>)
    at /usr/include/c++/6/functional:1391
#10 std::_Bind_simple<K::kA(K::Gw*, uWS::Hub*)::<lambda()>()>::operator() (this=<optimized out>)
    at /usr/include/c++/6/functional:1380
#11 std::thread::_State_impl<std::_Bind_simple<K::kA(K::Gw*, uWS::Hub*)::<lambda()>()> >::_M_run(void) (
    this=<optimized out>) at /usr/include/c++/6/thread:197
#12 0x000000000084bbff in execute_native_thread_routine ()
#13 0x00007fa51dbd96ba in start_thread (arg=0x7fa51d807700) at pthread_create.c:333
#14 0x00007fa51d90f3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

(gdb) print __mutex.__data.__owner
$2 = 8046

ctubio · 2017-12-09T14:43:00Z

and when this happens the app just stops without doing nothing? or what is the consequence?

piezza · 2017-12-09T15:30:55Z

Yes, it stops then without doing nothing. On the webinterface I only get this Disconnected html side then. Also it doesn't update the orders (what is the really bad part about that ;-) )

Today I updated to your "Coinbase libquickfix" commit. Until now I had no more coinbase deathlocks I think. Same issue occures on Bitfinex also on my system. So this seems to be a more general problem. When I tested this in a Vmware VM which was times slower as my other system, it ran without errors (I tested for half a day). I know another guy who is evaluating on a slower embedded system who also had no problem with Bitfinex in the last two days so far.

In this shoutbox in the K webinterface one other guy also complained that the bot was very unstable. It sounded somehow as the same problem I have.

By the way, thanks for making this bot public and for this very fast support :-)

ctubio · 2017-12-09T19:11:45Z

i keep this issue closed cos i cannot really reproduce this no matter how long i run coinbase or bitfinex (cos since is closed maybe others will open new with more info); anyway i will continue cleaning up all possible mutex to get rid of them on next commits kxMutex will be the last i think :( but someday

piezza · 2017-12-12T17:40:08Z

I have some more informations here: this bug seems only to be there in the free version
As far as I understand the two threads conflicting each other are from submitting the results/receiving the next XMR mining jobs
I didn't check against current version with fixes but this version with above mentioned "Coinbase libquickfix" did not solve the problem. K still crashed after some hours.

ctubio · 2017-12-12T20:55:00Z

thanks'' you are right; that is the last thread; will try to avoid it too soOn

lets reopen now that you clearly identified the issue and a solution is possible (thanks again'¡)

ctubio added the 🔥 fixme! label Dec 5, 2017

ctubio added the 😭 non reproducible bug label Dec 5, 2017

ctubio closed this as completed Dec 6, 2017

ctubio added ✔️ acknowledged 💀 bug and removed 🔥 fixme! 😭 non reproducible bug labels Dec 12, 2017

ctubio reopened this Dec 12, 2017

ctubio changed the title ~~Deathlock in ws communication~~ --free-version Deathlock in ws communication Dec 12, 2017

ctubio closed this as completed in d18e3b9 Dec 13, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

--free-version Deathlock in ws communication #463

--free-version Deathlock in ws communication #463

piezza commented Dec 5, 2017 •

edited by ctubio

ctubio commented Dec 5, 2017

ctubio commented Dec 5, 2017 •

edited

piezza commented Dec 5, 2017 •

edited

ctubio commented Dec 6, 2017 •

edited

piezza commented Dec 7, 2017

ctubio commented Dec 9, 2017

piezza commented Dec 9, 2017

ctubio commented Dec 9, 2017

piezza commented Dec 12, 2017

ctubio commented Dec 12, 2017 •

edited

--free-version Deathlock in ws communication #463

--free-version Deathlock in ws communication #463

Comments

piezza commented Dec 5, 2017 • edited by ctubio

ctubio commented Dec 5, 2017

ctubio commented Dec 5, 2017 • edited

piezza commented Dec 5, 2017 • edited

ctubio commented Dec 6, 2017 • edited

piezza commented Dec 7, 2017

ctubio commented Dec 9, 2017

piezza commented Dec 9, 2017

ctubio commented Dec 9, 2017

piezza commented Dec 12, 2017

ctubio commented Dec 12, 2017 • edited

piezza commented Dec 5, 2017 •

edited by ctubio

ctubio commented Dec 5, 2017 •

edited

piezza commented Dec 5, 2017 •

edited

ctubio commented Dec 6, 2017 •

edited

ctubio commented Dec 12, 2017 •

edited