New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
strange behavior on termux #16751
Comments
I haven't used bitcoin on termux, but I do use termux, and you may want to experiment with termux-wake-lock: https://wiki.termux.com/wiki/Termux-wake-lock because by default android is very keen to put background processes to sleep, and also to turn of the wifi radio to conserve battery, and termux-wake-lock prevents this. |
Thanks, @ryanofsky, for pointing out the command line tool. I wasn't aware of it, always activating the wake-lock in the Android UI. That said, this is not the problem here. Bitcoind works fine when I run it in a proot ubuntu jail within termux. Only when compiling it for natively running in termux are there these (very deterministic) issues. |
Oh, I didn't realize you were running with proot inside termux, not instead of termux. It sounds like there are two possibly related problems here: 1) RPC commands being slow and 2) bitcoin hanging on shutdown. The hang on shutdown might be easy to debug with a stack trace. You could compile bitcoin with It also seems likely the shutdown hang is related to the RPC commands responding slowly might be harder to debug. Though from the timestamps in the posted log, it at least seems like the stop RPC is being dispatched pretty quickly. |
Interesting: The debugger immediately tells me this directly at program start: |
That happens because libcrypto probes for hardware capabilities. You can run |
Issue unreproducible for latest bitcoin build from termux-packages master branch. |
confirmed. issue doesn't arise in master-branch (build on android API 24), but does persist in the same build with API 20. Easy test: start bitcoind, terminate (CTRL-C) after startup. In the Android 5 (API 20) version, this doesn't stop the process, but logs
This might give some insight into what thread stays alive. |
the logs suggest that |
Running in gdb (and skipping the illegal instruction) is interesting, because after sending signal 2 once, bitcoind hangs, but after sending a second signal 2 (from gdb), it actually terminates (unlike without gdb). Here's the state of all threads after hanging:
|
Btw, these are the logs around the shutdown. Looks to me as if the http event thread is the culprit.
|
The only noticeable difference between the master (ndk 24) and android-5 (ndk-21) builds that I have found so far is that ldd on android-5 gives |
@stefanwouldgo There noticeable differences in used libc on Android 5 and Android 7+ devices. |
I don't think libc should be the culprit, because the working builds for Android 5 (with all dependencies included from the bitcoin repo) still dynamically link to libc. |
I've now tried to replace every dependency except openssl with the one in depends, with no apparent effect. So my current theory is that the problem is in openssl. Unfortunately, the earliest existing termux version of this is already beyond 1.0.1k from depends (and all versions are heavily patched), and I haven't been able to use only openssl from depends (but not the other dependencies). |
FWIW: OpenSSL is not a dependency of recent versions of Bitcoin Core, definitely not without the GUI, and recently not even with the GU unless So I strongly doubt it can cause this behavior. |
Problem persists on my arm/Android-5, but also doesn't appear on aarch-64/Android-6. Can anybody try and verify independently on arm/Android<7? All you need to do is install termux,
(this should install 0.18.1-1 on arm, if it installs 0.18.1-3 then the termux-devs might have pulled my request including full static depends which doesn't have the issue)
then wait if bitcoind fully terminates or hangs at |
openssl hasn't really been used when compiling without GUI for a long time. See bitcoin/bitcoin#16751 (comment)
Yep, 0.19.0.1 does have the same issue, so openssl cannot be the culprit.
That doesn't seem to be related. The same error crops up in a working build (with full dependencies), but doesn't lead to the described behavior. |
Is this still an issue with a recent version of Bitcoin Core? If yes, what are the steps to reproduce? |
Bitcoind seems to run ok, but anything that communicates with it via rpc such as lnd or bitcoin-cli is very slow/blocking for a long time. bitcoin-cli is blocking while waiting for bitcoind to answer. shutting down via bitcoin-cli stop (or even ctrl-c) doesn't terminate. bitcoind can only be finally shut down via kill -9.
bitcoind behaves normally when run on the same machine using ubuntu running in proot (with standard libraries like glibc) instead of pure termux with bionic.
here's a piece of debug log when doing bitcoin-cli stop:
`2019-08-29T08:49:20Z libevent: epoll_dispatch: epoll_wait reports 1
2019-08-29T08:49:20Z libevent: event_active: 0xb83355fc (fd 10), res 2, callback 0xb6a01141
2019-08-29T08:49:20Z libevent: event_process_active: event: 0xb83355fc, EV_READ call 0xb6a01141
2019-08-29T08:49:20Z libevent: evhttp_get_request_connection: new request from 127.0.0.1:59610 on 77
2019-08-29T08:49:20Z libevent: Attempting connection to 127.0.0.1:59610
2019-08-29T08:49:20Z libevent: event_del: 0xafcea858 (fd -1), callback 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_del: 0xafcea898 (fd -1), callback 0xb69f99b5
2019-08-29T08:49:20Z libevent: event_add: event: 0xafcea898 (fd 77), EV_WRITE call 0xb69f99b5
2019-08-29T08:49:20Z libevent: Epoll ADD(4) on fd 77 okay. Old events were 0; read change was 0 (none); write change was 1 (add); close change was 0 (none)
2019-08-29T08:49:20Z libevent: event_add: event: 0xafcea858 (fd 77), EV_READ call 0xb69f98c5
2019-08-29T08:49:20Z libevent: Epoll MOD(5) on fd 77 okay. Old events were 4; read change was 1 (add); write change was 0 (none); close change was 0 (none)
2019-08-29T08:49:20Z libevent: event_del: 0xafcea898 (fd 77), callback 0xb69f99b5
2019-08-29T08:49:20Z libevent: Epoll MOD(1) on fd 77 okay. Old events were 6; read change was 0 (none); write change was 2 (del); close change was 0 (none)
2019-08-29T08:49:20Z libevent: event_add: event: 0xafcea858 (fd 77), EV_READ EV_TIMEOUT call 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_add: event 0xafcea858, timeout in 30 seconds 0 useconds, call 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_del: 0xafcea898 (fd 77), callback 0xb69f99b5
2019-08-29T08:49:20Z libevent: event_add: event: 0xafcea858 (fd 77), EV_READ EV_TIMEOUT call 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_add: event 0xafcea858, timeout in 30 seconds 0 useconds, call 0xb69f98c5
2019-08-29T08:49:20Z libevent: timeout_next: event: 0xafcea858, in 30 seconds, 0 useconds
2019-08-29T08:49:20Z libevent: epoll_dispatch: epoll_wait reports 1
2019-08-29T08:49:20Z libevent: event_active: 0xafcea858 (fd 77), res 2, callback 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_process_active: event: 0xafcea858, EV_READ call 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_add: event: 0xafcea858 (fd 77), EV_READ EV_TIMEOUT call 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_add: event 0xafcea858, timeout in 279493 seconds 881162 useconds, call 0xb69f98c5
2019-08-29T08:49:20Z libevent: evhttp_add_header: key: Host val: 127.0.0.1
2019-08-29T08:49:20Z libevent: evhttp_add_header: key: Connection val: close
2019-08-29T08:49:20Z libevent: evhttp_add_header: key: Authorization val: Basic XX...
2019-08-29T08:49:20Z libevent: evhttp_add_header: key: Content-Length val: 37
2019-08-29T08:49:20Z libevent: evhttp_read_header: checking for post data on 77
2019-08-29T08:49:20Z libevent: evhttp_get_body_length: bytes to read: 37 (in buffer 37)
2019-08-29T08:49:20Z libevent: event_del: 0xafcea858 (fd 77), callback 0xb69f98c5
2019-08-29T08:49:20Z libevent: Epoll DEL(1) on fd 77 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2019-08-29T08:49:20Z libevent: event_del: 0xafcea858 (fd 77), callback 0xb69f98c5
2019-08-29T08:49:20Z libevent: event_del: 0xafcea858 (fd 77), callback 0xb69f98c5
2019-08-29T08:49:20Z Received a POST request for / from 127.0.0.1:59610
2019-08-29T08:49:20Z ThreadRPCServer method=stop user=cookie
2019-08-29T08:49:20Z libevent: evhttp_add_header: key: Content-Type val: application/json
2019-08-29T08:49:20Z libevent: evhttp_add_header: key: Connection val: close
2019-08-29T08:49:20Z libevent: event_active: 0xafe37410 (fd -1), res 0, callback 0xb6d05e4d
2019-08-29T08:49:20Z Interrupting HTTP server
2019-08-29T08:49:20Z Interrupting HTTP RPC server
2019-08-29T08:49:20Z Interrupting RPC
2019-08-29T08:49:20Z tor: Thread interrupt
2019-08-29T08:49:20Z libevent: event_active: 0xafe376e0 (fd -1), res 1, callback 0xb69fc735
2019-08-29T08:49:20Z Shutdown: In progress...
2019-08-29T08:49:20Z Stopping HTTP RPC server
2019-08-29T08:49:20Z Unregistering HTTP handler for / (exactmatch 1)
2019-08-29T08:49:20Z Unregistering HTTP handler for /wallet/ (exactmatch 0)
2019-08-29T08:49:20Z addcon thread exit
2019-08-29T08:49:20Z net thread exit
2019-08-29T08:49:20Z Stopping RPC
2019-08-29T08:49:20Z msghand thread exit
2019-08-29T08:49:20Z RPC stopped.
2019-08-29T08:49:20Z Stopping HTTP server
2019-08-29T08:49:20Z Waiting for HTTP worker threads to exit
2019-08-29T08:49:20Z libevent: event_del: 0xb8335494 (fd 9), callback 0xb6a01141
2019-08-29T08:49:20Z libevent: Epoll DEL(1) on fd 9 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2019-08-29T08:49:20Z libevent: event_del: 0xb83355fc (fd 10), callback 0xb6a01141
2019-08-29T08:49:20Z libevent: Epoll DEL(1) on fd 10 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2019-08-29T08:49:20Z Waiting for HTTP event thread to exit
2019-08-29T08:49:21Z ERROR: Error while reading proxy response
2019-08-29T08:49:21Z opencon thread exit
`
and this is where both bitcoind and bitcoin-cli hang.
This can be reproduced every time: start bitcoind. do anything with bitcoin-cli.
this is a self-compiled version of v0.18.1 for termux, the same happens on v0.18.0.
The machine is a Sony Xperia Z android 5 smartphone running termux on flash media.
I suspect it might be because of the bionic libs or libevent(_pthread), which is also modified because the native termux package doesn't contain the pthread-library. I can provide details on the compilation process if that's helpful. Maybe @esotericnonsense has any ideas, since he seems to have run bitcoind on termux before, though seemingly in a chroot.
The text was updated successfully, but these errors were encountered: