Segfault

[Deathorn]

Hello, I have a problem running eletrumx, and do not know what to do next. So I build v.1.0.8 it with the help of AUR package in Arch Linux. And when I start electrumx server, I got this from logs:

systemd[1]: Started ElectrumX Server.
electrumx-server[29570]: INFO:root:ElectrumX server starting
electrumx-server[29570]: INFO:Daemon:daemon #1 at 127.0.0.1:8332/ (current)
electrumx-server[29570]: INFO:BlockProcessor:switching current directory to /srv/electrumx
electrumx-server[29570]: INFO:BlockProcessor:using leveldb for DB backend
electrumx-server[29570]: INFO:BlockProcessor:created new database
electrumx-server[29570]: INFO:BlockProcessor:creating metadata directory
electrumx-server[29570]: INFO:BlockProcessor:software version: ElectrumX 1.0.5
electrumx-server[29570]: INFO:BlockProcessor:DB version: 5
electrumx-server[29570]: INFO:BlockProcessor:coin: Bitcoin
electrumx-server[29570]: INFO:BlockProcessor:network: mainnet
electrumx-server[29570]: INFO:BlockProcessor:height: -1
electrumx-server[29570]: INFO:BlockProcessor:tip: 0000000000000000000000000000000000000000000000000000000000000000
electrumx-server[29570]: INFO:BlockProcessor:tx count: 0
electrumx-server[29570]: INFO:BlockProcessor:sync time so far: 00s
electrumx-server[29570]: INFO:BlockProcessor:reorg limit is 200 blocks
electrumx-server[29570]: INFO:BlockProcessor:flushing DB cache at 1,200 MB
electrumx-server[29570]: INFO:PeerManager:accepted 15/15 new peers of 15 from coins.py
electrumx-server[29570]: INFO:Controller:RPC server listening on localhost:8000
electrumx-server[29570]: INFO:Prefetcher:catching up to daemon height 460,400 (460,401 blocks behind)
electrumx-server[29570]: INFO:Prefetcher:verified genesis block with hash 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
electrumx-server[29570]: INFO:BlockProcessor:our height: 9 daemon: 460,400 UTXOs 0MB hist 0MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 77,509 daemon: 460,400 UTXOs 12MB hist 26MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 111,009 daemon: 460,400 UTXOs 21MB hist 64MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 122,509 daemon: 460,400 UTXOs 33MB hist 110MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 129,509 daemon: 460,403 UTXOs 51MB hist 164MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 132,734 daemon: 460,404 UTXOs 75MB hist 212MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 135,788 daemon: 460,404 UTXOs 108MB hist 273MB
electrumx-server[29570]: INFO:BlockProcessor:flushed to FS in 0.4s
electrumx-server[29570]: INFO:BlockProcessor:flushed history in 21.1s for 1,234,096 addrs
electrumx-server[29570]: INFO:BlockProcessor:flush #1 took 22.0s.  Height 135,788 txs: 1,008,210
electrumx-server[29570]: INFO:BlockProcessor:tx/sec since genesis: 4,776, since last flush: 4,777
electrumx-server[29570]: INFO:BlockProcessor:sync time: 03m 31s  ETA: 20h 57m 44s
electrumx-server[29570]: INFO:BlockProcessor:our height: 137,511 daemon: 460,425 UTXOs 129MB hist 42MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 140,654 daemon: 460,443 UTXOs 159MB hist 103MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 143,318 daemon: 460,447 UTXOs 184MB hist 154MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 146,616 daemon: 460,450 UTXOs 213MB hist 200MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 147,570 daemon: 460,450 UTXOs 222MB hist 215MB
electrumx-server[29570]: INFO:BlockProcessor:our height: 151,047 daemon: 460,474 UTXOs 235MB hist 258MB
electrumx-server[29570]: INFO:BlockProcessor:flushed to FS in 0.2s
kernel: electrumx-serve[30486]: segfault at 2d4c ip 00007fea12e8dd8c sp 00007fe9cd1b6748 error 4 in libleveldb.so.1.20[7fea12e76000+5d000]
systemd[1]: Started Process Core Dump (PID 30487/UID 0).
systemd[1]: electrumx.service: Main process exited, code=killed, status=11/SEGV
electrumx-rpc[30512]: cannot connect - is ElectrumX catching up, not running, or is 8000 the wrong RPC port?
systemd[1]: electrumx.service: Unit entered failed state.
systemd[1]: electrumx.service: Failed with result 'signal'.

And then a long text of "Stack trace of thread". If I start electrumx again, I got this immediately

systemd[1]: Started ElectrumX Server.
electrumx-server[31708]: INFO:root:ElectrumX server starting
electrumx-server[31708]: INFO:Daemon:daemon #1 at 127.0.0.1:8332/ (current)
electrumx-server[31708]: INFO:BlockProcessor:switching current directory to /srv/electrumx
electrumx-server[31708]: INFO:BlockProcessor:using leveldb for DB backend
kernel: electrumx-serve[31708]: segfault at 2c58 ip 00007fc826400d8c sp 00007ffc4c663ac8 error 4 in libleveldb.so.1.20[7fc8263e9000+5d000]

What is wrong?

[bauerj]

It would be nice to see the actual stack trace.

Does it crash in _ZNK7leveldb20InternalFilterPolicy12CreateFilterEPKNS_5SliceEiPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE ?

[Deathorn]

Yes, it seems so
_ZNK7leveldb20InternalFilterPolicy12CreateFilterEPKNS_5SliceEiPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE (libleveldb.so.1)

[bauerj]

I assume you are using leveldb 1.20? If so, can you try to downgrade to 1.19?

[Deathorn]

Downgraded, crash, the same string appears in logs.

[bauerj]

It's probably some incompatibility with leveldb/snappy/plyvel.

Can you report this upstream?

[Deathorn]

Hm, I'm not sure what to report and where. Could you tell me versions that you use and are working?

[bauerj]

I'm not sure either. Python code can not cause a segmentation fault. Since electrumx is written in Python, the issue caused somewhere else.

You should report it to either wbolster/plyvel or google/leveldb.

Since I'm using rocksdb exclusively (it results in smaller databases and a bit better performance) I can't provide information about working versions. However, as far as I know, the versions in Ubuntu 16.04 do work.

[Deathorn]

Ok, I switched to rocksdb and it runs fine for now, electrumx passed flush 9, height 217,164 with no crashes so far.

[Deathorn]

I had no luck with RocksDB (#168), but I downgraded to Leveldb v 1.18 and it worked. Not the best solution, but at least I have electrum node running.

[kyuupichan]

FWIW on my DragonFlyBSD system I use LevelDB 1.19 without issues. I suspect there is some unresolved library conflict on your system with 1.19; likely something somewhere needs to be recompiled; it usually indicates a bug in one of the package descriptions.

[kyuupichan]

If you have an option to recompile (yourself, from source) or reinstall everything from packages, that might help. A bit drastic, for sure.

[anduck]

Are you running Arch Linux in an unprivileged container?

[epichub]

container? do you mean a docker or other cgroup container? I am getting this error with electrumx running as a electrumx user in arch (running on the "metal")

[anduck]

I mean a LXC container. I had this same problem when I ran it inside that. Apparently privileges are not the source of this problem, then.

[epichub]

downgrading to leveldb 1.18 also worked for me.

[smmalis37]

I also experienced the segfault on 1.20 on Arch, and downgrading to 1.18 fixed it for me as well.

[daywalker90]

I also experienced the segfault on 1.20 on debian buster, downgrading to 1.18 was difficult and resulted in electrumx not starting with ImportError: /opt/python/lib/python3.6/site-packages/plyvel/_plyvel.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN7leveldb2DB4OpenERKNS_7OptionsERKNSt7__cxx1112basic_stringIcSt11char_trai [... couldn't copy everything]
i tried rocksdb then but python-rocksdb gave me compile errors...

[kyuupichan]

I'm inclined to close these 2 issues @Deathorn as I'm near-certain they are rooted in system library issues unique to your O/S and likely to only one or two versions. Are they still present on more recent O/S upgrades?

[smmalis37]

Since you posted that message I tried updating from 1.18 to 1.20 and running for a bit, and haven't experienced a segfault. However, when I originally had the crash it occurred relatively early on while it was catching up to my already up-to-date bitcoin node. It's possible that the bug only occurs on some earlier blocks, and now that I'm past them the later version is fine. I'm not sure. If I do get a segfault again I'll update here, but after 3 days of being up I haven't yet.

[kyuupichan]

I'm going to close this - it's not something there is any reason to believe is a bug in ElectrumX; it's python after all, and many are using it fine.

[smmalis37]

Yeah, I've been fine on 1.20 for the last 3 weeks now.