zelcashd crashes

[PSLLSP]

I run ZELcore wallet, "Lich King, v4.12.0". I noticed some instruction to run full node so I try to replicate it.

I try to run full FLUX node, I have to synchronize block-chain first. That is painful process, zelcashd crashes in the background. I have to restart synchronization process again and again. I am at date 24 Jun 2019, in that time I meet some trouble.

I run ZELcore at Linux Mint 19.3 (with the latest updates), it is 64-bit Linux, it has 16GB of RAM and SSD with more than 70 GB of free space, 4 core CPU (AMD X4 610)... I run zelcore.AppImage (md5sum 79d2761e4adef72d56f769e0f5dfc614).

ZELcore has nice GUI and it hides low level details, until it works, it is OK. I needed some time to realize, that zelcashd is started in the background. I see it runs for 10 or 20 minutes and than it stops, it crashes or it is killed by the kernel. I cannot see any message about problem in debug.log. When zelcashd process crashes, GUI shows "Status Connecting..." but it doesn't try to start the process. I see that some clients are put to blacklist, many of them:

2021-04-05 10:13:06 UpdateTip: new best=000000269730dd3fe29b004e75e248894abeb667335948f8b4ac1a9158f30291  height=364209  log2_work=45.071749  tx=1027960  date=2019-06-22 11:50:44 progress=0.186269  cache=34.1MiB(22003tx)
2021-04-05 10:13:06 ERROR: ContextualCheckTransaction(): zelnodes tx seen before active
2021-04-05 10:13:06 ERROR: AcceptToMemoryPool: ContextualCheckTransaction failed
2021-04-05 10:13:06 Misbehaving: 62.122.203.220:16125 (40 -> 50)
2021-04-05 10:13:06 UpdateTip: new best=0000004d171ac976a72be1b67a87ae9e419b018509521eeecd77ee9d53b0653a  height=364210  log2_work=45.071751  tx=1027962  date=2019-06-22 11:51:29 progress=0.186270  cache=34.1MiB(22004tx)

I modified line "daemon=0" in ~/.zelcash/zelcash.conf and I started zelcashd from CLI, it runs in foreground and I see it crashes in few minutes - Segmentation fault (core dumped):

$ ~/ZelCore/bin/zelcash/zelcashd
...
Thank you for running a Zelcash node!
You're helping to strengthen the network and contributing to a social good :)

In order to ensure you are adequately protecting your privacy when using
Zelcash, please see <https://z.cash/support/security/>.

     Downloading blocks | 372480 / ~832010 (44%)
            Connections | 16
  Network solution rate | 406151 Sol/s

You are currently not mining.
To enable mining, add 'gen=1' to your zelcash.conf and restart.

Since starting this node 17 minutes, 18 seconds ago:
- You have validated 70077 transactions!

[Press Ctrl+C to exit] [Set 'showmetrics=0' to hide]
Segmentation fault (core dumped)

This is tail of debug.log:

$ tail ~/.zelcash/debug.log 
2021-04-05 11:19:04 UpdateTip: new best=0000003407dee94cf781b80aa8cc0e017deefd3521494e08bcb94d45571b9f3f  height=372491  log2_work=45.085761  tx=1053968  date=2019-07-04 01:58:13 progress=0.190967  cache=96.2MiB(57525tx)
2021-04-05 11:19:04 UpdateTip: new best=00000014fa31b4b1b371e4527a1154bdc748a6f4807a6ad1e32b7c9922b69948  height=372492  log2_work=45.085763  tx=1053971  date=2019-07-04 01:59:56 progress=0.190967  cache=96.2MiB(57528tx)
2021-04-05 11:19:04 UpdateTip: new best=00000012f2ae1b9ecff32aca086ec8ba8292f8e1c786981448ddeb76b229a882  height=372493  log2_work=45.085764  tx=1053972  date=2019-07-04 02:00:05 progress=0.190967  cache=96.2MiB(57529tx)
2021-04-05 11:19:04 UpdateTip: new best=0000005da3582657776582c54ed193cc13b8d672fd69be3a960f93e33a407bb9  height=372494  log2_work=45.085766  tx=1053973  date=2019-07-04 02:00:14 progress=0.190968  cache=96.2MiB(57530tx)
2021-04-05 11:19:04 UpdateTip: new best=0000004cdb8b97aeba2fad503e87b5581f849b2da8a7b0ce2e197d43290a8d94  height=372495  log2_work=45.085768  tx=1053977  date=2019-07-04 02:00:46 progress=0.190968  cache=96.2MiB(57532tx)
2021-04-05 11:19:04 UpdateTip: new best=00000034089759e74b0a45a45d8e37517034ca87da3ea3a297feb099e1a3ccbe  height=372496  log2_work=45.08577  tx=1053979  date=2019-07-04 02:01:30 progress=0.190969  cache=96.2MiB(57533tx)
2021-04-05 11:19:04 UpdateTip: new best=00000049d86067952875266c1ad5ba76f17658999d37ff1ca4c264e390e1559c  height=372497  log2_work=45.085772  tx=1053980  date=2019-07-04 02:01:44 progress=0.190969  cache=96.2MiB(57534tx)
2021-04-05 11:19:04 UpdateTip: new best=0000001a2417e9466972d6245447699f7c6bd40fcf216e5713893b5bd93f75fb  height=372498  log2_work=45.085774  tx=1053986  date=2019-07-04 02:09:37 progress=0.190970  cache=96.2MiB(57536tx)
2021-04-05 11:19:04 UpdateTip: new best=0000000b0287eb79a19fefffab012de62b05e7667c19d42a97ee2d0836588fc4  height=372499  log2_work=45.085775  tx=1053990  date=2019-07-04 02:11:26 progress=0.190971  cache=96.2MiB(57537tx)
2021-04-05 11:19:05 UpdateTip: new best=00000052e2ac144c2872ff641c646e41dac166ac577bc9b0837f501aba19de4a  height=372500  log2_work=45.085777  tx=1053993  date=2019-07-04 02:15:44 progress=0.190971  cache=96.2MiB(57538tx

It started at 2021-04-05 11:01:46, it crashed at 2021-04-05 11:19:05

I miss "cli" tool to talk with zelcashd

I run zelcashd under debuger "gdb", it gives more detail about crash:

...
    Downloading blocks | 369236 / ~832020 (44%)
            Connections | 16
     Downloading blocks | 370936 / ~832020 (44%)
            Connections | 16
     Downloading blocks | 370948 / ~832010 (44%)
            Connections | 16
  Network solution rate | 333883 Sol/s

     Downloading blocks | 372490 / ~832020 (44%)
            Connections | 16
  Network solution rate | 387309 Sol/s

You are currently not mining.
To enable mining, add 'gen=1' to your zelcash.conf and restart.

Since starting this node 16 minutes, 42 seconds ago:
- You have validated 70062 transactions!

[Press Ctrl+C to exit] [Set 'showmetrics=0' to hide]

Thread 21 "zelcash-msghand" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffbd7fa700 (LWP 10582)]
0x000055555562e0f8 in ?? ()
(gdb)

The tail of ~/.zelcash/debug.log at time of crash:

2021-04-05 11:45:31 UpdateTip: new best=0000001a2417e9466972d6245447699f7c6bd40fcf216e5713893b5bd93f75fb  height=372498  log2_work=45.085774  tx=1053986  date=2019-07-04 02:09:37 progress=0.190964  cache=96.2MiB(57536tx)
2021-04-05 11:45:31 UpdateTip: new best=0000000b0287eb79a19fefffab012de62b05e7667c19d42a97ee2d0836588fc4  height=372499  log2_work=45.085775  tx=1053990  date=2019-07-04 02:11:26 progress=0.190964  cache=96.2MiB(57537tx)
2021-04-05 11:45:31 UpdateTip: new best=00000052e2ac144c2872ff641c646e41dac166ac577bc9b0837f501aba19de4a  height=372500  log2_work=45.085777  tx=1053993  date=2019-07-04 02:15:44 progress=0.190965  cache=96.2MiB(57538tx)

Started at 2021-04-05 11:28:49, crashed at 2021-04-05 11:45:31

[TheTrunk]

Hi, zelcore does not ship with cli tool as it talks to the zelcashd server directly. It is possible to download the cli tool in the zelcash release:
https://github.com/zelcash/zelcash/releases/tag/v5.0.0
We are mostly doing releases targeting Ubuntu but shall be compatible on other distributions as well.
The idea of trying to start zelcash (or any daemon) zelcore comes with when a daemon crashes is great - will try to implement it.
Since this seems to be a zelcash issue, rest will be discussed and targeted in zelcash/zelcash repository 👍

[PSLLSP]

I observed the issue and restarting zelcashd doesn't help. I tried to run it in a loop while sleep 3; do time zelcashd; done, but I see it runs in a circuit, it cannot overcome some problem on blockchain, it downloads the same blocks again and again and again... When block 372500 is reached, zelcashd crashes.

I created issue for zelcash, RunOnFlux/fluxd#134

[TheTrunk]

Please try to use the bootstrap in the meantime until we have a fix. Note that bootstrap uses txindex=1
https://fluxnodeservice.com/