Segmentation fault

[shenyaqi9527]

What caused the mistake? After running for a while, the error is reported and the process is killed. Then I ran it again and was able to synchronize, but then the error occurred again.Is this the memory limit? This error occurs when the memory reaches 1GB.

[erikd]

Please cut and paste the actual log output rather than including a screenshot. Screenshots are unreadable for people with high DPI monitors.

[shenyaqi9527]

[cardano-sl.*production*:Info:ThreadId 382] [2020-01-17 07:31:07.48 UTC] Trying to apply blocks w/o rollback. First 3: [MainBlockHeader:
    hash: 19b1f1eec6f9abb145114bfeda8cad76f2ff9fda4ff3e4ccdaf369f4052f9c3b
    previous block: 1b3c31eb41b0d38af18d9cf908a7c1848911d081271fb29b88d5b010932b2eba
    slot: 8364th slot of 152nd epoch
    difficulty: 3290025
    leader: pub:993a8f05
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:993a8f05, dPk = pub:89c29f8c } }
    block: v0.2.0
    software: cardano-sl:1
, MainBlockHeader:
    hash: a7b1b58758880db395796ce8b8cba290de717097a3a1de5b50e0a9923a2941f0
    previous block: 19b1f1eec6f9abb145114bfeda8cad76f2ff9fda4ff3e4ccdaf369f4052f9c3b
    slot: 8365th slot of 152nd epoch
    difficulty: 3290026
    leader: pub:0bdb1f5e
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:0bdb1f5e, dPk = pub:5fddeeda } }
    block: v0.2.0
    software: cardano-sl:1
, MainBlockHeader:
    hash: f25b190e0f961f05b111952b72a8cba6b30cffa4caac4c60eda64697471dc606
    previous block: a7b1b58758880db395796ce8b8cba290de717097a3a1de5b50e0a9923a2941f0
    slot: 8366th slot of 152nd epoch
    difficulty: 3290027
    leader: pub:1bc97a2f
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:1bc97a2f, dPk = pub:61261a95 } }
    block: v0.2.0
    software: cardano-sl:1
]
Last 3: [MainBlockHeader:
    hash: 197d5cfea25e990f6893e1250ea248ac25c0465db43abef95677b32c0d3ebbff
    previous block: 14b798e52f1b215d77b8be0ff1315ca25885f37dc27cacb7ac6db897d877f8a1
    slot: 8425th slot of 152nd epoch
    difficulty: 3290086
    leader: pub:9a6fa343
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:9a6fa343, dPk = pub:8b532076 } }
    block: v0.2.0
    software: cardano-sl:1
, MainBlockHeader:
    hash: 5a0795022c4786191d90eaf83f0a58c927d399a4779b1a61b78a0188de439c3a
    previous block: 197d5cfea25e990f6893e1250ea248ac25c0465db43abef95677b32c0d3ebbff
    slot: 8426th slot of 152nd epoch
    difficulty: 3290087
    leader: pub:0bdb1f5e
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:0bdb1f5e, dPk = pub:5fddeeda } }
    block: v0.2.0
    software: cardano-sl:1
, MainBlockHeader:
    hash: d571750aee77c352ae4a3be20b1f229d4e3d6c549668a06b2a19b3b8bc301843
    previous block: 5a0795022c4786191d90eaf83f0a58c927d399a4779b1a61b78a0188de439c3a
    slot: 8427th slot of 152nd epoch
    difficulty: 3290088
    leader: pub:0bdb1f5e
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:0bdb1f5e, dPk = pub:5fddeeda } }
    block: v0.2.0
    software: cardano-sl:1
]
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.48 UTC] MemPool metrics wait: ApplyBlock queue length is 1
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.48 UTC] MemPool metrics acquire: ApplyBlock wait time was 12mcs
[cardano-sl.*production*:Info:ThreadId 382] [2020-01-17 07:31:07.48 UTC] Verifying and applying blocks...
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.48 UTC] Rolling: verifying
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.48 UTC] verifyBlocksPrefix: 64
[cardano-sl.*production*:Info:ThreadId 382] [2020-01-17 07:31:07.48 UTC] slogVerifyBlocks: Consensus era is Original
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.71 UTC] Rolling: Verification done, applying unsafe block
[cardano-sl.node:Debug:ThreadId 316] [2020-01-17 07:31:07.72 UTC] applying some blocks (non-rollback)
[cardano-sl.*production*:Info:ThreadId 382] [2020-01-17 07:31:07.74 UTC] Verifying and applying blocks done
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.74 UTC] MemPool metrics release: ApplyBlock modify time was 258859mcs size is 0
[cardano-sl.*production*:Debug:ThreadId 382] [2020-01-17 07:31:07.74 UTC] Not relaying block in recovery mode
[cardano-sl.*production*:Info:ThreadId 382] [2020-01-17 07:31:07.74 UTC] Blocks have been adopted: [19b1f1eec6f9abb1, a7b1b58758880db3, f25b190e0f961f05, 0e907e6cf3f6ca4d, d4f74c5d312b0549, e4d06fea6f988ce9, ee279270b61135e9, 6c7998c9ba54d17e, 94853e29ce7200e6, f08ed74a4fcfd8e3, 834b772bd7901ffb, fb514730332114d2, c8a4b3a5697044ee, 5c9a27718384810e, 98573eeb0e53dbd4, a3be980a5027b156, 3713b244551e732b, ff9fa80350026e9e, 1704bdb34a16c6be, 229b40d03c219294, dfcbec3a426b4ca3, 860e4a48c46e67fa, 40b75fe17d58e8bf, faddbd98928a5527, 7b1d29bc099b5b19, ab08ff7a02399e92, ae699dcf30f5ef08, a6d99576b5002d9a, 62b679c913e1c9ee, 4d01d5340c42fdb3, 196d33de29e35ae9, 98ee850aac9a1b9d, 4f3037f91ccd71fd, 24fbc16a4479d02f, 53ac78eb4d6ab7f2, d48b6feb41138da5, 676d449b1ca1fb74, 3327651659423dee, 1a3c9a6c2f095cd7, 16b36ed4da3e05ff, 2223acc65da5f5db, e8d4e12a616f4227, 443ff6e4fffeac0b, 22db605e0059c4d8, 643d9645b54e09b5, fdf7bfe14b34aed0, 6a1650622dec4d07, 1af30b82a5ee01a8, 1e585328b2eda019, db55785d3ea21869, 0ac37f9d03244062, cc9d0a297ea9af53, ebffb3fc15d7b62b, e7b81541b23ace3a, a48604d65cef0180, 7af2bd4c948d283f, 013dc0ea380f67a1, de88e7e0c354e93a, a762148ba5fbf0a6, 432652c976cbbdc4, 14b798e52f1b215d, 197d5cfea25e990f, 5a0795022c478619, d571750aee77c352]
^Z[9]   Segmentation fault      nohup sudo ./mainnet.sh > log.log 2>&1  (wd: /data/ada/cardano-sl)
(wd now: /data/ada/cardano-sl/state-wallet-mainnet/logs/pub)

[erikd]

What version (use the git hash) is this?

Is this repeatable? Ie, it you run it again, do you get the same error?

[shenyaqi9527]

master 1a792d7 [origin/master] Merge #4242
yes,this is repeatable
@erikd

[erikd]

That git hash is from Sep 2019.

Try checking out tag 3.2.0 which is from Nov 22, 2019.

If that does not fix it, I will look at this on Monday morning. Its currently 9pm Friday here.

[shenyaqi9527]

thank you @erikd

[shenyaqi9527]

@erikd
I use "nix-build -A connectScripts.mainnet.wallet -o mainnet.sh" command to build.
mainnet.sh:
#!/nix/store/vlb7kcc1k035vpyrgsj9kk7380yh68wd-bash-4.4-p23/bin/bash

set -euo pipefail

if [[ "${1-}" == "--delete-state" ]]; then
echo "Deleting state-wallet-mainnet ... "
rm -Rf state-wallet-mainnet
shift
fi
if [[ "${1-}" == "--runtime-args" ]]; then
RUNTIME_ARGS="${2-}"
shift 2
else
RUNTIME_ARGS=""
fi

echo "Keeping state in state-wallet-mainnet"
mkdir -p state-wallet-mainnet/logs

echo "Launching a node connected to 'mainnet' ..."
export LC_ALL=en_GB.UTF-8
export LANG=en_GB.UTF-8

if [ ! -d state-wallet-mainnet/tls ]; then
mkdir -p state-wallet-mainnet/tls/server && mkdir -p state-wallet-mainnet/tls/client
/nix/store/ra45xgy1ngy9bpn12h5fib7m81925i80-cardano-sl-tools-3.2.0-exe-cardano-x509-certificates/bin/cardano-x509-certificates
--server-out-dir state-wallet-mainnet/tls/server
--clients-out-dir state-wallet-mainnet/tls/client
--configuration-file /nix/store/r02jsbcld1cmy47y1cxr8c9l6y9z7a8n-tls-config-mainnet.yaml
--configuration-key mainnet_full
fi
ln -sf /nix/store/0gzajk6rskv7xigvwhgly1zrn3m75d4r-curl-wallet-mainnet state-wallet-mainnet/curl

exec /nix/store/ya8iqz0l34w9mszd06ir3pchasryqz4a-cardano-wallet-3.2.0-exe-cardano-node/bin/cardano-node
--configuration-file /nix/store/j4rz117v3paa3ys3abfkxacvghyd7chn-cardano-sl-config/lib/configuration.yaml --configuration-key mainnet_full
--tlscert state-wallet-mainnet/tls/server/server.crt
--tlskey state-wallet-mainnet/tls/server/server.key
--tlsca state-wallet-mainnet/tls/server/ca.crt
--log-config /nix/store/j4rz117v3paa3ys3abfkxacvghyd7chn-cardano-sl-config/log-configs/connect-to-cluster.yaml
--topology "/nix/store/kiwxslk8q90j8rrjj4vqnnc9np5a9bhy-topology-mainnet"
--logs-prefix "state-wallet-mainnet/logs"
--db-path "state-wallet-mainnet/db"
--wallet-db-path 'state-wallet-mainnet/wallet-db'
--no-client-auth

--keyfile state-wallet-mainnet/secret.key
--wallet-address 0.0.0.0:8090
--wallet-doc-address 127.0.0.1:8091
--ekg-server 127.0.0.1:8000 --metrics
+RTS -N2 -qg -A1m -I0 -T -RTS

$RUNTIME_ARGS

[erikd]

I use "nix - build - A connectScripts.mainnet.wallet - o mainnet.sh" command to build.

There seem to be some extra spaces around the - character in that. It should be nix-build -A and -o.

I just did:

> git checkout 3.2.0 -b tag-3.2.0
> nix-build -A connectScripts.mainnet.wallet -o mainnet.sh

and it worked as expected.

[shenyaqi9527]

@erikd
That's probably why I copied it.It added the space for me automatically.

The process ended abruptly.
same error

[cardano-sl.*production*:Debug:1689] [2020-01-20 02:54:16.92 UTC] Rolling: verifying
[cardano-sl.*production*:Debug:1689] [2020-01-20 02:54:16.92 UTC] verifyBlocksPrefix: 64
[cardano-sl.*production*:Info:1689] [2020-01-20 02:54:16.92 UTC] slogVerifyBlocks: Consensus era is Original
^Z[5]   Segmentation fault      sudo nohup ./mainnet.sh > log.log 2>&1

[6]+  Stopped                 tail -200f log.log

Do I need to upgrade my server? now my server : 2c 4GB RAM

[erikd]

4G should be enough RAM, especically if nothing else is happening on that machine.

I am currently running the ./mainnet.sh script. Any idea what epoch you are getting up to when it segfaults?

[erikd]

And now it segfaults for me too! At 5046th slot of 3rd epoch.

[shenyaqi9527]

I think this error occurs when the process reaches 1gb of memory.Because I restarted the process it can continue to synchronize.I rebooted once, now at 10365th slot of 13th epoch

[erikd]

I am running on a 16G VM, and I was able to recreate that problem so that is not it.

Oh, hang on, you are running on a 64 bit CPU aren't you?

[erikd]

I also checked out the HEAD of the develop branch and that synced to epoch 4 without a problem.

Then I switched back to the 3.2.0 tag ans synced from scratch to epoch 5, again without a problem.

I wonder if there is a peer somewhere on the network that is serving up corrupted blocks.

[shenyaqi9527]

admin@ada:/data/ada$ getconf LONG_BIT
64

On Friday, I restarted the node an infinite number of times and finished synchronizing. On Monday, the process was killed.

[erikd]

What base OS and OS version are you running this on?

[shenyaqi9527]

Ubuntu 18.04

[erikd]

Ubuntu 18.04 should be fine.

I would try deleting the ./state-wallet-mainnet directory, and then try resyncing. Each time it segfaults, record the slot and epoch number and restart it. When you have about 10 entries, post the list here.

[shenyaqi9527]

8280th slot of 8th epoch
18274th slot of 18th epoch
20293rd slot of 19th epoch
7566th slot of 31st epoch
1618th slot of 33rd epoch
4089th slot of 36th epoch
15837th slot of 55th epoch

[erikd]

Ok, if you delete the ./state-wallet-mainnet directory and run it again listing the first 10 entries here.

[shenyaqi9527]

What do you mean?

[erikd]

Delete ./state-wallet-mainnet directory and do the same test again. Would be useful to know if we get the same results.

[shenyaqi9527]

Do I need to delete this directory when I run it again

[erikd]

Yes. Thats is the state directory where the node stores blocks.

[shenyaqi9527]

[cardano-sl.*production*:Debug:1689] [2020-01-20 04:56:29.75 UTC] Handling block w/ LCA, which is a07d3104
[cardano-sl.*production*:Info:1689] [2020-01-20 04:56:29.75 UTC] Trying to apply blocks w/o rollback. First 3: [MainBlockHeader:
    hash: 92d68c2ba61d115b3c53f1c857c77355062c2ee76680e8afed94980e7fbba239
    previous block: a07d310498f2417e1d6ade2dd2e3da8f802995407845d164616053dfc683a44d
    slot: 18033rd slot of 26th epoch
    difficulty: 579559
    leader: pub:9a6fa343
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:9a6fa343, dPk = pub:8b532076 } }
    block: v0.1.0
    software: cardano-sl:0
, MainBlockHeader:
    hash: c5342a48f472640576a177d2992b77f7939e406e463d00bc480ddb339747e85f
    previous block: 92d68c2ba61d115b3c53f1c857c77355062c2ee76680e8afed94980e7fbba239
    slot: 18034th slot of 26th epoch
    difficulty: 579560
    leader: pub:0bdb1f5e
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:0bdb1f5e, dPk = pub:5fddeeda } }
    block: v0.1.0
    software: cardano-sl:0
, MainBlockHeader:
    hash: d8462f486a46f0688786e3880c7982a6f69ce8f0adcf18c1e1089bb9480b7490
    previous block: c5342a48f472640576a177d2992b77f7939e406e463d00bc480ddb339747e85f
    slot: 18035th slot of 26th epoch
    difficulty: 579561
    leader: pub:9a6fa343
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:9a6fa343, dPk = pub:8b532076 } }
    block: v0.1.0
    software: cardano-sl:0

The log is not finished. The process is killed.But this time no errors were reported.

[erikd]

@shenyaqi9527 I need a list of where (ie epoch and slot number) the process gets killed, started from scratch. Please run it again so I can compare it with the last list.

[shenyaqi9527]

Do I need to delete "./state-wallet-mainnet"

[shenyaqi9527]

[cardano-sl.*production*:Debug:1686] [2020-01-20 05:11:51.96 UTC] verifyBlocksPrefix: 64
[cardano-sl.*production*:Info:1686] [2020-01-20 05:11:51.96 UTC] slogVerifyBlocks: Consensus era is Original
cardano-node: internal error: evacuate: strange closure type 0
    (GHC version 8.4.4 for x86_64_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

[erikd]

I am really beginning to suspect that your machine is having hardware issues. Can you run some form of diagnostic on it?

[erikd]

What epoch/slot are you up to?

[shenyaqi9527]

14651st slot of 54th epoch
root 20 0 1.001t 959396 21924 S 120.6 23.8 31:46.79 cardano-node

[shenyaqi9527]

17635th slot of 67th epoch.This is where the process is killed.

[74788.914053] cardano-node:w[8914]: segfault at 840ccae7a0 ip 00007f4de11d9c55 sp 00007f4dc59cea98 error 6 in libc-2.27.so[7f4de1088000+1aa000]

[erikd]

The node has been running for over 20 hours under valgrind and has not yet failed.

Unfortunately, valgrind runs things at last 10 times slower, so it has only managed to sync from zero to epoch 59 in that time.

I am also a bit concerned that valgrind itself may reduce the chance of the bug being triggered.

[erikd]

I gave up on that valgrind run so I could try running it under gdb. Unfortunately the heavy use of exceptions within the node app means gdb cannot be used.

Trying without valgrind again.

[erikd]

Another segfault (without valgrind):

Jan 22 12:30:26 nix kernel: [150967.689171] cardano-node:w[26733]: segfault at 841eca1560 ip 00007f47db188d6e sp 00007f47a6ff8a98 error 6 in libc-2.27.so[7f47db037000+1aa000]
Jan 22 12:30:26 nix kernel: [150967.689177] Code: ff ff 0f 18 89 40 fe ff ff c5 fe 6f 01 c5 fe 6f 49 e0 c5 fe 6f 51 c0 c5 fe 6f 59 a0 48 81 e9 80 00 00 00 48 81 ea 80 00 00 00 <c4> c1 7d e7 01 c4 c1 7d e7 49 e0 c4 c1 7d e7 51 c0 c4 c1 7d e7 59

According to this, error 6 means "(data) write to an unmapped area".

This is probably due to some C code accessed via the C FFI.

[erikd]

I noticed that the Nix build that we are using is linking to version 5.11 of RocksDB whereas Debian has version 5.17 of that library.

Now trying to native Debian build of cardano-node rather than the Nix build.

[erikd]

And I almost immediately got a segfault with the version I built without Nix. 😢

[erikd]

Currently building a profiled version of cardano-node under Debian. Hoping that gives a proper traceback. 🤞

[erikd]

Ok, profling got me my first traceback:

*** Exception (reporting due to +RTS -xc): (THUNK), stack trace:
  Pos.DB.Block.Internal.putSerializedBlunds.\,
  called from Pos.DB.Block.Internal.putSerializedBlunds,
  called from Pos.DB.Block.Internal.dbPutSerBlundsRealDefault,
  called from Cardano.Wallet.Kernel.Mode.dbPutSerBlunds,
  called from Pos.DB.Block.Load.putBlunds,
  called from Pos.DB.Block.Slog.Logic.slogApplyBlocks,
  called from Pos.DB.Block.Logic.Internal.applyBlocksDbUnsafeDo,
  called from Pos.DB.Block.Logic.Internal.applyBlocksUnsafe.app,
  called from Pos.DB.Block.Logic.Internal.applyBlocksUnsafe,
  called from Pos.DB.Block.Logic.VAR.rollingVerifyAndApply.\,
  called from Pos.DB.Block.Logic.VAR.rollingVerifyAndApply,
  called from Pos.DB.Block.Logic.VAR.verifyAndApplyBlocks,
  called from Pos.Network.Block.Logic.applyWithoutRollback.applyWithoutRollbackDo,
  called from Pos.DB.GState.Lock.stateLockHelper.\,

Going to run it a couple of more times to make sure it crashes the same way each time.

[erikd]

Have run this a number of times and always get a traceback starting with putSerializedBlunds.

@dcoutts asked why there is no C code in that traceback. I think that is because of incompatible debug formats. GHC's profiling uses Dwarf debugging symbols and the C code in the backtrace either may not be enabled or may be an incompatible format.

[shenyaqi9527]

Has the problem been dealt with? @erikd

[erikd]

It has not. I have been assigned to other higher priority work.

[shenyaqi9527]

No one is dealing with this problem right now? @erikd

[erikd]

No one that I am aware of. It seems that you are one of the few people who has been hitting this problem, and hence the priority has been downgraded.

My advice is to set the node up as a systemd service or something and have it restart automatically.

[DZDomi]

We are also running into the same issue with all of our 3 nodes. This is the Dockerfile we are using to run our node:

FROM nixos/nix:2.3

ENV CARDANO_VERSION 3.2.0

RUN apk update && \
    apk add git curl bzip2 bash

ADD nix.conf /etc/nix/nix.conf

WORKDIR /opt

RUN git clone https://github.com/input-output-hk/cardano-sl.git

WORKDIR /opt/cardano-sl

RUN git checkout $CARDANO_VERSION
RUN nix-build -A connectScripts.mainnet.wallet -o connect-to-mainnet && \
    nix-build -A connectScripts.mainnet.explorer -o connect-explorer-to-mainnet && \
    nix-build -A connectScripts.testnet.wallet -o connect-to-testnet && \
    nix-build -A connectScripts.testnet.explorer -o connect-explorer-to-testnet

ADD entrypoint.sh .

ENTRYPOINT ["./entrypoint.sh"]

entrypoint.sh:

#!/usr/bin/env bash

if [[ "$1" == "explorer" ]]; then
  if [[ -n "$TESTNET" ]]; then
    exec ./connect-explorer-to-testnet --runtime-args --web-port="$RPC_PORT"
  else
    exec ./connect-explorer-to-mainnet --runtime-args --web-port="$RPC_PORT"
  fi
fi

if [[ "$1" == "node" ]]; then
  if [[ -n "$TESTNET" ]]; then
    sed -i "s/--wallet-address 127.0.0.1:8090/--wallet-address 0.0.0.0:$RPC_PORT/g" connect-to-testnet
    exec ./connect-to-testnet --runtime-args --no-tls
  else
    sed -i "s/--wallet-address 127.0.0.1:8090/--wallet-address 0.0.0.0:$RPC_PORT/g" connect-to-mainnet
    exec ./connect-to-mainnet --runtime-args --no-tls
  fi
fi

nix.conf:

cores = 0
max-jobs = auto
sandbox = false
substituters = https://hydra.iohk.io https://cache.nixos.org
trusted-substituters =
trusted-public-keys  = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

@erikd

[michdr]

I can confirm that it happens both in v3.2.0 and v3.1.0.
Any estimate when is this bug going to be taken care of? @erikd

[erikd]

The code base is this repo has been maintenance mode for close to a year. This bug is difficult to reproduce suggesting it is machine specific (insufficient memory?). As such it almost certainly will never be fixed.

However, the new code base in the cardano-node repository is nearing completion and can actually connect to mainnet and operate as a full node. It does not yet work with Daedalus. I am not sure if it currently connects to Daedalus.

To provide the best advice on your best path forward, it would be useful if you could tell me your goals.

[michdr]

Thanks @erikd . So, is cardano-node expected to be a de-facto replacement for cardano-sl?

The goal is to be able to operate a full and reliable node for various operations on the mainnet (create transactions, verify existing ones, etc.).

[erikd]

So, is cardano-node expected to be a de-facto replacement for cardano-sl?

Yes!