blst gives "illegal instruction"

[necrotic1979]

🐞 Bug Report

Description

Tried using blst for the first time today after updating to version 26. Crashed immediately on starting to sync. Repeated a few times with same results.

Has this worked before in a previous version?

No, blst is new to version 26

🔬 Minimal Reproduction

command line as follows: user@debianserver:~/prysm$ ./prysm.sh beacon-chain --http-web3provider=https://goerli.infura.io/v3/4cc7cc8813efb6154d72ba71d0c1 --blst

🔥 Error

SIGILL: illegal instruction
PC=0x146f2a1 m=3 sigcode=2

This occurs after the node starts syncing as follows:

"

Beacon chain is up to date.
Verifying binary integrity.
beacon-chain-v1.0.0-alpha.26-linux-amd64: OK
gpg: Signature made Mon 21 Sep 2020 05:05:20 PM MST
gpg: using RSA key 0AE0051D647BA3C1A917AF4072E33E4DF1A5036E
gpg: Good signature from "Preston Van Loon preston@prysmaticlabs.com" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 0AE0 051D 647B A3C1 A917 AF40 72E3 3E4D F1A5 036E
Verified /home/djohnston/prysm/dist/beacon-chain-v1.0.0-alpha.26-linux-amd64 has been signed by Prysmatic Labs.
Starting Prysm beacon-chain --http-web3provider=https://goerli.infura.io/v3/4cc7cc8813f621bgb6154d72ba71d0c1 --blst
[2020-09-22 14:05:47] INFO flags: Using "max_cover" strategy on attestation aggregation
[2020-09-22 14:05:47] WARN flags: Enabling new BLS library blst
[2020-09-22 14:05:47] INFO node: Checking DB database-path=/home/djohnston/.eth2/beaconchaindata
[2020-09-22 14:05:47] INFO node: Starting beacon node version=Prysm/v1.0.0-alpha.26/303edbde58c2a6883ee59ec8d57eb7d53e5ad867. Built at: 2020-09-21 22:25:58+00:00
[2020-09-22 14:05:47] INFO rpc: RPC-API listening on port address=127.0.0.1:4000
[2020-09-22 14:05:47] INFO gateway: Starting JSON-HTTP API address=127.0.0.1:3500
[2020-09-22 14:05:47] WARN rpc: You are using an insecure gRPC server. If you are running your beacon node and validator on the same machines, you can ignore this message. If you want to know how to enable secure connections, see: https://docs.prylabs.network/docs/prysm-usage/secure-grpc
[2020-09-22 14:05:47] INFO blockchain: Blockchain data already exists in DB, initializing...
[2020-09-22 14:05:47] INFO slotutil: Chain genesis time reached genesisTime=2020-08-04 06:00:08 -0700 MST genesisValidators=20084
[2020-09-22 14:05:48] INFO powchain: Connected to eth1 proof-of-work chain endpoint=https://goerli.infura.io/v3/4cc7cc8813e643bfb6154d72ba71d0c1
[2020-09-22 14:05:48] INFO rpc: New gRPC client connected to beacon node addr=127.0.0.1:53508
[2020-09-22 14:05:48] INFO initial-sync: Starting initial chain sync...
[2020-09-22 14:05:48] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=0
[2020-09-22 14:05:48] INFO p2p: Started discovery v5 ENR=enr:-LK4QII8Zo6pssaiCU4kQHpy2igjsHuqV9LG7KxBlwKph8yJcHJ0qKELIpjXAL_EgQzczKrUAZIDqE0VGZYyPQpIxhwBh2F0dG5ldHOIAAAAAAAAAACEZXRoMpDnp11aAAAAAf__________gmlkgnY0gmlwhMCoAHKJc2VjcDI1NmsxoQPc5O0_RczSQgiJLfxWSp7_2-xsgdSd4RG2_MrzBFqZJoN0Y3CCMsiDdWRwgi7g
[2020-09-22 14:05:48] INFO p2p: Node started p2p server multiAddr=/ip4/192.168.0.114/tcp/13000/p2p/16Uiu2HAmTXGWRYBKCccducPi7YFDrbDsD7jMkZGpxsRH3nf54g9b
[2020-09-22 14:05:51] INFO p2p: Peer connected activePeers=3 direction=Outbound multiAddr=/ip4/27.104.206.44/tcp/13000/p2p/16Uiu2HAmTh5CZBQ2BXq3gJ7foop9ZrHeVfST5bkrUEoEV5SXhySe
[2020-09-22 14:05:52] INFO p2p: Peer connected activePeers=6 direction=Outbound multiAddr=/ip4/73.153.162.115/tcp/9000/p2p/16Uiu2HAmHtA9c421PUkbreVUSDRnXF2KuAn7QC2UwksiodjpudFt
[2020-09-22 14:05:53] INFO initial-sync: Waiting for enough suitable peers before syncing required=3 suitable=2
[2020-09-22 14:05:54] INFO p2p: Peer connected activePeers=5 direction=Outbound multiAddr=/ip4/70.95.235.194/tcp/9000/p2p/16Uiu2HAmDo5XinaW3y7t6XN7HLDQEv3SmzDgGa4CJoCvXn6ZAV3k
[2020-09-22 14:05:54] INFO p2p: Peer connected activePeers=6 direction=Outbound multiAddr=/ip4/3.132.193.49/tcp/9000/p2p/16Uiu2HAmDozn3ksBmKS83i9KD6QETymzPCTw9ZxszGYvmKxfDmDK
[2020-09-22 14:05:57] INFO p2p: Peer connected activePeers=6 direction=Outbound multiAddr=/ip4/97.113.102.194/tcp/9000/p2p/16Uiu2HAkym8851nxNH6HkQnv5MQUhunv91DB2vbfnU2poHyPBR7p
[2020-09-22 14:06:00] INFO p2p: Peer connected activePeers=7 direction=Outbound multiAddr=/ip4/46.93.197.62/tcp/9000/p2p/16Uiu2HAmEYUpARzqtw6mDyPjbpDPo6CXXgAEpmLRLh5DEB5Y1QGV
[2020-09-22 14:06:01] INFO initial-sync: Processing block batch of size 49 starting from 0xd7432a99... 355138/355229 - estimated time remaining 37s blocksPerSecond=2.5 peers=6
SIGILL: illegal instruction
PC=0x146f2a1 m=3 sigcode=2

full terminal output here:

blst_error_log.txt

🌍 Your Environment

Operating System:

setup is on a relatively bare bones linux machine (amd 64) running Debian 10 (Buster -stable) recently updated. Have been running version 24 for days without issues. I skipped updating to version 25.

What version of Prysm are you running? (Which release)

alpha.26

Anything else relevant (validator index / public key)?
Validator Index #34301) / 0xb8c566fb0c31030d25d4c78dbd5d1f643d70ec727a806a425b5a56037c0a427bad13a1e6b06dba114ef6d009f17c5169

[wackerow]

Had same issue after upgrading to alpha-26 and incorporating --blst flag. SIGILL: illegal instruction then a large console dump and crash.
Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-1018-raspi aarch64).

Staying on alpha-26 and removing --blst flag allowed beaconchain to resume normal functioning.

[MysticRyuujin]

This happening on my Intel NUC / Ubuntu / Docker setup as well.

beacon_crash.log

[nisdas]

Just to give an update I have opened an issue here:
supranational/blst#31 .

For now you can run without --blst to prevent the panics, hopefully we can get this fixed by the next release.

[nisdas]

Hey guys just to give an update #7352 was merged in which should fix the issues you guys have been encountering. This is slated to be included in alpha.27, so just keep a look out on that.

[0xKiwi]

Any updates on if you've encountered this issue again? @necrotic1979 @wackerow @MysticRyuujin ?

[wackerow]

Nope! Been working perfectly on my raspi! (edit: to clarify, yes I have been using the --blst flag and it has been working great)

[MysticRyuujin]

I haven't tried against since reporting. But I can try again next time I run a node.

[necrotic1979]

I haven't tried yet either. I'm using the binary so I dont want to restart my node until the next version is out (alpha.29 has that sync bug that I thankfully didn't update into). Will update next week after the next release

[canbal99]

I have a similar crash on RPi4 (Ubuntu 20.04) with --blst flag. No problem if I remove blst flag
Version Log: beacon-chain version Prysm/LocalBuild/f31d40cf347d871535625cc7f5a10f16afc72be8. Built at: 2020-10-19T13:04:48+0200
beaconchain.txt

NOTE: I don't have any problem with alpha 29 version. There must be an issue with my local go build (using go build instead bazel).
NOTE2: --blst flag working properly with bazel build

[necrotic1979]

--blst working now for me on the beta version