TON validator process is continuously crashing several times per day

[begetan]

TON validator process is continuously crashing several times per day

It should not crash, but should produce and error message if it is related to the configuration

The main thread crashes:

[ 1][t 0][2024-01-22 11:51:38.968182546][TcpListener.cpp:64][!TcpListener[port:14414]]  Accept
[pid 409601] [time 1705924299] Signal: 6
------- Stack Backtrace -------
/usr/bin/ton/validator-engine/validator-engine(+0x27b0f9)[0x56546b24f0f9]
/usr/bin/ton/validator-engine/validator-engine(+0x2793c6)[0x56546b24d3c6]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f73d7c40520]
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f73d7c949fc]
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f73d7c40476]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f73d7c267f3]
/usr/bin/ton/validator-engine/validator-engine(+0x2862e7)[0x56546b25a2e7]
/usr/bin/ton/validator-engine/validator-engine(+0x298b33)[0x56546b26cb33]
/usr/bin/ton/validator-engine/validator-engine(+0x286043)[0x56546b25a043]
/usr/bin/ton/validator-engine/validator-engine(+0x2c0bd5)[0x56546b294bd5]
/usr/bin/ton/validator-engine/validator-engine(+0x3fbf2d)[0x56546b3cff2d]
/usr/bin/ton/validator-engine/validator-engine(+0x2997f5)[0x56546b26d7f5]
/usr/bin/ton/validator-engine/validator-engine(+0x2a154a)[0x56546b27554a]
/usr/bin/ton/validator-engine/validator-engine(+0x29f649)[0x56546b273649]
/usr/bin/ton/validator-engine/validator-engine(+0x29f6ae)[0x56546b2736ae]
/usr/bin/ton/validator-engine/validator-engine(+0x1f5d10)[0x56546b1c9d10]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7f73d7c92ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7f73d7d24850]

Other threads produce warnings periodicaly:

[ 1][t 7][2024-01-22 11:51:30.538971216][manager.cpp:2380][!manager]    STATUS: last_masterchain_block_ago=328.5s last_known_key_block_ago=1976.5s shard_client_ago=344.5s
[ 0][t 7][2024-01-22 11:51:39.783803855][adnl-peer-table.cpp:234][!PeerTable][&it != local_ids_.end()]  dst=Y1pSAcU0Mv/hqOJUp9sSuD8/T7qEi84/q3IC9jYrD9w=
[ 0][t 7][2024-01-22 11:53:14.834421801][adnl-peer-table.cpp:234][!PeerTable][&it != local_ids_.end()]  dst=Y1pSAcU0Mv/hqOJUp9sSuD8/T7qEi84/q3IC9jYrD9w=

We deployed a node using MyTonCtrl. It missed proper validator configuration: bug

We restored wallet from the backup, populated mytocore.db file manually with validator wallet name and current ADNL address. It started to work, but randomly crashes multiply times per day.

Got it with MyTonCtrl
mytonctrl: b9d5937
validator: 9728bc6

There are multiply machines with the same spec: 16 vCPUs, 64GB RAM, 0.5TB NVMe.
Only one machine is affected.