Ptrus/feature/txsource crash points

[ptrus]

Fixes: #3409

TODO:

• test a daily txsource run with crash points enabled: https://buildkite.com/oasisprotocol/oasis-core-long-tests/builds/288#e13c4fc6-83cd-4381-83be-cf1ba1fd283b
  • wtih current configuration there were 75 crashes in the above 4 hour test run, probably can reduce the probability a bit further 😁
• test run with longepochs & crash points enabled: https://buildkite.com/oasisprotocol/oasis-core-long-tests/builds/304#29a1b084-4444-4eae-83fd-62738a0f904c
  • reduced the prob. by a half from the previous run, now there's about ~10 crashes per hour

[codecov]

Codecov Report

Merging #3449 (9756566) into master (a15d7dd) will increase coverage by 0.06%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##           master    #3449      +/-   ##
==========================================
+ Coverage   66.38%   66.45%   +0.06%     
==========================================
  Files         373      373              
  Lines       33943    33954      +11     
==========================================
+ Hits        22534    22564      +30     
+ Misses       8131     8124       -7     
+ Partials     3278     3266      -12     

Impacted Files	Coverage Δ
go/common/crash/crash.go	82.05% <75.00%> (-3.03%)	⬇️
go/oasis-node/cmd/common/metrics/disk.go	65.38% <0.00%> (-19.24%)	⬇️
go/oasis-node/cmd/common/metrics/resource.go	78.94% <0.00%> (-10.53%)	⬇️
go/staking/api/grpc.go	51.18% <0.00%> (-6.30%)	⬇️
go/consensus/tendermint/apps/staking/query.go	35.71% <0.00%> (-4.77%)	⬇️
go/storage/mkvs/lookup.go	72.63% <0.00%> (-4.22%)	⬇️
.../consensus/tendermint/apps/roothash/state/state.go	74.28% <0.00%> (-2.86%)	⬇️
go/consensus/tendermint/staking/staking.go	54.02% <0.00%> (-2.30%)	⬇️
go/consensus/tendermint/roothash/roothash.go	70.76% <0.00%> (-1.24%)	⬇️
go/consensus/tendermint/abci/mux.go	59.46% <0.00%> (-1.14%)	⬇️
... and 28 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a15d7dd...9756566. Read the comment docs.