libs/autofile: panic when trying to checkHeadSizeLimit

[melekes]

Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile.(*Group).checkHeadSizeLimit(0xc4207975e0)
Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: /Users/philipstanislaus/go/src/github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile/group.go:225 +0xa9
Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile.(*Group).processTicks(0xc4207975e0)
Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: /Users/philipstanislaus/go/src/github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile/group.go:207 +0x2d
Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: created by github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile.(*Group).OnStart
Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: /Users/philipstanislaus/go/src/github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile/group.go:105 +0x3f

Refs #2392 (comment)

[melekes]

@philipstanislaus I can't find the error message itself, only the stack trace. Do you remember what was the error or maybe you still have some logs (I can't find the above panic in any of the log files you've provided)?

[philipstanislaus]

@melekes here is the complete log: https://s3.us-east-2.amazonaws.com/tendermint-testnet-issues/2018-10-09_testnet_disconnects_under_load/log-node0.txt

The error is on line 935359 of that file. Thanks!

[melekes]

There're a few lines missing (lines which contain the error message)

Okt 09 12:50:04 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: 02650070EE17ABA57300AB40729FCD8494C3199D (183 bytes)
Okt 09 12:50:12 ip-172-31-12-75.us-east-2.compute.internal thorchaind[2387]: github.com/thorchain/THORChain/vendor/github.com/tendermint/tendermint/libs/autofile.(*Group).ch
eckHeadSizeLimit(0xc4207975e0)

Example panic: https://gobyexample.com/panic

[melekes]

I think #2764 is the best thing we can do until we know the exact reason.

[melekes]

@philipstanislaus if you manage to find the error message, please comment on this issue.

[ebuchman]

Hmm weird that we can't see the top of the panic here. Are you using a defer/recover somewhere and not printing the full stack trace?

[philipstanislaus]

@ebuchman the reason is log rate limiting here. I think we are not going to find out what happened here – I will run the same load with the same configuration on the upgraded testnet without rate limited logs and comment here if I get similar results again, as @melekes suggested. Thanks for your help!