Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Tendermint unresponsive under heavy load #1642
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Tendermint version (use
ABCI app (name for built-in, URL for self-written if it's publicly available):
You can reproduce the issue by doing the follwing
Here are the logs from tm-bench
Here are the logs of Tendermint in the unresponsive state.
It means either no txs are making it to Tendermint or there is a deadlock?
@melekes Following is how it looks like on my system
➜ tm-bench git:(master) ✗ nproc 8 ➜ tm-bench git:(master) ✗ uname -a Linux arch 4.14.40-1 #1 SMP PREEMPT Wed May 9 20:10:25 UTC 2018 x86_64 GNU/Linux ➜ tm-bench git:(master) ✗ tendermint version 0.19.3-aab98828
Also, the JSON RPC api became un-responsive i.e.
create_empty_blocks = false
@melekes I tried it out with
[consensus] max_block_size_txs = 2000 create_empty_block = false [mempool] size = 10000
My larger concern in undersanding how Tendermint works i.e. it seem to me that untill the mempool cap is hit tendermint keeps growing the mempool. Only after the cap is hit it switches context, creates and commits a block and then switches back to receiving more transaction untill again the mempool limit is reached. Is my observation even correct?
If such is the case then I would like to understand why does tendermint not take opportunity to create a block as soon as
Furthermore, I would also like understand that what
Hey, we upgraded to Tendermint 0.19.7. We are still experiencing the same issue, but now it runs longer before locking up.
This should not be the case. Tendermint should be making blocks as soon as txs are available.
I'm having trouble replicated anything like the mempool growing unbounded. Certainly with too many txs through tm-bench it seems the RPC will time out, but then become immediately responsive again (ie. I can re run
Right - committing a block with transactions can change the validity of txs in the mempool. Its really up to you and your app how important the recheck parameter is to you. It's fine to just include the txs and have them return errors during DeliverTx. Currently with the mempool, we guarantee that an honest/correct proposer will never include transactions that will error in DeliverTx.
I believe, this issue can be closed.
This was related not to Tendermint but a weird bug in docker-compose.
This should be fixed by #1673.
@olegomano could you perhaps double-check if you can still reproduce any of it?