Amulet Security Advisory for CometBFT: ASA-2023-002
Component: CometBFT
Criticality: Low
Affected versions: All
Affected users: Validators, Chain Builders + Maintainers
Summary
A default configuration in CometBFT has been found to be large for common use cases, and may affect block times and consensus participation when fully utilized by chain participants. It is advised that chains consider their specific needs for their use case when setting the BlockParams.MaxBytes consensus parameter. Chains are encouraged to evaluate the impact of having proposed blocks with the maximum allowed block size, especially on bandwidth usage and block latency. Additionally, the timeout_propose parameter should be computed using the maximum allowed block size as a reference. This issue does not represent an actively exploitable vulnerability that would result in a direct loss of funds, however it may have a slight impact on block latency depending on a network’s topography.
When setting a large BlockParams.MaxBytes, there are two main implications:
- Increased bandwidth to propagate a block
- Increased latency to propagate a block
When combined, this may result in less round participation, and in some cases additional rounds may be required to meet the consensus threshold, which could lead to timeouts depending on the topography of a network and environmental factors. These factors can include the number of validators on a network, geographic distribution, network connectivity (including latency, bandwidth, and reachability), the functionality of the modules implementing the logic for a transaction in your chain, etc. The cost to propagate a 21MB block, the default value for BlockParams.MaxBytes, will be far higher than the cost of propagating a smaller 1MB block. CometBFT recommends tuning this parameter to a smaller limit if full initial-round participation is an important quality for your chain.
Considerations
CometBFT is designed to be configurable by chains, and implements many different configuration variables and parameters to allow chain developers, validators, node operators, and chain participants to customize it best to their use case. A high-performing validator may find it necessary to experiment with tuning local configuration, optimizing network and compute resources, and implementing controls to inhibit spam.
Next Steps for Chains and Validators
To increase awareness of the potential impacts of this default parameter, the CometBFT team has updated the documentation (cometbft/cometbft#1405, v0.34.x, v0.37.x, v0.38.x) for builders and maintainers of chain applications. Additionally, it is recommended that:
-
Chain ecosystems and their maintainers set a BlockParams.MaxBytes configuration appropriate for their use case at the application level; in some cases, fine-tuning BlockParams may require a network upgrade.
-
Chain ecosystems and their maintainers evaluate how gas is used and required on their chain, including gas and fee parameters, no-fee or fee-exempt message policies, and ensure that any custom modules integrate with the gas and fee frameworks. This is especially important for chains that may have implemented custom modules or functionality to allow IBC messages to bypass fees.
-
Chain ecosystems and their maintainers audit all of their currently-set parameters and configurations to ensure that they are appropriate for their use case.
-
All validators develop and implement anti-spam measures on their nodes. Amulet encourages validators to form working groups to collaborate on spam prevention and on tooling that can be implemented by node operators across the Interchain.
-
All validators consider developing and implementing tooling that would allow them to sample incoming transactions to enable them to fine-tune the level of service they would like to provide to be resilient in slowdown scenarios. Amulet also encourages validators to collaborate on tooling that can be implemented by node operators across the Interchain.
The CometBFT team has also revisited all the checks performed by the consensus protocol regarding proposed blocks. This investigation has confirmed that proposed blocks with size exceeding the BlockParams.MaxBytes limit established by the application are not accepted by nodes. The team notwithstanding has decided to introduce additional sanity checks for the size of proposed blocks (cometbft/cometbft#1408), allowing for an early identification and rejection of invalid or oversized blocks. These code changes will be included in the next release of each branch of CometBFT.
As more chains adopt the Interchain Stack for new and cutting-edge use cases, the CometBFT team recommends that all chains regularly evaluate their parameters and configurations to ensure they meet the needs of their ecosystem as their networks mature.
For more information about CometBFT, see https://docs.cometbft.com.
This issue was reported via the vulnerability disclosure channel at security@interchain.io on Friday, September 23, 2023. If you believe you have found a bug in the Interchain Stack or would like to contribute to the program by reporting a bug, please see https://hackerone.com/cosmos.
Note from Amulet on the Security Advisory Process:
In the interest of timely resolution of this issue for validators and node operators, the Amulet team has chosen to use existing processes and resources for distributing security advisories within the Cosmos and Interchain Ecosystems. Stay tuned as we implement an improved, more robust security advisory distribution system that will provide equitable access to information about security issues in the Interchain Stack.
References
Amulet Security Advisory for CometBFT: ASA-2023-002
Component: CometBFT
Criticality: Low
Affected versions: All
Affected users: Validators, Chain Builders + Maintainers
Summary
A default configuration in CometBFT has been found to be large for common use cases, and may affect block times and consensus participation when fully utilized by chain participants. It is advised that chains consider their specific needs for their use case when setting the
BlockParams.MaxBytesconsensus parameter. Chains are encouraged to evaluate the impact of having proposed blocks with the maximum allowed block size, especially on bandwidth usage and block latency. Additionally, thetimeout_proposeparameter should be computed using the maximum allowed block size as a reference. This issue does not represent an actively exploitable vulnerability that would result in a direct loss of funds, however it may have a slight impact on block latency depending on a network’s topography.When setting a large
BlockParams.MaxBytes, there are two main implications:When combined, this may result in less round participation, and in some cases additional rounds may be required to meet the consensus threshold, which could lead to timeouts depending on the topography of a network and environmental factors. These factors can include the number of validators on a network, geographic distribution, network connectivity (including latency, bandwidth, and reachability), the functionality of the modules implementing the logic for a transaction in your chain, etc. The cost to propagate a 21MB block, the default value for
BlockParams.MaxBytes, will be far higher than the cost of propagating a smaller 1MB block. CometBFT recommends tuning this parameter to a smaller limit if full initial-round participation is an important quality for your chain.Considerations
CometBFT is designed to be configurable by chains, and implements many different configuration variables and parameters to allow chain developers, validators, node operators, and chain participants to customize it best to their use case. A high-performing validator may find it necessary to experiment with tuning local configuration, optimizing network and compute resources, and implementing controls to inhibit spam.
Next Steps for Chains and Validators
To increase awareness of the potential impacts of this default parameter, the CometBFT team has updated the documentation (cometbft/cometbft#1405, v0.34.x, v0.37.x, v0.38.x) for builders and maintainers of chain applications. Additionally, it is recommended that:
Chain ecosystems and their maintainers set a
BlockParams.MaxBytesconfiguration appropriate for their use case at the application level; in some cases, fine-tuningBlockParamsmay require a network upgrade.Chain ecosystems and their maintainers evaluate how gas is used and required on their chain, including gas and fee parameters, no-fee or fee-exempt message policies, and ensure that any custom modules integrate with the gas and fee frameworks. This is especially important for chains that may have implemented custom modules or functionality to allow IBC messages to bypass fees.
Chain ecosystems and their maintainers audit all of their currently-set parameters and configurations to ensure that they are appropriate for their use case.
All validators develop and implement anti-spam measures on their nodes. Amulet encourages validators to form working groups to collaborate on spam prevention and on tooling that can be implemented by node operators across the Interchain.
All validators consider developing and implementing tooling that would allow them to sample incoming transactions to enable them to fine-tune the level of service they would like to provide to be resilient in slowdown scenarios. Amulet also encourages validators to collaborate on tooling that can be implemented by node operators across the Interchain.
The CometBFT team has also revisited all the checks performed by the consensus protocol regarding proposed blocks. This investigation has confirmed that proposed blocks with size exceeding the
BlockParams.MaxByteslimit established by the application are not accepted by nodes. The team notwithstanding has decided to introduce additional sanity checks for the size of proposed blocks (cometbft/cometbft#1408), allowing for an early identification and rejection of invalid or oversized blocks. These code changes will be included in the next release of each branch of CometBFT.As more chains adopt the Interchain Stack for new and cutting-edge use cases, the CometBFT team recommends that all chains regularly evaluate their parameters and configurations to ensure they meet the needs of their ecosystem as their networks mature.
For more information about CometBFT, see https://docs.cometbft.com.
This issue was reported via the vulnerability disclosure channel at security@interchain.io on Friday, September 23, 2023. If you believe you have found a bug in the Interchain Stack or would like to contribute to the program by reporting a bug, please see https://hackerone.com/cosmos.
Note from Amulet on the Security Advisory Process:
In the interest of timely resolution of this issue for validators and node operators, the Amulet team has chosen to use existing processes and resources for distributing security advisories within the Cosmos and Interchain Ecosystems. Stay tuned as we implement an improved, more robust security advisory distribution system that will provide equitable access to information about security issues in the Interchain Stack.
References