Add vm.catchRevert / vm.ignoreRevert functionality, to allow better fuzz testing for complicated cases #7246
Description
Component
Forge
Describe the feature you would like
Problem
Fuzz tests are very useful to check specific conditions about complicated math and logic across the range of all possible inputs. Most smart contracts have a variety of safety checks like safe arithmetic, input sanitization etc. that result in many different error codes.
If someone wants to write a fuzz test that checks only one aspect of the logic across the whole range of inputs, they currently have to handle all other possible reverts explicitly. In many cases, this is equivalent to the task of rewriting much of the smart contract logic.
Example
For example, in this test -
function test_swap_amm_pathIndependence(
bool _isZeroToOne,
uint256 _reserve0,
uint256 _reserve1,
uint256 _amountIn,
uint160 _sqrtSpotPriceX96,
uint160 _sqrtPriceLowX96,
uint160 _sqrtPriceHighX96
) public{}
If someone just wanted to check the following things, across all possible variations of reserves, swap conditions, and spot price ranges:
- Liquidity never changes after a swap
- A swap of equal amounts in the reverse direction can never extract more money than what they started off with.
- The swap reverts if the price equals the lower and higher ranges
etc.
It is very hard to write a test like this currently, because across this wide range of inputs, we could receive hundreds of different errors because of invalid states like -
- ERC20 Errors - insuffienct amount, insufficient allowance etc.
- Arithmetic Errors - Overflow, underflow, division by 0
- Smart contract-specific errors - Maybe the spot price is out of bounds, or maybe reserves are incorrectly distributed.
It becomes almost impossible to handle all of these errors explicitly in the test. We know that these errors represent the correct functioning of the program, so we simply want to ignore them and move on to the next fuzz run, or the next part of the test.
I couldn't find a straightforward way to achieve this in foundry.
expectRevert does not work here, because for an external call there are some errors that may or may not occur depending upon the input.
To write expectRevert statements for all possible errors, would be like recreating the smart contract logic in the tests.
Solution
We introduce new cheat codes in the VM library called ignoreRevert and catchRevert. These instruct the tests that there is a possibility of a revert here, but we don't want to fail the test because of this.
There can be some variations of this -
ignoreAllReverts- ignore any revert that comes out of this specific function call.catchRevert(error code)- if a revert happens, then go to this block, else continue execution normally.catchAllReverts
Additional context
The Patchy Workaround
There is a possible workaround that involves using try-catch statements for such tests.
Example
try pool.swap(params) returns (uint256, uint256 amountOutSecond) {
assertGe(_amountIn, amountOutSecond, 'pathIndependence');
} catch (bytes memory reason) {
if (keccak256(reason) == keccak256(abi.encodePacked(hex'd19ac625'))) {
console.log('Reverted because of 0 amountOut');
return;
} else {
emit LogBytes(reason);
revert('revert swap 2');
}
}
} catch {
- You need to manually calculate and check the error codes, for the errors that you care about handling.
- As the number of calls where you want to ignore the errors increases, the try-catch nesting becomes very complicated and confusing.