Section 5 Lesson 12: Stateless Fuzzing

[Pedrog75]

Hi guys,

I'm trying to run my first stateless fuzzing test:

// SPDX-License-Identifier: SEE LICENSE IN LICENSE
pragma solidity ^0.8.18;

import { Test } from "forge-std/Test.sol";
import { StatelessFuzzCatches } from "src/invariant-break/StatelessFuzzCatches.sol";

contract StatelessFuzzCatchesTest is Test {
    StatelessFuzzCatches sfc;

    function setUp() public {
      sfc = new StatelessFuzzCatches();
    }

    function testFuzzCatchesBugStateless(uint128 randomNumber) public view {
      assert(sfc.doMath(randomNumber) != 0);
    }
}

However, when I run this command line: forge test --mt testFuzzCatchesBugStateless

I get this error:
proptest: Aborting shrinking after the PROPTEST_MAX_SHRINK_ITERS environment variable or ProptestConfig.max_shrink_iters iterations (set 0 to a large(r) value to shrink more; current configuration: 0 iterations)

First time, I get this kind of error, I run foundryup, still the same. Looked for a file or folder Proptest/ ProptestConfig nothing neither for PROPTEST_MAX_SHRINK_ITERS.

Someone have a clue on this?

Cheers

[EngrPips]

Hello @Pedrog75, I think this is a rust issue because when I searched for the error, I found https://github.com/rust-lang/cargo/issues/6258. I don't know how to resolve this, but maybe @PatrickAlphaC will have a solution for you. They will respond to this as soon as they are available.

[PatrickAlphaC]

Interesting. @Pedrog75 could you make this an issue on the foundry repo? Along with a step by step guide of how you get this error?

[Pedrog75]

Yes sure, I will do it, thank you!

[EngrPips]

Awesome, Please let us know how that plays out.

[Pedrog75]

Thank you for your support, I really appreciate it!

Just to let you know:
When myNumber == 2 as shown below:

// SPDX-License-Identifier: MIT
pragma solidity 0.8.20;

// INVARIANT: doMath should never return 0
contract StatelessFuzzCatches {
    /*
     * @dev Should never return 0
     */
    function doMath(uint128 myNumber) public pure returns (uint256) {
        if (myNumber == 2) {
            return 0;
        }
        return 1;
    }
}

I then run forge test --mt testFuzzCatchesBugStateless, I get this output:

Proptest: Aborting shrinking after the PROPTEST_MAX_SHRINK_ITERS environment variable or ProptestConfig.max_shrink_iters iterations (set 0 to a large(r) value to shrink more; current configuration: 0 iterations)

Ran 1 test for test/invariant-break/StatelessFuzzCatchesTest.t.sol:StatelessFuzzCatchesTest
[FAIL. Reason: panic: assertion failed (0x01); counterexample: calldata=0x7d8339960000000000000000000000000000000000000000000000000000000000000002 args=[2]] testFuzzCatchesBugStateless(uint128) (runs: 0, μ: 0, ~: 0)
Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 54.12ms (16.10ms CPU time)

Ran 1 test suite in 461.57ms (54.12ms CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)

Failing tests:
Encountered 1 failing test in test/invariant-break/StatelessFuzzCatchesTest.t.sol:StatelessFuzzCatchesTest
[FAIL. Reason: panic: assertion failed (0x01); counterexample: calldata=0x7d8339960000000000000000000000000000000000000000000000000000000000000002 args=[2]] testFuzzCatchesBugStateless(uint128) (runs: 0, μ: 0, ~: 0)

Encountered a total of 1 failing tests, 0 tests succeeded

In addition to the proptest issue, even if the test seems not to be running ((runs: 0, μ: 0, ~: 0)), it is still able to find the right output to break the invariant.

When I change myNumber == 2 to myNumber == 28 and test it again, the test runs 1938 times and finds the right value to break the invariant.

No files changed, compilation skipped
proptest: Aborting shrinking after the PROPTEST_MAX_SHRINK_ITERS environment variable or ProptestConfig.max_shrink_iters iterations (set 0 to a large(r) value to shrink more; current configuration: 0 iterations)
proptest: Saving this and future failures in cache/fuzz/failures

Ran 1 test for test/invariant-break/StatelessFuzzCatchesTest.t.sol:StatelessFuzzCatchesTest
[FAIL. Reason: panic: assertion failed (0x01); counterexample: calldata=0x7d833996000000000000000000000000000000000000000000000000000000000000001c args=[28]] testFuzzCatchesBugStateless(uint128) (runs: 1938, μ: 5659, ~: 5659)
Suite result: FAILED. 0 passed; 1 failed; 0 skipped; finished in 118.38ms (117.34ms CPU time)

Ran 1 test suite in 437.41ms (118.38ms CPU time): 0 tests passed, 1 failed, 0 skipped (1 total tests)

Failing tests:
Encountered 1 failing test in test/invariant-break/StatelessFuzzCatchesTest.t.sol:StatelessFuzzCatchesTest
[FAIL. Reason: panic: assertion failed (0x01); counterexample: calldata=0x7d833996000000000000000000000000000000000000000000000000000000000000001c args=[28]] testFuzzCatchesBugStateless(uint128) (runs: 1938, μ: 5659, ~: 5659)

Here is my foundry.toml file setup for fuzz tests:

[fuzz]
runs = 10000
seed = '0x2'

Overall, sometimes we are able to find a value to break the invariant with runs or without.

[EngrPips]

I don't understand this behavior of foundry. Let's look forward to their response. I know I will learn new stuff.

[Pedrog75]

From foundry contributor on my issue: [...]The prop test messages can be ignored, they'll be removed in future versions.[...]
Here is the link to the issue: foundry-rs/foundry#8064

[EngrPips]

That's interesting. So, the prop test message is just some weird stuff we should ignore.