feat: Add invariant testing #760

lucas-manuel · 2023-01-06T16:00:34Z

closes #497

gakonst · 2023-01-06T17:30:49Z

.vscode/settings.json

@@ -0,0 +1,3 @@
+{


mds1

Love it so far, just took a pass and left some small comments. Think I'll need to see it a bit more fleshed out before coming up with some naming ideas

projects/invariant_testing/Basic4626Deposit.sol

src/forge/advanced-testing.md

src/forge/invariant-testing.md

PaulRBerg · 2023-01-23T13:15:28Z

I also read through the first draft of this, and I like it a lot - it's straight to the point.

My documentation requests:

The fact that Forge considers all functions that start with invariant_ an invariant test - I think that this should be said at the beginning of the document.
Explain what the call_override config option does.
Explain how targetSender and excludeSender work.

PaulRBerg · 2023-01-24T20:42:02Z

@lucas-manuel, quick terminology question.

In August 2022, you referred in this comment to the target contracts that encapsulate the calls to the real contracts that are to be invariant tested as "actors".

However, in this PR, it looks like you have changed the name to "handler" - am I correct that you now prefer this terminology over "actor"?

lucas-manuel · 2023-01-25T16:39:25Z

@lucas-manuel, quick terminology question.

In August 2022, you referred in this comment to the target contracts that encapsulate the calls to the real contracts that are to be invariant tested as "actors".

However, in this PR, it looks like you have changed the name to "handler" - am I correct that you now prefer this terminology over "actor"?

Thats right - we're moving to handlers since the term "actor" was used when we had to use one contract per user, before pranking was possible. Now that multiple actors can be used within a give contract, we determined that handler should be used instead.

PaulRBerg · 2023-01-25T18:03:12Z

Thanks for confirming, @lucas-manuel.

Speaking of pranking multiple actors - the useRandomLp modifier used in maple-core-v2 is something that all invariant users may need. We should consider renaming it to something more general-purpose (e.g. useRandomUser or useRandomActor) and ship it with the InvariantTest contract part of Forge Std.

PaulRBerg · 2023-01-25T19:35:07Z

Potentially dumb question - can vm.assume be used in invariant tests?

I see no reference to this cheatcode in Lucas' examples, but I'm not sure if this is because Maple uses a different constraining system (constrictToRange), or because fuzzing assumptions are not meant to be used in an invariant context. And if not, why? It might be worth it to explain this in the docs.

If I understand this correctly, under "Bounded Testing", the user is expected to bound the inputs to avoid reverts.

src/forge/invariant-testing.md

lucas-manuel · 2023-01-26T03:45:04Z

Gonna work to push this through now, what is missing? @mds1 @gakonst @PaulRBerg @pcaversaccio

lucas-manuel · 2023-01-26T03:45:58Z

Potentially dumb question - can vm.assume be used in invariant tests?

I see no reference to this cheatcode in Lucas' examples, but I'm not sure if this is because Maple uses a different constraining system (constrictToRange), or because fuzzing assumptions are not meant to be used in an invariant context. And if not, why? It might be worth it to explain this in the docs.

If I understand this correctly, under "Bounded Testing", the user is expected to bound the inputs to avoid reverts.

Correct me if I'm wrong @mds1 but I believe vm.assume would work in the same way as in fuzz tests for this. Haven't used it personally though.

lucas-manuel · 2023-01-26T03:46:39Z

Thanks for confirming, @lucas-manuel.

Speaking of pranking multiple actors - the useRandomLp modifier used in maple-core-v2 is something that all invariant users may need. We should consider renaming it to something more general-purpose (e.g. useRandomUser or useRandomActor) and ship it with the InvariantTest contract part of Forge Std.

This is an interesting idea, thoughts @mds1 ?

pcaversaccio · 2023-01-26T07:42:25Z

Gonna work to push this through now, what is missing? @mds1 @gakonst @PaulRBerg @pcaversaccio

Proper descriptions on all the invariant helper functions like targetSender, targetArtifact etc. would be very nice.

PaulRBerg · 2023-01-26T10:44:53Z

I believe vm.assume would work in the same way as in fuzz tests for this

It doesn't. I'm getting Failed Assertion errors if I have vm.assume statements in the invariant tests. If this is not intended behavior, should I open an issue in forge-std?

Proper descriptions on all the invariant helper functions

Yes! A markdown table for this would be awesome.

Gonna work to push this through now, what is missing?

The greatest conceptual leap for me were the handler contracts - I spent a few days looking at the maple-core-v2 repository, and the LoanHandler and the LpHandler contracts were the most difficult to understand.

I see that you added more explanations and a few diagrams about them in the meantime (6 hrs ago), which are awesome. But I think that there are still a few other core concepts that would be good to document, too:

How should actors be generated? This may depend upon the business logic of the smart contract under test, but we should be able to come up with a few rules of thumb, e.g. generate the actors in the constructor and store them in an array.
How should actors and other contract storage items be fuzzed? In maple-core-v2, you are using these seed arguments (warpSeed_, borrowerIndexSeed_), and it took me a while to understand their role. It would also be good to mention the need for a useRandomActor modifier - at least until Foundry offers a built-in solution for this.
Should handler contracts inherit from Test? Or Utils? If not, how can they get access to vm, bound, and so forth?

mds1 · 2023-01-26T14:48:08Z

It doesn't. I'm getting Failed Assertion errors if I have vm.assume statements in the invariant tests. If this is not intended behavior, should I open an issue in forge-std?

@PaulRBerg Nice catch, just created foundry-rs/foundry#4190 to track

Speaking of pranking multiple actors - the useRandomLp modifier used in maple-core-v2 is something that all invariant users may need. We should consider renaming it to something more general-purpose (e.g. useRandomUser or useRandomActor) and ship it with the InvariantTest contract part of Forge Std.

This is an interesting idea, thoughts @mds1 ?

This is an interesting idea. More generally maybe we can add a template InvariantHandler contract to forge-std. @lucas-manuel @PaulRBerg if either of you have ideas on what this could look like, create an issue in forge-std and we can continue the conversation there.

Gonna work to push this through now, what is missing? @mds1 @gakonst @PaulRBerg @pcaversaccio

Awesome, thanks! Later today I'll review what's currently here and reply with some thoughts

PaulRBerg · 2023-01-26T19:00:07Z

More generally maybe we can add a template InvariantHandler contract to forge-std.

I like this idea. Definitely one or two helpers for actor generation and management would be nice. I may open an issue for this later.

gakonst · 2023-02-05T03:37:19Z

@lucas-manuel I think one final pass based on the above comments and we're good to merge as a first iteration

pcaversaccio · 2023-02-06T01:30:01Z

Haven't checked again the content of this PR but apparently you thought it would be a good idea to rename InvariantTest to StdInvariant (foundry-rs/forge-std@b4f1215). Broke all of my invariant tests in the CI lol - in any case if we somewhere use the old term InvariantTest this should be updated.

PaulRBerg · 2023-02-06T07:21:06Z

@pcaversaccio you should set the branch to v1 in your .gitmodules (foundry-rs/forge-std#275).

Nonetheless, I agree with you that all references to InvariantTest should be changd to StdInvariant.

pcaversaccio · 2023-02-06T08:37:14Z

@PaulRBerg yeah agreed - however in my particular case here I didn't want to use the v1 branch since the development iterations where important to me (i.e. invariant testing) and therefore I always pulled directly the master branch every two days or so. That's why I immediately realised that something is broken. But generally the pinning or release-branch strategy you're suggesting is also something I usually follow.

pcaversaccio · 2023-02-06T09:24:13Z

Also, I quickly wanted to emphasise that people should know that Test inherits from it by default now. Because if you do the following MyContract is Test, StdInvariant {...} it will not compile due to the linearisation pattern by Solidity. However, folks will read the release of forge-std and will get confused:

lucas-manuel · 2023-02-06T16:22:57Z

Speaking of useful modifiers, something like this may also be worth documenting:
mapping(string => uint256) public calls;
uint256 public totalCalls;

modifier instrument(string memory func) {
    calls[func]++;
    totalCalls++;
    _;
}
This can be applied to handler contract functions like this:
function myFunc() external instrument("myFunc") {
    // ...
}
The purpose being to get a call summary (e.g. here).

However, I am not sure whether adding a note about the above is worth it given the intention to add native support for instrumentation in Foundry (foundry-rs/foundry#3607).

I like this idea but yeah I think we should push for the native support, thoughts @mds1 @gakonst ?

mds1 · 2023-02-06T16:26:49Z

However, folks will read the release of forge-std and will get confused

The screenshot is correct when using v1.3.0 of forge-std, when v1.4.0 is released the release notes will document the breaking change and new usage info.

I agree with updating the docs to reflect this upcoming usage / naming though, cc @lucas-manuel

I like this idea but yeah I think we should push for the native support, thoughts @mds1 @gakonst ?

Agreed native support is preferable. There may be higher priority fixes/features though so I do think a template handler contract / modifiers in forge-std would be useful in the meantime. I also like this pattern that I recently came across from @horsefacts, and people can then implement their own getters to parse the calldata

lucas-manuel · 2023-02-06T16:28:04Z

However, folks will read the release of forge-std and will get confused

The screenshot is correct when using v1.3.0 of forge-std, when v1.4.0 is released the release notes will document the breaking change and new usage info.

I agree with updating the docs to reflect this upcoming usage / naming though, cc @lucas-manuel

I like this idea but yeah I think we should push for the native support, thoughts @mds1 @gakonst ?

Agreed native support is preferable. There may be higher priority fixes/features though so I do think a template handler contract / modifiers in forge-std would be useful in the meantime. I also like this pattern that I recently came across from @horsefacts, and people can then implement their own getters to parse the calldata

Should I add these to the docs?

gakonst · 2023-02-06T16:32:25Z

I've talked about Native support here: foundry-rs/foundry#3607 (comment). We could but I think is for separate discussion.

horsefact's pattern is still not too common so I'd avoid including it

anything else we need to get this over the line?

lucas-manuel · 2023-02-06T16:55:56Z

Just adding the target definitions and will push for final review.

lucas-manuel · 2023-02-06T17:29:04Z

Alright good for review 👍

horsefacts · 2023-02-06T17:32:07Z

horsefact's pattern is still not too common so I'd avoid including it

Agree, this is more an exploration than something I'd recommend right now.

mds1 · 2023-02-06T17:57:09Z

src/forge/invariant-testing.md

-There are different ways to assert invariants:
+Examples of invariants are:
+- *"The xy=k formula always holds"* for Uniswap
+- *"The sum of all owner balances is equal to the total supply"* for an ERC-20 token.


Nit, since "owner" sounds like a privileged actor

Suggested change

- *"The sum of all owner balances is equal to the total supply"* for an ERC-20 token.

- *"The sum of all user balances is equal to the total supply"* for an ERC-20 token.

mds1 · 2023-02-06T18:06:09Z

One comment here we should fix, but otherwise this looks really great! Awesome job @lucas-manuel 🔥

lucas-manuel · 2023-02-06T18:27:00Z

One comment here we should fix, but otherwise this looks really great! Awesome job @lucas-manuel 🔥

Updated! 👍

mds1 · 2023-02-08T21:57:37Z

@gakonst @lucas-manuel anything blocking merge here?

think we might just need info around how targetContracts can change since it automatically adds contracts deployed during an invariant run, cc @joshieDo for details

lucas-manuel · 2023-02-09T01:45:16Z

@gakonst @lucas-manuel anything blocking merge here?

think we might just need info around how targetContracts can change since it automatically adds contracts deployed during an invariant run, cc @joshieDo for details

Good to go on my end

PaulRBerg

Just had a final read, looks lit. Thanks again @lucas-manuel for writing this up. There are only a few minor typos - besides these, LGTM.

I'm glad to see the useActor modifier included in this PR 🙌

PaulRBerg · 2023-02-09T10:36:58Z

src/forge/invariant-testing.md

+| `excludeSender(address newExcludedSender_)` | Adds a given address to the to the `_excludedSenders` array. This set of addresses is explictly excluded from the target senders. |
+| `excludeArtifact(string memory newExcludedArtifact_)` | Adds a given string to the to the `_excludedArtifacts` array. This set of strings is explictly excluded from the target artifacts. |
+| `targetArtifact(string memory newTargetedArtifact_)` | Adds a given string to the to the `_targetedArtifacts` array. This set of strings is used for the target artifacts.  |


Double to the to the typo.

PaulRBerg · 2023-02-09T10:37:31Z

src/forge/invariant-testing.md

+### Target Contract Setup
+
+Target contracts can be set up using the following three methods:
+1. Contracts that are manually added to the the `targetContracts` array are added to the set of target contracts.


Double the the typo.

pcaversaccio

Fixed typo: explictly ==> explicitly & I think : makes more sense where I suggested it. Anything else LGTM.

pcaversaccio · 2023-02-09T10:42:38Z

src/forge/invariant-testing.md

+
+Invariant testing is a powerful tool to expose incorrect logic in protocols. Due to the fact that function call sequences are randomized and have fuzzed inputs, invariant testing can expose false assumptions and incorrect logic in edge cases and highly complex protocol states.
+
+Invariant testing campaigns have two dimensions, `runs` and `depth`.


Suggested change

Invariant testing campaigns have two dimensions, `runs` and `depth`.

Invariant testing campaigns have two dimensions, `runs` and `depth`:

pcaversaccio · 2023-02-09T10:47:40Z

src/forge/invariant-testing.md

+| Function | Description |
+|-|-|
+| `excludeContract(address newExcludedContract_)` | Adds a given address to the `_excludedContracts` array. This set of contracts is explicitly excluded from the target contracts.|
+| `excludeSender(address newExcludedSender_)` | Adds a given address to the to the `_excludedSenders` array. This set of addresses is explictly excluded from the target senders. |


Suggested change

| `excludeSender(address newExcludedSender_)` | Adds a given address to the to the `_excludedSenders` array. This set of addresses is explictly excluded from the target senders. |

| `excludeSender(address newExcludedSender_)` | Adds a given address to the to the `_excludedSenders` array. This set of addresses is explicitly excluded from the target senders. |

pcaversaccio · 2023-02-09T10:48:00Z

src/forge/invariant-testing.md

+|-|-|
+| `excludeContract(address newExcludedContract_)` | Adds a given address to the `_excludedContracts` array. This set of contracts is explicitly excluded from the target contracts.|
+| `excludeSender(address newExcludedSender_)` | Adds a given address to the to the `_excludedSenders` array. This set of addresses is explictly excluded from the target senders. |
+| `excludeArtifact(string memory newExcludedArtifact_)` | Adds a given string to the to the `_excludedArtifacts` array. This set of strings is explictly excluded from the target artifacts. |


Suggested change

| `excludeArtifact(string memory newExcludedArtifact_)` | Adds a given string to the to the `_excludedArtifacts` array. This set of strings is explictly excluded from the target artifacts. |

| `excludeArtifact(string memory newExcludedArtifact_)` | Adds a given string to the to the `_excludedArtifacts` array. This set of strings is explicitly excluded from the target artifacts. |

lucas-manuel · 2023-02-09T16:43:01Z

Made all requested changes thanks @PaulRBerg @pcaversaccio

gakonst · 2023-02-10T20:10:26Z

let's send it

lucas-manuel added 2 commits January 6, 2023 10:59

feat: add table

61f8b6d

fix: update table formatting

5d5f9ad

gakonst reviewed Jan 6, 2023

View reviewed changes

.vscode/settings.json Outdated

@@ -0,0 +1,3 @@

{

Copy link

Member

gakonst Jan 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

feat: remove vscode setting and add more info

8f5fede

mds1 reviewed Jan 6, 2023

View reviewed changes

gakonst mentioned this pull request Jan 19, 2023

Document invariant testing + patterns #497

Closed

PaulRBerg reviewed Jan 23, 2023

View reviewed changes

src/forge/invariant-testing.md Outdated Show resolved Hide resolved

PaulRBerg reviewed Jan 23, 2023

View reviewed changes

src/forge/invariant-testing.md Outdated Show resolved Hide resolved

PaulRBerg mentioned this pull request Jan 23, 2023

docs: document "invariant_" prefix lucas-manuel/invariant-examples#4

Merged

feat: update to add more example code

529d741

This was referenced Jan 25, 2023

"InvariantTest" does not have a "vm" foundry-rs/forge-std#282

Closed

Add one-line explainers for each function in InvariantTest foundry-rs/forge-std#283

Open

pcaversaccio reviewed Jan 25, 2023

View reviewed changes

src/forge/invariant-testing.md Outdated Show resolved Hide resolved

lucas-manuel and others added 2 commits January 25, 2023 21:29

fix: update based on comments

1e4f00c

feat: add diagrams and handlers section

75afb61

lucas-manuel added 2 commits January 25, 2023 21:48

fix: reorder pages

a62fc2a

fix: add section on depth counter and fix target contracts section

d520de0

PaulRBerg mentioned this pull request Jan 26, 2023

feat(forge): config option to ignore contracts deployed in setUp function foundry-rs/foundry#4187

Closed

feat: break out handlers into sections, add definitions for targets

51573e9

lucas-manuel changed the title ~~feat: Add invariant testing [WIP]~~ feat: Add invariant testing Feb 6, 2023

mds1 reviewed Feb 6, 2023

View reviewed changes

fix: owner => user

93ec6d3

fix: update fail on revert default

7d09bd0

PaulRBerg suggested changes Feb 9, 2023

View reviewed changes

pcaversaccio reviewed Feb 9, 2023

View reviewed changes

fix: update typos and add colon

c635496

0x4007 mentioned this pull request Feb 10, 2023

Invariant Testing ubiquity/ubiquity-dollar#563

Open

gakonst merged commit c388d35 into foundry-rs:master Feb 10, 2023

PaulRBerg mentioned this pull request Mar 31, 2023

feat: vm.assume support in invariant tests foundry-rs/foundry#4190

Closed

PaulRBerg mentioned this pull request May 23, 2023

Invariant tests function breakdown foundry-rs/foundry#5011

Closed

	- "The sum of all owner balances is equal to the total supply" for an ERC-20 token.
	- "The sum of all user balances is equal to the total supply" for an ERC-20 token.


		Invariant testing is a powerful tool to expose incorrect logic in protocols. Due to the fact that function call sequences are randomized and have fuzzed inputs, invariant testing can expose false assumptions and incorrect logic in edge cases and highly complex protocol states.

		Invariant testing campaigns have two dimensions, `runs` and `depth`.

	\| `excludeSender(address newExcludedSender_)` \| Adds a given address to the to the `_excludedSenders` array. This set of addresses is explictly excluded from the target senders. \|
	\| `excludeSender(address newExcludedSender_)` \| Adds a given address to the to the `_excludedSenders` array. This set of addresses is explicitly excluded from the target senders. \|

	\| `excludeArtifact(string memory newExcludedArtifact_)` \| Adds a given string to the to the `_excludedArtifacts` array. This set of strings is explictly excluded from the target artifacts. \|
	\| `excludeArtifact(string memory newExcludedArtifact_)` \| Adds a given string to the to the `_excludedArtifacts` array. This set of strings is explicitly excluded from the target artifacts. \|

feat: Add invariant testing #760

feat: Add invariant testing #760

Conversation

lucas-manuel commented Jan 6, 2023 • edited by gakonst Loading

Choose a reason for hiding this comment

mds1 left a comment

Choose a reason for hiding this comment

PaulRBerg commented Jan 23, 2023 • edited Loading

PaulRBerg commented Jan 24, 2023 • edited Loading

lucas-manuel commented Jan 25, 2023

PaulRBerg commented Jan 25, 2023 • edited Loading

PaulRBerg commented Jan 25, 2023

lucas-manuel commented Jan 26, 2023

lucas-manuel commented Jan 26, 2023

lucas-manuel commented Jan 26, 2023

pcaversaccio commented Jan 26, 2023 • edited Loading

PaulRBerg commented Jan 26, 2023 • edited Loading

mds1 commented Jan 26, 2023

PaulRBerg commented Jan 26, 2023

gakonst commented Feb 5, 2023

pcaversaccio commented Feb 6, 2023 • edited Loading

PaulRBerg commented Feb 6, 2023 • edited Loading

pcaversaccio commented Feb 6, 2023 • edited Loading

pcaversaccio commented Feb 6, 2023 • edited Loading

lucas-manuel commented Feb 6, 2023

mds1 commented Feb 6, 2023

lucas-manuel commented Feb 6, 2023

gakonst commented Feb 6, 2023 • edited Loading

lucas-manuel commented Feb 6, 2023

lucas-manuel commented Feb 6, 2023

horsefacts commented Feb 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mds1 commented Feb 6, 2023

lucas-manuel commented Feb 6, 2023

mds1 commented Feb 8, 2023

lucas-manuel commented Feb 9, 2023

PaulRBerg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcaversaccio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucas-manuel commented Feb 9, 2023

gakonst commented Feb 10, 2023

lucas-manuel commented Jan 6, 2023 •

edited by gakonst

Loading

PaulRBerg commented Jan 23, 2023 •

edited

Loading

PaulRBerg commented Jan 24, 2023 •

edited

Loading

PaulRBerg commented Jan 25, 2023 •

edited

Loading

pcaversaccio commented Jan 26, 2023 •

edited

Loading

PaulRBerg commented Jan 26, 2023 •

edited

Loading

pcaversaccio commented Feb 6, 2023 •

edited

Loading

PaulRBerg commented Feb 6, 2023 •

edited

Loading

pcaversaccio commented Feb 6, 2023 •

edited

Loading

pcaversaccio commented Feb 6, 2023 •

edited

Loading

gakonst commented Feb 6, 2023 •

edited

Loading