Add test utilities that make it easier to test contract consistency over time #1082
Assignees
Comments
|
Once this is implemented, or if this is abandoned, we should document how to test contracts for consistency: |
6 tasks
github-merge-queue bot
pushed a commit
that referenced
this issue
Nov 8, 2023
…r pks (#1135) ### What Remove use of rand from generated addresses, nonces, salts, and issuer pks, replace with sequential values instead. ### Why The SDK generates random addresses, nonces, salts, and issuer pks. This randomness makes it near impossible to write any test that produces predictable and consistent total system state. The SDK doesn't need to be using random values for these things. It just needs to produce values that don't matter and that are unique. Predictable and consistent values make it easier to write tests that can be used over time to assess the consistency of a contract. This is an important attribute of tests so that folks can evaluate if their contract is consistent even with dependency upgrades, SDK upgrades, and protocol upgrades. This change also introduces the concept of a `Snapshot`, which is a superset of the existing `LedgerSnapshot`. The `LedgerSnapshot` contains everything needed to load in a ledger from a network or other source. `Snapshot` contains everything needed to rehydrate the state of an SDK testutils backed Env. Related discussion in Discord: https://discord.com/channels/897514728459468821/1171226885171191949 For #1082
github-merge-queue bot
pushed a commit
that referenced
this issue
Nov 11, 2023
### What Autosave a test snapshot file on every test exit. ### Why To encourage the over-time consistency testing of contracts. To provide a mechanism where any developer who commits the generated files can see changes in observable behavior that arises unexpectedly over time. Close #1082 ### Merging To be merged to `main` after: - [x] Adding events to the `Snapshot`. - [ ] ~Adding budget to the `Snapshot`. _(Maybe, seems less clear if should be included.)_~ - [x] Support multiple Envs in a single test writing multiple files. - [x] #1142 - [x] stellar/rs-soroban-env#1182 - [x] #1135
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It would benefit contract developers for the SDK to contain some really easy to use tools for testing contract consistency over time. By really easy to use, I mean they are automatically enabled by default, or it's possible to enable them or use the feature with a single flag.
The SDK provides a reasonably good framework for testing contracts and allows developers to write assertions on return values, emitted events, final ledger state. And it allows the developer to write these assertions and exercise multiple contracts at once, providing a complete integration test experience in the simplicity of unit tests. The framework also provides some convenience functions for saving the ledger state to file.
Developers can use the above to check that over time a contracts behavior doesn't change at all, or if it does, it is trivial to confirm if the change is intended by viewing diffs of ledger state and events.
However, for developers to get that type of consistency check, they need to write a few lines of code into ever test they write. This is a reasonably high bar.
We should investigate and find a way to have every test automatically write out to disk:
If this data is written to disk to a common folder developers can choose to commit it to VCS (e.g. git), or choose to ignore it by adding the folder to their
.gitignoreor similar file. If committed developers will naturally see changes over time when upgrading envs, upgrading sdks, upgrading dependencies (code or contract), and making changes to their own contract.Given that consistency is often paramount, and surprise inconsistency can lead to bugs and exploits, this seems worth putting some effort into even if the feature is not trivial to implement.
Related to #1095.
The text was updated successfully, but these errors were encountered: