Make tests first-class citizens

[greg-szabo]

We identified four different kinds of tests in a discussion with Thane:

1. Regular tests (usually unit tests and integration tests of different Rust components), that can run by cargo test.
2. Model-based tests that require a specific environment set up with docker and jsonattr.
3. Tendermint-Go integration tests that need a Tendermint-go endpoint set up (using docker).
4. GaiaD integration tests that need a GaiaD endpoint set up (using docker). (Note: this doesn't exist yet.)

Note: the Rust book defines integration tests as tests among different public-facing components of the Rust code. We enhance this interpretation with "testing different components of the ecosystem against the public-facing components of the Rust code".

We would like both our CI and local dev runs to have an easy method to run all these tests. Additionally, it would be nice to make sure that maintaining these environments are easy too. At last, we want to make sure that the results of these tests are easy to troubleshoot.

The current method is to run cargo test --all for the regular tests and run the ignored tests cargo test -- --ignored for the Tendermint-Go integration tests. The problem with this approach is that it can't be made more granular: the gaiad endpoint would require an --ignored2 option while we might want to exclude the model-based tests in some cases.

Someone mentioned the option of moving tests into separate config features.

The model-based tests take the approach of pinpointing a group of tests with the --tests switch.

Thane also mentioned using cargo-make as an option to unite the testing methods. Shiplift also came up as an option to manage Docker executions. Earthly.dev also came up as a way of merging Makefiles, Dockerfiles and Bash scripts into a more manageable format.

We should discuss and make a plan so we can start organizing our tests better and implement these additional features.

[greg-szabo]

I just found cargo-xtask which describes itself as a way to add free-form automation to a Rust project, a-la make, npm run or bespoke bash scripts. and does this in a cargo/rust-native way (no external dependencies).

The rust-analyzer project uses it to generate data structs and tests.

[greg-szabo]

There's been some stride to get this done. The current open PRs & issues cover most of the integrations test organizing. What's left is to actually improve the integration testing (possibly with more elaborate tests). That should be a separate issue.

The other set of items that are left is the model-based testing but Andrey and Shivani have a lot on their plate to get that done. Most probably that will also go into another issue.

Thane's idea of using cargo-make turned out to be really useful.

[greg-szabo]

I think the above changes are enough to set the stage for better testing. After merging #753, #754 and #758, we can close this issue.
#757 (gaiad fixtures) and model-based-testing will be dealt with separately, they need more research and development.