QSM DB tests don't check that all tags are covered.

[jonathanknowles]

Release	Operating System	Cause
next	Windows & OSX & Linux)	Code

Context

Module Cardano.Wallet.DB.StateMachine defines set of Tag values to represent various test scenarios that are considered important or interesting. It also supplies a tag function to match Tag values to arbitrary sequences of commands.

However, the test suite doesn't actually check that all possible Tag values are covered during execution. This creates a risk that we might actually not be covering all the cases we'd like. (For example: see PR #349, which fixes a bug where CreateThreeWallets was never generated.)

Resolution Plan

• Adjust prop_sequential so that it fails if one or more tags are not covered.

PR

Number	Base
#367	develop

QA

[jonathanknowles]

Fix available: #367

[rvl]

If this issue is called a BUG, then to avoid BUGs, I would have to keep my PRs open until everything has been implemented fully and therefore free of BUGs.

[KtorZ]

@rvl Don't feel attacked with the word "bug", maybe it's too strong of a word. In the workflow, we simply open bugs for oversights or, things we caught after a task was completed or merged. This allows us to keep track of the work more thoroughly than having PR staying opened for too long.

[piotr-iohk]

lgtm 👍

[jonathanknowles]

@KtorZ @rvl

Perhaps I can explain my reasoning for adding the "bug" tag here:

1. With a traditional test suite, we expect to see failure if one or more test cases don't pass.
2. If a traditional test suite runs to completion reporting success ✅, but silently fails to run one or more test cases, then we'd probably consider it a bug, as it violates our expectations of what a test suite is supposed to do.
3. When we go from traditional testing to state machine-based testing, hand-written test cases (to test scenarios we care about) are replaced with tags and a tag matching function (again, to check that we're testing scenarios that we care about).
4. If a tag doesn't match during a given run, then it could be considered equivalent to a test case in a traditional test suite not being run.
5. If a state machine test runs to completion without matching a given tag and yet still reports success, this is equivalent to a traditional test suite running to completion without running all the tests but still reporting success.

So that is why I added the "bug" tag in this case. (Feel free to point out holes in my reasoning!)

However I can understand @rvl's argument here, that we have to stop somewhere when making a PR, or else PRs will always be in an incomplete state. The original QSM DB PR #259 was already a huge amount of work. (We can always submit further PRs to make something better.)

Therefore, I could understand if people wanted to label something like this as an "improvement".