Skip to content
This repository has been archived by the owner. It is now read-only.

High-level interface to the state machine API #355

Merged
merged 2 commits into from Sep 14, 2019
Merged

High-level interface to the state machine API #355

merged 2 commits into from Sep 14, 2019

Conversation

@edsko
Copy link
Contributor

@edsko edsko commented Sep 8, 2019

Submitting a draft PR to let you guys know we've been working on this and made quite a bit of progress. However, I am yet to use these abstractions in my real/large QSM test setup (or indeed in the QSM tests themselves that use the patterns from my blog post), so I'm sure a bit more further tweaking will be required. However, quite pleased with how this turned out.

This is joint work with Alfredo Di Napoli.

High-level interface to the state machine API

This captures the patterns described in http://www.well-typed.com/blog/2019/01/qsm-in-depth/ as a proper Haskell abstraction. The basic idea is that we have a concept of commands and responses (just like in the lower level API), along with two interpretors, one for the system under test and one for the model. We then compare that they can run in lockstep, comparing responses at every step, insisting that they must be equal.

Of course requiring that responses must be really equal is too strong: the system under test might return a real file handle for example, where the model will return a mock file handle. So we compare responses up to a mapping from real handles to mock handles, which is maintained by the Lockstep infrastructure.

It comes in two variants: one n-ary variant supporting an arbitrary number of different types of handles, and one where there is precisely one. The n-ary version is needed if testing for example a system that might return both DB handles and file handles, but in most cases the simple one suffices and is much easier to use. The simple API is defined in terms of the n-ary one (which requires some more sophisticated types).

The IORefs test is a nice example that demonstrates quite how succinctly we can write model tests with these abstractions.

Otherwise cabal will confuse the RQLite module with the Rqlite module on
case-insensitive OS (like OSX).

Signed-off-by: Edsko de Vries <edsko@well-typed.com>
@edsko
Copy link
Contributor Author

@edsko edsko commented Sep 8, 2019

(This PR is really just one commit. The first commit is #354.)

@stevana
Copy link
Collaborator

@stevana stevana commented Sep 9, 2019

Cool, I like the two variants, simple and N-ary, and how short the example turned out!

@edsko
Copy link
Contributor Author

@edsko edsko commented Sep 10, 2019

@stevana If you like the approach, if you prefer you could merge this as is. I think what's here works, and is correct, but it might not be quite general enough, we might need some additional hooks. (In particular, I want some hooks for labelling and for test setup.) I'm not sure when I'll get to that though. Up to you, whichever way you prefer.

@stevana
Copy link
Collaborator

@stevana stevana commented Sep 10, 2019

I'm happy to merge this, but it would be good to fix a couple of things that CI is complaining about:

CI is a bit broken at the moment, I've created an issue for it -- but that's unrelated to this PR. Nevertheless we should probably fix CI before merging this... Do you need this released by the way?

@edsko
Copy link
Contributor Author

@edsko edsko commented Sep 10, 2019

Do you need this released by the way?

No, I probably can't quite use this in its currents state anyway, I'll need the generalizations. Merging would primarily be to avoid bit-rot I suppose.

@edsko
Copy link
Contributor Author

@edsko edsko commented Sep 10, 2019

The 8.2 problem is easily fixed, but not entirely sure what to do about sop-core?

@stevana
Copy link
Collaborator

@stevana stevana commented Sep 10, 2019

Should also be an easy fix, the error message in the log explains it quite well:

In the dependencies for quickcheck-state-machine-0.6.0:

    sop-core needed, but the stack configuration has no specified version 

             (latest matching version is 0.5.0.0)

needed since quickcheck-state-machine is a build target.

Some different approaches to resolving this:

  * Recommended action: try adding the following to your extra-deps

    in /home/travis/build/advancedtelematic/quickcheck-state-machine/stack.yaml:

- sop-core-0.5.0.0@sha256:8734ab38b8c84837094eec657da0b58942e481e20166131f34cf6c7fe9787b07,2827

@edsko
Copy link
Contributor Author

@edsko edsko commented Sep 10, 2019

Ah, okay, that works for you? You don't mind a dependency that's not in lts-12?

@stevana
Copy link
Collaborator

@stevana stevana commented Sep 10, 2019

Yeah, that's fine. (I think we already got other dependencies that are not part of Stackage.)

This captures the patterns described in
http://www.well-typed.com/blog/2019/01/qsm-in-depth/ as a proper Haskell
abstraction. The basic idea is that we have a concept of commands and responses
(just like in the lower level API), along with two interpretors, one for the
system under test and one for the model. We then compare that they can run in
lockstep, comparing responses at every step, insisting that they must be equal.

Of course requiring that responses must be really _equal_ is too strong: the
system under test might return a real file handle for example, where the model
will return a mock file handle. So we compare responses up to a mapping from
real handles to mock handles, which is maintained by the Lockstep
infrastructure.

It comes in two variants: one n-ary variant supporting an arbitrary number of
different types of handles, and one where there is precisely one. The n-ary
version is needed if testing for example a system that might return both DB
handles and file handles, but in most cases the simple one suffices and is much
easier to use. The simple API is defined in terms of the n-ary one (which
requires some more sophisticated types).

The IORefs test is a nice example that demonstrates quite how succinctly we can
write model tests with these abstractions.

This is joint work with Alfredo Di Napoli.

Signed-off-by: Edsko de Vries <edsko@well-typed.com>
@edsko
Copy link
Contributor Author

@edsko edsko commented Sep 12, 2019

@stevana Not sure if the remaining errors are related tot this PR..?

@stevana
Copy link
Collaborator

@stevana stevana commented Sep 12, 2019

@edsko: No, I don't think so.

@stevana stevana marked this pull request as ready for review Sep 14, 2019
@stevana stevana merged commit 41718e9 into advancedtelematic:master Sep 14, 2019
1 of 2 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants