Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High-level interface to the state machine API #355

Merged
merged 2 commits into from Sep 14, 2019

Conversation

@edsko
Copy link
Contributor

edsko commented Sep 8, 2019

Submitting a draft PR to let you guys know we've been working on this and made quite a bit of progress. However, I am yet to use these abstractions in my real/large QSM test setup (or indeed in the QSM tests themselves that use the patterns from my blog post), so I'm sure a bit more further tweaking will be required. However, quite pleased with how this turned out.

This is joint work with Alfredo Di Napoli.

High-level interface to the state machine API

This captures the patterns described in http://www.well-typed.com/blog/2019/01/qsm-in-depth/ as a proper Haskell abstraction. The basic idea is that we have a concept of commands and responses (just like in the lower level API), along with two interpretors, one for the system under test and one for the model. We then compare that they can run in lockstep, comparing responses at every step, insisting that they must be equal.

Of course requiring that responses must be really equal is too strong: the system under test might return a real file handle for example, where the model will return a mock file handle. So we compare responses up to a mapping from real handles to mock handles, which is maintained by the Lockstep infrastructure.

It comes in two variants: one n-ary variant supporting an arbitrary number of different types of handles, and one where there is precisely one. The n-ary version is needed if testing for example a system that might return both DB handles and file handles, but in most cases the simple one suffices and is much easier to use. The simple API is defined in terms of the n-ary one (which requires some more sophisticated types).

The IORefs test is a nice example that demonstrates quite how succinctly we can write model tests with these abstractions.

Otherwise cabal will confuse the RQLite module with the Rqlite module on
case-insensitive OS (like OSX).

Signed-off-by: Edsko de Vries <edsko@well-typed.com>
@edsko

This comment has been minimized.

Copy link
Contributor Author

edsko commented Sep 8, 2019

(This PR is really just one commit. The first commit is #354.)

@stevana

This comment has been minimized.

Copy link
Collaborator

stevana commented Sep 9, 2019

Cool, I like the two variants, simple and N-ary, and how short the example turned out!

@edsko

This comment has been minimized.

Copy link
Contributor Author

edsko commented Sep 10, 2019

@stevana If you like the approach, if you prefer you could merge this as is. I think what's here works, and is correct, but it might not be quite general enough, we might need some additional hooks. (In particular, I want some hooks for labelling and for test setup.) I'm not sure when I'll get to that though. Up to you, whichever way you prefer.

@stevana

This comment has been minimized.

Copy link
Collaborator

stevana commented Sep 10, 2019

I'm happy to merge this, but it would be good to fix a couple of things that CI is complaining about:

CI is a bit broken at the moment, I've created an issue for it -- but that's unrelated to this PR. Nevertheless we should probably fix CI before merging this... Do you need this released by the way?

@edsko

This comment has been minimized.

Copy link
Contributor Author

edsko commented Sep 10, 2019

Do you need this released by the way?

No, I probably can't quite use this in its currents state anyway, I'll need the generalizations. Merging would primarily be to avoid bit-rot I suppose.

@edsko

This comment has been minimized.

Copy link
Contributor Author

edsko commented Sep 10, 2019

The 8.2 problem is easily fixed, but not entirely sure what to do about sop-core?

@stevana

This comment has been minimized.

Copy link
Collaborator

stevana commented Sep 10, 2019

Should also be an easy fix, the error message in the log explains it quite well:

In the dependencies for quickcheck-state-machine-0.6.0:

    sop-core needed, but the stack configuration has no specified version 

             (latest matching version is 0.5.0.0)

needed since quickcheck-state-machine is a build target.

Some different approaches to resolving this:

  * Recommended action: try adding the following to your extra-deps

    in /home/travis/build/advancedtelematic/quickcheck-state-machine/stack.yaml:

- sop-core-0.5.0.0@sha256:8734ab38b8c84837094eec657da0b58942e481e20166131f34cf6c7fe9787b07,2827
@edsko

This comment has been minimized.

Copy link
Contributor Author

edsko commented Sep 10, 2019

Ah, okay, that works for you? You don't mind a dependency that's not in lts-12?

@stevana

This comment has been minimized.

Copy link
Collaborator

stevana commented Sep 10, 2019

Yeah, that's fine. (I think we already got other dependencies that are not part of Stackage.)

@edsko edsko force-pushed the edsko:edsko/lockstep branch from 43e494c to c3ff7be Sep 10, 2019
This captures the patterns described in
http://www.well-typed.com/blog/2019/01/qsm-in-depth/ as a proper Haskell
abstraction. The basic idea is that we have a concept of commands and responses
(just like in the lower level API), along with two interpretors, one for the
system under test and one for the model. We then compare that they can run in
lockstep, comparing responses at every step, insisting that they must be equal.

Of course requiring that responses must be really _equal_ is too strong: the
system under test might return a real file handle for example, where the model
will return a mock file handle. So we compare responses up to a mapping from
real handles to mock handles, which is maintained by the Lockstep
infrastructure.

It comes in two variants: one n-ary variant supporting an arbitrary number of
different types of handles, and one where there is precisely one. The n-ary
version is needed if testing for example a system that might return both DB
handles and file handles, but in most cases the simple one suffices and is much
easier to use. The simple API is defined in terms of the n-ary one (which
requires some more sophisticated types).

The IORefs test is a nice example that demonstrates quite how succinctly we can
write model tests with these abstractions.

This is joint work with Alfredo Di Napoli.

Signed-off-by: Edsko de Vries <edsko@well-typed.com>
@edsko edsko force-pushed the edsko:edsko/lockstep branch from c3ff7be to 8dd8a89 Sep 10, 2019
@edsko

This comment has been minimized.

Copy link
Contributor Author

edsko commented Sep 12, 2019

@stevana Not sure if the remaining errors are related tot this PR..?

@stevana

This comment has been minimized.

Copy link
Collaborator

stevana commented Sep 12, 2019

@edsko: No, I don't think so.

@stevana stevana marked this pull request as ready for review Sep 14, 2019
@stevana stevana merged commit 41718e9 into advancedtelematic:master Sep 14, 2019
1 of 2 checks passed
1 of 2 checks passed
continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
DCO DCO
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.