Help with stateful testing

[alathon]

Hi there,

What if I need to be able to use the result of run() of one Command to provide state that subsequent Command(s) can use?

A very simple example would be if you had an API with two methods; one returns a random number, and another API call requires you to pass that number back to retrieve some value. I'd like to add the random number retrieved in run() to the state, so that subsequent Command objects can check for its existence and run the retrieval Command against that particular number. In this case, since its a random number being returned, I cannot 'simulate' this in the test suite; I cannot use nextState(), since it does not have access to the result of run(). Is this on purpose, and if so am I just thinking about this all wrong?

The bigger picture is that running sequences of commands where a Command later in a sequence depends on the results of previous commands is something I'm having trouble understanding how to do; at least when the result of running a Command isn't something I can simulate, because I simply don't know whats going to be returned to me.

[rickynils]

@alathon This has been discussed before, and it actually was possible to do this in the previous implementation of Commands. I'm now convinced that I should try to re-add this in the current implementation, to be able to support use cases like the one you've described.

[marcsaegesser]

I'm running into this same problem trying to test a database that has an auto increment field. The new value of the auto increment field is known inside run() but not inside nextState() so I can't update my model.

I'd love to not use auto increment but I'm writing tests for an existing product and this isn't something I can change.

[alathon]

@marcsaegesser - I've forked ScalaCheck and implemented the needed changes myself. However, I recommend against using my fork, unless you very specifically only do sequential tests (i.e. don't make use of ScalaCheck's ability to test for deadlocks and race conditions, by doing parallel testing with multiple threads/workers).

There are two immediate issues with my fork (beyond me updating it a lot as it is part of my thesis work):

1. It is a breaking change, as the signature of run() and nextState() changes to incorporate a new argument.
2. You cannot use my fork for parallel testing. It does not properly implement the change. Parallel testing in ScalaCheck works by computing a list of end states first, and I believe I will need to fundamentally change the way parallel testing is done so that nextState() can be passed a dynamic term. I will be looking into this, but I cannot give an ETA, as it depends on when I will need it for my masters thesis.

@rickynils Is there any sense in me spending some time to produce a pull request for a 'clean' version of my changes to ScalaCheck? What it allows is the representation of something that will hold values in the future, so that you can model things which you cannot predict what will be, because they come from an external system, or are random by nature. For example, see this pastie for a working implementation of the process spawn example from QuickCheck: For Fun and profit, which is not possible in the current version of ScalaCheck, because you cannot predict/model process ID's.

[rickynils]

@alathon A PR is always appreciated :) With a 'clean' version do you mean something that would work with the parallel testing, or a cleaned up version of your existing implementation?

My goal is to get this into ScalaCheck (as we've had it before). It might turn out to work badly together with the parallel testing. If so, we should maybe see if the parallel stuff can be moved to a separate module, implemented on top of the rest.

[alathon]

@rickynils I mean a cleaned up version of my existing implementation. In other words: Should I do a PR with my existing, sequentially working version, and then we can iterate on it to make it work for parallel testing? Or is it better to perhaps raise an Issue on my fork, and we discuss/consider it there?

I agree wrt. the parallel stuff, if it can be. I kind of also wish that ScalaCheck had a cleaner separation between the stateful testing and the Props/Test module, but I'm not sure if this issue is the right place to talk about it, so I'll open a separate Issue for that.

[BardurArantsson]

Just to add a "me too": I'm a bit more fortunate in that I basically "just" need access to State in run(). One workaround I've found is to change your Sut to wrap the thing you're really testing and add methods/vars which will allow you to add code to your run() methods to manually track what you need for subsequent commands. This workaround is somewhat limited, though -- and tedious/error-prone since you're essentially updating your model twice, once in nextState() and once in run.

Would adding the State as an argument to run() be any simpler/easier than what's being proposed here... and would it be sufficient for people's needs?

[alathon]

I'll try and cobble together a PR for this during the weekend, so it can be glossed over. It changes the signature of run() and nextState(), but should not change whether existing code works, once you've added those extra arguments to your methods.

[PipocaQuemada]

What's the status of this issue, out of curiosity?

[alathon]

I never got around to this, as my thesis ended and I got distracted. @rickynils do you want a dump of what I wrote? I probably won't have time to rebase against current and shape it into a sensible PR, but I could send the suggested changes your way?

[jpgneves]

I just ran into this as well - wanting to build commands that depends on the output of previous commands.

My use case is probably something very typical. I have certain commands that can only be executed after an Authentication command and for which I need to pass the access tokens I received from the Authentication command.

I could probably work around this in very ugly and fragile ways, but I would like my model to ensure that actions that wouldn't require authentication still work before and after I do authenticate.

I'm not very familiar with the ScalaCheck codebase, so I don't have much of an idea where to start, unfortunately.

[alathon]

@jpgneves Note that my fork of ScalaCheck is something like 200 commits behind master, as I haven't touched this since September 2016. And so I wouldn't (necessarily) recommend using it. I don't have an opportunity to work in Scala at work, so I'm not likely to return to this anytime soon.

You're welcome to check it out and see if it fits what you need, there are a number of 'symbolic' test files under test/scala/org/scalacheck/commands/symbolic that showcase what you can do (stateful command generation and in-place resolution of those references at runtime). F.ex: https://github.com/alathon/scalacheck/blob/13bd1a6d7efc9e274fefd381c9161b872a083550/jvm/src/test/scala/org/scalacheck/commands/symbolic/MultiPidRegistrationSpecification2.scala shows how you can generate commands that list/spawn/get/register/unregister processes.

I was never really happy with the result, it should've probably been implemented by making a monad wrapper around commands and that would've made the API much nicer. But alas, thesis time pressure :) You can find the fork at https://github.com/alathon/scalacheck, the branch you want to look at it is called 't2'

[jpgneves]

Thanks @alathon , this is a great starting point regardless. I'll take a shot at rebasing and refining and see where that leads me! :)

[satyagraha]

+1 for something along these lines; AFAICS effectively the behaviour of run and nextState needs to be coalesced in some way.

[dmurvihill]

It sounds like we are all waiting for an executive decision about what to do between this feature request and parallel testing. I have a hard requirement that my SUT should never repeat the same output twice and while the use case is not currently parallel it easily could be in the future. So I'm keenly interested in getting unblocked here and potentially could help with implementing whatever is decided.