Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
add specs for TestRunStarted message #613
gasparnagy left a comment
I think it is very hard to see how we want to progress with this because the TestRunStarted event is too simple. I suggest adding at least one other (preferably one with a payload) so that we can better judge.
What I see already:
In Cucumber JVM the test run started event has a payload, a timestamp to indicate when the run started. Everty other event also has a timestamp.
So we could add a ISO 8601 time stamp. The time should always be in UTC. 2019-05-18T07:18:58Z
This also raises a few questions:
I also think we are missing some principles about the channel. How can it be used? Currently only the channel can only be used by a single cucumber executor at a time. Is this intentional?
If we are to allow concurrent execution of pickles, why also not concurrent execution of executors?
Either way, adding an executor UUID to each event would allow us todistinguish the events from different executors on the same channel.
@mpkorstanje We have another branch where we add a timestamp to the TestRunStarted message. We didn't send it yet as PR, because we wanted first to get feedback on the formulation of the specs, so we don't have to redo the specs for it multiple times.
About the executor UUID: would be fine for me. Reason is, that because of how the SpecFlow+Runner is implemented, we will send multiple TestRunStarted messages (one for each Thread). With that, we could differentiate them.
After discussion here, we removed the receiving side. We wrote it because of the behaviour of the SpecFlow+Runner (multiple messages). We solve this now with a different scenario.
About the term testsuite: that's one of the reason for this small PR. To get the first feedback to the problem domain language.
I'm really glad to see us using feature files to specify these messages!
I'd like to see some concrete details about what is in the message. I would prefer to see something like #615 as the first iteration, rather than having this very SpecFlow-specific stuff in the second scenario about running in parallel.
@SabotageAndi now I'm a bit confused by your explanation. Say I have 12 pickles and I execute them in parallel on 3 threads. Will this mean that these are separate 3 test runs of 4 pickles.How would a listener to the event stream know that the original 12 pickles are all done?
This sounds like a rather significant difference. In Cucumber JVM one executor sends a single started/finished event and spins up several threads to execute the pickles in between these events. It is possible to have multiple executors running in parallel. They'd be executing different runs.
In a diagram:
@mattwynne I realize this is rather abstract but I've had a hell of time retrofitting concurrency into Cucumber JVM. If we don't get this right from the ground up it will be very painful in the future. And I guess I'm less interested in the other parts of the protocol, the idea of using events has already been proven.
I think it's great to see progress towards a shared test suite for all Cucumber implementations.
Cucumber messages are fairly low level implementation details, and I am not convinced Gherkin is the best way to express how the messages should be constructed and processed. It feels to me like some sort of approval test suite is better suited. This is the approach taken for Gherkin implementations.
We also need to distinguish between the messages themselves and the Cucumber implementations producing/consuming these messages. This
I think we need to take a step back and agree on what we want to achieve with a shared test suite before we start implementing it.