Event protocol #69

Merged
merged 12 commits into from Sep 9, 2016

Projects

None yet

5 participants

@aslakhellesoy
Member

This is a fairly large commit that marks the start of a specification for the Cucumber event protocol.

This is relevant for:

  • Cucumber implementations
  • Reusable formatters (HTML, pretty etc)
  • Gherkin-Lint (it would emit events just like Cucumber, to be picked up by formatters)

Have a look and let me know what you think. I'm happy to discuss it over in the contributors room on gitter.

/cc @cucumber/developers
/cc Gherkin-Lint team: @r-cochran @enkessler @vsiakka @allustin

@brasmusson brasmusson and 4 others commented on an outdated diff Sep 8, 2016
event-protocol/README.md
+As the event specification evolves to support a richer set of events there are some
+key principles to consider:
+
+#### File format agnostic
+
+While Gherkin is currently the only file format that will be used in `source`
+events, no events should assume that all files will be Gherkin documents. This
+allows for alternative document formats in the future.
+
+#### No Cucumber execution semantics
+
+Some consumers (such as the [HTML Formatter](../html-formatter/README.md)) may
+consume events emitted by other consumers than Cucumber (for example [Gherkin-Lint](../gherkin-lint/README.md)).
+
+For this reason, events containing information about execution should be represented
+in a more generic way.
@brasmusson
brasmusson Sep 8, 2016 Contributor

I'm not sure about this. Execution is about TestCases and TestSteps, Linting Gherkin is about an AST, there are two transformations between them (AST -> Pickles -> TestCases). Trying to have the execution and the linting to emit the same type of events does not sound right to me.

@enkessler
enkessler Sep 8, 2016

This is true. Any linter will necessarily have to interact at the AST level because a rule like 'you forget to add any steps to this placeholder scenario' will not be doable because Pickles would not be generated for such a scenario.

@aslakhellesoy
aslakhellesoy Sep 8, 2016 Member

In order for this protocol to be useful it should be derived from the needs of the consumers. This is a typical consumer driven contract case.

I've been working in parallel with a HTML Formatter which contains a React/Redux component for rendering a Gherkin AST in a W3C DOM. The React views use a JavaScript data model (a redux state), based on the Gherkin AST.

This state is built by Redux reducers, which are simply pure functions that receive a state and a Cucumber Event, then returns a new state.

The state is an Immutable.Map where keys are paths/uris and values are Gherkin ASTs. Runtime information such as scenario and step pass/fail, stack traces and screenshots is captured in attachment events that have a media type and a data blob. This is a very generic data structure that could in principle be used to represent more cucumber-specific events such as “a stepdef was matched” or “a scenario started”. We’d just need to define custom media types for these events. In one way it’s an event protocol that allows other event protocols to be embedded inside it.

In brief - let's see. When a consumer needs some information that would be clumsy to represent as attachment events, we can discuss whether it makes sense to add a more specific type.

@brasmusson
brasmusson Sep 9, 2016 Contributor

The consumer in this case is only a subset of the formatters, the feature file oriented formatters - such as the pretty formatter and the html formatters, most formatter are not interested in the source they are rather interested in the result of test steps. The optimize The Event Protocol for only one type of consumer makes its usefulness rather limited, I do not see how a event protocol for feature file oriented formatters can be useful to implement parallel and distributed execution (which is stated as one of its uses).

@aslakhellesoy
aslakhellesoy Sep 9, 2016 edited Member

I think the best way to design a good event protocol is to implement several consumers in parallel. The HTML Formatter is being actively developed. Let's make the pretty formatter actively developed too.

What if I move cucumber/cucumber-pretty-formatter to the cucumber/cucumber repo, under a pretty-formatter directory?

Then that development can more easily use the same validation utilities in event-protocol, and it's easier to send pull requests that modify a consumer and the protocol spec in the same commit.

/cc @l3pp4rd

@mattwynne
mattwynne Sep 9, 2016 Member

+1 @aslakhellesoy I think this is a good example of where a monorepo will make things easier.

FWIW I think console-formatter might be a better name since we're planning on using the same binary to take in events and spit out different formatted console outputs. We decided (after realising the pretty formatter was too big a first step) to start with the progress formatter, for example. I could see this also doing JUnit and other common console outputs.

@l3pp4rd
l3pp4rd Sep 9, 2016 Member

We can move it @aslakhellesoy I do not see why not. I'm also not worried about diverging the events for a while and let the patterns express themselves while implementing different types of formats.

I also agree that gherkin AST should be managed by event consumer if needed. Sending source event makes it possible. I doubt that we could ever build pretty formatter only with pickles without having a gherkin AST tree in order to determine whether it was a scenario outline example or a scenario.

Though, I've made some progress with the progress formatter based on the event stream. We will align the event specification after we are capable of implementing HTML and pretty-format.

2016-09-06-210017

@l3pp4rd
l3pp4rd Sep 9, 2016 Member

Concerning the series field. I think this is not necessary. Because when cucumber is used, whether it is a ruby or java implementation, first of all it is a console tool, so running cucumber -f events | generic-formatter is always an unique usage.

If for example you have generic-formatter -server -p 8080 running in the background, it could create a unix socket file, through which all events could be multiplexed to this server with a specific series so that the stream could be identified and the server could publish all events back to js open websockets. In that case, js could have the gherkin AST in order to instantly reflect on HTML. Not sure yet how @aslakhellesoy implemented it, but guess should be similar.

My point is that you will always pass a stream of events through a single connection. because there is http2 and there are websockets a single run - a single connection - an unique series. It could pass user name from gitconfig together with series when initializing this connection. I do not think this should be in the event specification.

@aslakhellesoy
aslakhellesoy Sep 10, 2016 Member

Consider the scenario where a long-running server is receiving events over a TCP socket. Each of the connections to that socket would transport a unique series of events - no problem.

But these events could then be multiplexed into a single stream of events - for example to a persistent EventSource or WebSocket connection in a browser. This connection is open as long as the browser is open - independently of when Cucumber runs.

How would the browser tell the events apart if they don't have a series field or similar @l3pp4rd ?

@l3pp4rd
l3pp4rd Sep 11, 2016 edited Member

When you open a connection to this TCP socket and start to produce the event stream then you can identify all events coming from this connection with a unique series, username and more. And then they are multiplexed they can have this unique identifier with them. Like:

{
  "user": "john",
  "series": "df1d3970-644e-11e6-8b77-86f30ca893d3",
  "event": {
    "type": "start",
    "timestamp": 1471420027068
  }
}

Events can be wrapped into more context. Because their environment is different. I think composition here is a better answer. Running a console formatter is always an unique stream of events and there is no point in having the series. But when you start to multiplex these streams to a long running daemon, only then it becomes necessary to have more context, if it is a cloud where users could track their test run history or a local server, these are different and have more context compared to the console formatter.

aslakhellesoy and others added some commits Sep 6, 2016
@aslakhellesoy aslakhellesoy [html-formatter] Clean up test state and react code d675ab5
@aslakhellesoy aslakhellesoy The start of a spec for the event protocol
* Uses JSON Schema to validate and document events
* Defines three events - start, source and attachment
* The HTML formatter uses these events
27b7ac1
@aslakhellesoy aslakhellesoy s/consumer/producer/ 7c8d654
@aslakhellesoy aslakhellesoy [html-formatter] More consistent React prop types 4fc8e4c
@aslakhellesoy aslakhellesoy [html-formatter] Move attachment to line 3 818c854
@aslakhellesoy Steve Tooke Clarify TCP socket 010625d
@aslakhellesoy Steve Tooke Remove bundle d716bba
@aslakhellesoy aslakhellesoy Add attachment event for stack trace 7cacf17
@aslakhellesoy aslakhellesoy [event-protocol] Update links to event examples cffedeb
@aslakhellesoy aslakhellesoy Change Gherkin media type to text/vnd.cucumber.gherkin+plain 0f68999
@aslakhellesoy aslakhellesoy [event-protocol] media encoding and type required. No external attach…
…ments.
8bba06c
@aslakhellesoy aslakhellesoy [event-protocol] Update contributing docs
This is based on feedback from @brasmusson and @enkessler in
#69
1b69c23
@aslakhellesoy aslakhellesoy merged commit 1b69c23 into master Sep 9, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment