reimplement publish functionality as a plugin #2092

davidjgoss · 2022-07-23T21:24:23Z

🤔 What's changed?

This PR moves the "publish" functionality from being a pseudo-formatter to being a "plugin" - a new concept we're experimenting with (see #2091).

The plugin concept stays internal here - the idea is to get feedback on this draft API and experiment with a couple more use cases before releasing anything public-facing. This PR should be mergeable any time though.

So, the design. The structure of a plugin is basically:

It's an async function, which would be the default export for a plugin package, like with formatters now
It takes an argument which gives it some stuff to work with - configuration, environment stuff and a way to listen for events
It can listen for events - just Cucumber messages for now - and do stuff with them. Note that the hander is not async - we don't want to be wait on message consumers sending network requests etc. This would be different for things that can affect the test run e.g. reordering pickles - we'll want to allow those to be async.
It can optionally provide a cleanup function to be done at the end of the test run

I borrowed ideas from several places for this:

Cypress - the on('event', handler) style in particular, and I also like the format of lifecycle hook naming e.g. "filter:database"
ESLint - the way plugins are referenced declaratively in the configuration - i.e. no writing code to hook up plugins
React - optionally returning a "cleanup" function at the end of the initialising function is shamelessly stolen from useEffect

🏷️ What kind of change is this?

🏦 Refactoring/debt/DX (improvement to code design, tooling, documentation etc. without changing behaviour)

📋 Checklist:

I agree to respect and uphold the Cucumber Community Code of Conduct
I've changed the behaviour of the code
- I have added/updated tests to cover my changes.
My change requires a change to the documentation.
- I have updated the documentation accordingly.
Users should know about my change
- I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

This text was originally generated from a template, then edited by hand. You can modify the template here.

coveralls · 2022-07-23T21:27:51Z

Coverage decreased (-0.1%) to 98.236% when pulling 49571ef on feat/publish-as-plugin into 21a3e6c on main.

davidjgoss · 2022-07-23T21:29:30Z

features/publish.feature

@@ -106,7 +106,7 @@ Feature: Publish reports
  @spawn
  Scenario: when results are not published due to an error raised by the server, the banner is displayed
    When I run cucumber-js with env `CUCUMBER_PUBLISH_TOKEN=keyboardcat`
-    Then it fails
+    Then it passes


The only functional change here. For formatters, we catch errors on the stream(s), log them and then fail the test run at the end. This is intended to cope with things like file system snafus etc. Publish is now not a formatter, and at any rate this felt a bit off - failing to publish the report seems like it shouldn't fail the test run? That's debatable, though.

It does raise a couple of questions about error handling though:

If there's an uncaught error in a plugin, should we try to fail the test run gracefully-ish?

Should there be a way for a plugin to say "The test run should be marked as a failure because XYZ"? Or should it just throw if it wants to do that?

davidjgoss · 2022-07-23T21:31:21Z

src/api/run_cucumber.ts

@@ -25,7 +26,7 @@ export async function runCucumber(
  onMessage?: (message: Envelope) => void
 ): Promise<IRunResult> {
  const { cwd, stdout, stderr, env } = mergeEnvironment(environment)
-  const logger = new Console(stdout, stderr)
+  const logger = new Console(stderr)


This was an oversight when initially added. This logger is generally used for warn and error, but even for log (if used) it should still go to stderr - only formatters get to write stuff to stdout. Important now if we are going to make logger available to plugins.

davidjgoss · 2022-07-23T21:31:33Z

src/api/run_cucumber.ts

  const eventDataCollector = new EventDataCollector(eventBroadcaster)

  let formatterStreamError = false
-  const cleanup = await initializeFormatters({
+  const cleanupFormatters = await initializeFormatters({


Renamed for clarity.

davidjgoss · 2022-07-23T21:32:44Z

src/formatter/index.ts

-  | TtyWriteStream
-  | PassThrough
-  | HttpStream
+export type IFormatterStream = Writable


HttpStream was no longer applicable here but also this didn't feel like a useful type. We need a writable stream, is all.

davidjgoss · 2022-07-23T21:33:16Z

src/plugin/types.ts

+import { IRunEnvironment, IRunOptions } from '../api'
+import { Envelope } from '@cucumber/messages'
+
+export interface PluginEvents {


This would expand to cover other events we start to support.

mattwynne · 2022-08-08T21:24:54Z

Thoughts, from a first pass reading over this:

If we go with the rule stated above that "only formatters get to write stuff to stdout", then what happens if we embrace the idea that formatters are just a special type of plugin?
Similarly, with handling errors in the event listeners, we probably need to be consistent across plugins and formatters. For me, those event listeners should not get to impact the test run's success, but it also seems like a useful feature for errors to be reported to the user and to cause a non-zero exit status from the cucumber command itself.
I worry that the returning-a-cleanup-function thing will be non-intuitive to discover. It would feel more natural and extensible to me if there were a way to access the SupportCodeMethods API so you could simply add an AfterAll hook. What do you think about that?
Talking of which, if we plan to allow plugins to "Hook into core behaviour and change things" (as stated in the new docs page), I guess this use-case doesn't really give us any need to do that? So we'll want to look for other things we can factor into this plugin mechanism to help give it it's final shape, is that what you're thinking?

davidjgoss · 2022-08-09T07:31:48Z

Thanks for the feedback @mattwynne! To your points:

A formatter plugin would also get a write function in its context argument that would enable it to write to the relevant output stream.
Agreed, I'll do something to catch and log. I wonder if for handlers that change things (as opposed to just consuming messages) it should fail fast though?
Yeah, I get that it might not be the most intuitive, although it has some prior art. Allowing to add extra AfterAll here feels wrong to me, partly because it doesn't quite line up with what we want to be able to do (AfterAll hooks run on the worker not the coordinator for parallel, and are part of the test run, where there are more messages still to come) but also generally because I feel there should be a separation between user code and plugin code, it could get confusing to mix them. If you feel strongly about the return-a-cleanup-function thing, I could get on board with a cleanup function being part of the context argument, or maybe just on('end', async () => {}) as a supported event. WDYT?
Yep exactly. I think being able to filter and reorder pickles is an obvious candidate. I'll probably make a branch from this and see where that goes.

mattwynne · 2022-08-10T00:29:27Z

Thanks for the feedback @mattwynne! To your points:

A formatter plugin would also get a write function in its context argument that would enable it to write to the relevant output stream.

OK, let's cross that bridge when we come to it.

Agreed, I'll do something to catch and log. I wonder if for handlers that change things (as opposed to just consuming messages) it should fail fast though?

In my view, an event protocol should be "fire and forget" from our point of view, so not a good place to let people change things. I think we should find a different mechanism for that. Again, shall we cross that bridge when we come to it?

Yeah, I get that it might not be the most intuitive, although it has some prior art. Allowing to add extra AfterAll here feels wrong to me, partly because it doesn't quite line up with what we want to be able to do (AfterAll hooks run on the worker not the coordinator for parallel, and are part of the test run, where there are more messages still to come) but also generally because I feel there should be a separation between user code and plugin code, it could get confusing to mix them. If you feel strongly about the return-a-cleanup-function thing, I could get on board with a cleanup function being part of the context argument, or maybe just on('end', async () => {}) as a supported event. WDYT?

Fair enough. I hadn't thought as clearly as you about the distinction between user and plugin code, and where they would run. I still feel like there might be a need for plugins to be able to do something like what user code can do, but let's again wait until we have a use-case.

Yep exactly. I think being able to filter and reorder pickles is an obvious candidate. I'll probably make a branch from this and see where that goes.

I say we merge this in and then do that. It's good enough for now and has enough caveats on it that we can change it later.

davidjgoss · 2022-08-10T11:31:46Z

I say we merge this in and then do that.

Works for me! Will get the conflicts fixed and get it merged. Mind dropping an approval on here for completeness?

davidjgoss added 6 commits July 23, 2022 15:53

WIP

63ab911

move stray bits of config out of other areas

33058aa

rename for clarity

4477a4e

get last scenario working; handle errors

2a12666

loosen stream types

186eb16

sort out typing

7b2c398

davidjgoss commented Jul 23, 2022

View reviewed changes

davidjgoss requested review from mattwynne, charlierudolph and aurelien-reeves July 27, 2022 13:49

davidjgoss added 2 commits July 29, 2022 17:29

Merge branch 'main' into feat/publish-as-plugin

bcd8b41

add draft documentation

dc308f1

mattwynne approved these changes Aug 10, 2022

View reviewed changes

merge main

49571ef

davidjgoss merged commit 104c375 into main Aug 11, 2022

davidjgoss deleted the feat/publish-as-plugin branch August 11, 2022 20:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reimplement publish functionality as a plugin #2092

reimplement publish functionality as a plugin #2092

davidjgoss commented Jul 23, 2022

coveralls commented Jul 23, 2022 •

edited

Loading

davidjgoss Jul 23, 2022 •

edited

Loading

davidjgoss Jul 23, 2022 •

edited

Loading

davidjgoss Jul 23, 2022

davidjgoss Jul 23, 2022

davidjgoss Jul 23, 2022 •

edited

Loading

mattwynne commented Aug 8, 2022

davidjgoss commented Aug 9, 2022

mattwynne commented Aug 10, 2022 •

edited

Loading

davidjgoss commented Aug 10, 2022

reimplement publish functionality as a plugin #2092

reimplement publish functionality as a plugin #2092

Conversation

davidjgoss commented Jul 23, 2022

🤔 What's changed?

🏷️ What kind of change is this?

📋 Checklist:

coveralls commented Jul 23, 2022 • edited Loading

davidjgoss Jul 23, 2022 • edited Loading

Choose a reason for hiding this comment

davidjgoss Jul 23, 2022 • edited Loading

Choose a reason for hiding this comment

davidjgoss Jul 23, 2022

Choose a reason for hiding this comment

davidjgoss Jul 23, 2022

Choose a reason for hiding this comment

davidjgoss Jul 23, 2022 • edited Loading

Choose a reason for hiding this comment

mattwynne commented Aug 8, 2022

davidjgoss commented Aug 9, 2022

mattwynne commented Aug 10, 2022 • edited Loading

davidjgoss commented Aug 10, 2022

coveralls commented Jul 23, 2022 •

edited

Loading

davidjgoss Jul 23, 2022 •

edited

Loading

davidjgoss Jul 23, 2022 •

edited

Loading

davidjgoss Jul 23, 2022 •

edited

Loading

mattwynne commented Aug 10, 2022 •

edited

Loading