feat(provider-states): Add beforeEach and afterEach hooks to provider verification

[mefellows]

Adds support for before/after hooks on individual interactions on the verification process.

The before hook is fired before any of the other hooks (state handlers, request filters and the actual verification) and the after hook fires after all of the others have completed.

The full lifecycle is documented here: https://github.com/pact-foundation/pact-js/pull/529/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R516

Fixes #526

[mefellows]

@TimothyJones I'll add tests for this, because there is a functional impact to the change. Wanted to get it in front of you early to see if you had thoughts about doing this or not.

[TimothyJones]

I think the need for this illustrates that pact verification is best run outside of a test framework ;)

On a serious note, I have two comments:

1. If we are going to recommend that verification be run (or can be run) by a test framework, perhaps we should (also?) expose something that makes it easy to use the native before/after test hooks (eg, an array of verification promises or similar).

2. I’m not sure this fixes the linked issue- my read of the feature request is that it is asking for a companion teardown for each state setup method (state => { setup, teardown }), But, unless I’m missing something, this is generic beforeEach/afterEach.

Generally, I like this feature and I think it’s worth including - although I don’t think I would personally use it, it supports more scenarios, and that is definitely an improvement

[mefellows]

I think the need for this illustrates that pact verification is best run outside of a test framework ;)

Haha

If we are going to recommend that verification be run (or can be run) by a test framework, perhaps we should (also?) expose something that makes it easy to use the native before/after test hooks (eg, an array of verification promises or similar).

Yes, that would be nice. Using the json output from the verifier process would allow us to hook onto the results and report differently, but we'd really need to build the hooks into the verifier process itself to achieve what you're requesting (I think).

FWIW, the Golang process does the reporting on an interaction level (because there is really only one test running in the case of Golang. Harder to do with JS because you need an interface for each of the languages).

I’m not sure this fixes the linked issue- my read of the feature request is that it is asking for a companion teardown for each state setup method (state => { setup, teardown }), But, unless I’m missing something, this is generic beforeEach/afterEach.

You're right indeed. I was thinking of the way the Golang implementation worked, not the JVM.

I think, however, for all intents are purposes the same effect can be achieved without the need for companion teardown methods (for clarify, we are referring to this JVM feature). Having separate hooks per state is doable though and we could add those if needed.

I think the key use case here, is resetting a database back to the default. Because the after hook here happens at the end (always), that's doable. I've kept it consistent with the Golang implementation.

But I guess @brendan-donegan is better placed to answer - will this solve your needs?

[brendan-donegan]

If I understand correctly, the 'after' hook is always going to be the same for each interaction? I might be able to make that work but @TimothyJones is right to say that I was thinking of the way the JVM implementation works.

[TimothyJones]

Update on this: I took a stab at implementing something like

export type Hook = () => Promise<unknown>

export type StateHandlerFunction = 
  | Hook 
  | { setup: () => Promise<unknown>; teardown: () => Promise<unknown> }

export interface StateHandler {
  [name: string]: StateHandlerFunction
}

But unfortunately the Ruby binaries don't let us have clear control over teardown. This will have to wait for V3 :(

I've updated the PR so that we have beforeEach and afterEach (just a rename of matt's code) and we'll release it shortly.

[bethesque]

I can add a teardown hook if you really want it. It's supported in the Ruby impl, just not exposed in the CLI.

[TimothyJones]

I can add a teardown hook if you really want it. It's supported in the Ruby impl, just not exposed in the CLI.

Yes please! Do you want me to open an issue?

[mefellows]

I believe a per-state hook is possible in the current implementation. The caveat would be that the proxy in verifier.ts would need to track which state it's currently handling, and only invoke the state teardown for the current state (e.g. a condition in the current spot where the afterEach hook is currently firing).

This implies that requests can't be parallelised for a given mock service - which is already the case anyway. So I think it's a fair assumption to add to the proxy layer.

[bethesque]

@TimothyJones yeah, put it in the pact-provider-verifier. Though it seems a bit pointless if we're about to kill that code anyway!

[mefellows]

@TimothyJones yeah, put it in the pact-provider-verifier. Though it seems a bit pointless if we're about to kill that code anyway!

It'll be here in 12 months, especially for heavy users that won't be able to migrate to the v3 immediately.

[TimothyJones]

I don't think we're likely to kill it, at least until we have all languages running v3 - people will still want to read/write v2 contracts.