Support interaction level pending status

[bethesque]

I'm not sure if this is a good idea or not. Allow interactions to be marked as pending within the pact, meaning they get run but don't fail the overall verification status. Rather than being calculated by the Pact Broker, this pending flag would be set by the consumer team on an individual pact.

It would also allow putting interactions that can be used for a stub that don't necessarily need to be successfully verified (for better or worse...)

[mefellows]

It would also allow putting interactions that can be used for a stub that don't necessarily need to be successfully verified (for better or worse...)

Interesting! I wonder if the concept of a pact “compression” is relevant here. i.e. what if the verifier could automatically compress interactions that overlap (based on the presented schema or some other indicator). That could allow for much richer stubbing behaviour, but also efficient verifications.

[bethesque]

You typed this before seeing my spiel on the PB doing the compression didn't you?

[mefellows]

I did yes, but just in case the intent above wasn't clear. It's not about compressing for speeding up provider tests (albeit that's one benefit) - it's more about making it less complex in the provider, i.e. having to support different provider/data states etc

So it needs more thinking of course to make that easy for both parties.

The goal being it could be used more generally as a mock service - e.g. in cypress e2e suites

[bernardobridge]

I actually mentioned something like this in reply to the new Hosted Mocks feature (https://pact-foundation.slack.com/archives/CLS16AVEE/p1594280878012000). I think this would complement that feature quite nicely.

In my experience it's something I've seen teams want to do when interacting with Pact, especially for the whole "frontend stubs" approach, and they've usually had to go about it in quite a round-a-bout way.

I also think it gives some flexibility to teams to decide which interactions in a contract are ready to be verified or not.

[bethesque]

I was more thinking of it being used for trunk based development, when people want to add new interactions on the consumer side that aren't actually toggled on yet.

[mefellows]

Yes I think perhaps it's a separate feature for stubbing. Perhaps having a way to mark the interaction as a "stub" (or perhaps this can be generalised into something else) would be useful.

To ensure we don't allow interactions that erode confidence (i.e. that are unique uses of the API that don't get validated by a provider), perhaps we can detect interactions that don’t overlap with existing tests. i.e. the presence of existing interactions combined creates a “schema”, if you will, and all non-validated interactions must comply with this schema

Incidentally, Pactflow could use this information to improve the hosted stubs and make them more useful also.

[bernardobridge]

Yeah I was alluding to feature toggles with my last statement too Beth, I can definitely see it being very useful there.

I just think it could also be used to help with the stubbing, and it was on my mind when I read this, I guess haha. I think Matt's suggestions are interesting!

[individual-it]

I have a use-case for marking individual interactions as "pending".

We have one consumer and one provider. The provider has two main versions. One version is the existing software and one is a complete rewrite. Both versions suppose to become API compatible one day, but are not yet. In the rewrite there are a lot of functions not implemented yet and additionally both versions have bugs.
I want to be able to write tests for the consumer to test its behaviour against what the API should be and then run those interactions against both provider versions. If an interaction is marked as "pending" it should not break the build but if it worked once on a specific provider version and starts to fail on that version it should break the build.
Ideally the developer should be notified if an interaction starts to pass.

To achieve that in other test layers we maintain a list of tests that are expected to fail in specific test environments see: https://dev.to/jankaritech/bdd-on-software-rewrite-6md

I think marking interactions as "pending" would get me nearly there with pact on-board tools

[bbarke]

This improvement would be fantastic for us.
We use the pact broker, but we don't use the webhooks to trigger builds. It is being used to simply share pacts.

Our builds are setup in a way where each pull request tests both the consumers and provider tests - the consumer pact json file(s) are shared with downstream provider projects, and the provider test for the service with the code change are pulled from the pact broker. This has successfully prevented bad pacts from entering the ecosystem, but it is a pain having to disable a consumer pact (skipping the test) until the provider is ready to implement it.

Having a way to mark a pact as 'pending' in the consumer test and having that respected in the pact file without needing the pact broker would be great!

We'd possibly have to build some automation to allow a pending pact to fail the build though if it started passing - I can see how having to manually remove a pending tag would fall though the cracks.

[mefellows]

Or... you could just using the pending pacts feature the broker already has 😉

It sounds like you're trying to solve problems the broker is designed to solve.

[uglyog]

Or... you could just using the pending pacts feature the broker already has wink

It sounds like you're trying to solve problems the broker is designed to solve.

The problem with the pending pacts feature the broker already has is it is derived by the data in the broker. This change makes it explicit. The consumer team knows what is WIP and what is not. Better for them to tell the broker that.

[mefellows]

Oh, that was me just being tongue in cheek about the other aspects of @bbarke's comments. This feature makes sense to me, despite potential for abuse.

Each to their own of course with respect to using the broker and the workflows it provides.

[TimothyJones]

The consumer team knows what is WIP and what is not. Better for them to tell the broker that.

There's a clear use case where the consumer team is like "ok we have the API client and the pact tests implemented, but we're not planning to hook it up to anything until the provider passes them". At the moment, you could kind of hack this by having separate consumers ("FooConsumer" and "FooConsumer WIP"), but it would be cool to have this support directly in the pact.

[JakeTheCorn]

I definitely would like something like this.

We use trunk based development and would like the ability to do this for each interaction needed

[mefellows]

For clarity, this now exists in the specification (meaning there is a place for it in a a pact generated contract with a specification set to 4 or higher) and it has been implemented in the core engine (https://github.com/pact-foundation/pact-reference/tree/master/rust). Languages need to implement this, and the corresponding test case for it.

Python is yet to move to the rust core. Follow along pact-foundation/pact-python#367 and related issues / pull requests for progress, as that is under active development.