Master issue for new 'pending pacts' feature

[bethesque]

Audience

This issue is for Pact library maintainers, to document and discuss the adoption of the new 'pacts for verification' endpoint.

Background

This piece of work requires that the new 'pacts for verification' API is being used.

For background on the feature, see http://blog.pact.io/p/d7c9601f-e2ca-4208-b1c0-f02eaf96c3c2/

Summary

When verifying a pact where pending is true, a verification failure should not make the overall build fail. I understand that this might be difficult, but it's going to make a lot of Pact users very happy!

What changes need to be made to the code that uses this new API?

For implementations that wrap the ruby standalone

1. Allow user to enable/disable feature

First, upgrade to the latest version of the pact-ruby-standalone.

The user will need to be able to configure whether or not this feature is enabled. For consistency, can we present this option as enablePending or enable_pending, depending on the language naming conventions (open to suggestions on this). For backwards compatibility, this should default to false. When the value of enablePending is true, then the flag --enable-pending should be used when calling the pact-provider-verifier.

That's all folks. You can have a play around with an example CLI here https://github.com/pact-foundation/pact-provider-verifier/blob/master/script/dev/broker.sh

For native implementations

1. Allow user to enable/disable feature

The user will need to be able to configure whether or not this feature is enabled. For consistency, can we present this option as enablePending or enable_pending, depending on the language naming conventions (open to suggestions on this). For backwards compatibility, this should default to false. The value of this setting will be sent to the "pacts for verification" endpoint as the includePendingStatus field, discussed below.

2. Add new parameters to the "pacts for verification" call

They are:

1. whether or not to include the "pending" information in the response
2. any tags that are going to be applied to the provider version when the verification results are published

See API documentation below.

The provider version tags need to be sent because the calculation for whether a pact is pending or not depends on the provider version tags (if tags are being used). Eg. a pact might be pending on master but not pending on feat-x branch. If the user does not actually use provider tags, then the pending status is calculated based on the entire verification history. It is slightly less useful in this case, but still an improvement on the previous "BREAK EVERYTHING" situation.

Note that this query does not actually apply the tags to the provider version - you're just specifying which tags will be applied. If your library does not yet implement provider version tagging, then please do this as well!

3. Allow verifications to fail without causing the overall test suite to fail

The main change is, of course, that if a pact is in pending status, its verification should run without causing the overall test suite to fail. I appreciate this might be difficult. Please discuss below if you run into difficulties.

4. Print before and after verification notices

A set of notices will be included in the response that will explain to the user how the pending feature works, and what the pending state is of each pact, before and after the test run. The "before_verification" notices should be displayed before the tests run. In regards to the "after_verification" notices, the verification endpoint cannot know before the tests have run whether or not the results will pass or fail, or be published or not, so a full set of notices is generated, and the pact client framework needs to select which one to display. I've made it so that you can generate the key using the booleans published and success to generate the key after_verification:success_${success}_published_${published}.

API documentation

URL

As specified in #307

Parameters

includePendingStatus: true|false (default false). When true, a pending boolean will be added to the verificationProperties in the response, and an extra message will appear in the notices array to indicate why this pact is/is not in pending state. This will allow your code to handle the response based on only what is present in the response, and not have to do ifs based on the user's options together with the response. As requested in the "pacts for verification" issue, please print out these messages in the tests if possible. If not possible, perhaps create a separate task which will list the pact URLs and messages for debugging purposes.

providerVersionTags: array of tag name(s) for the provider application version that will be published with the verification results. This is used by the Broker to determine whether or not a particular pact is in pending state or not.

Example: This data structure represents the way a user might specify "I want to verify the latest master pacts; when I publish the verification results, I'm going to tag the provider version with 'master'"; and I don't want my build to fail if the contract has changed.

{
  "consumerVersionSelectors": [{ latest: true, tag: "master" }],
  "providerVersionTags": ["master"],
  "includePendingStatus": true
}

Response body

{
  "_embedded": {
    "pacts": [
      {
        "verificationProperties": {
          "pending": true,
          "notices": [
            {
              "when": "before_verification",
              "text": "The pact at https://test.pact-dev.dius.com.au/pacts/provider/Bar/consumer/Foo/pact-version/dd222221d7d3d915ec6315ca3ebbd76831aab6a3 is being verified because it matches the following configured selection criterion: latest pact between a consumer and Bar"
            },
            {
              "when": "before_verification",
              "text": "This pact is in pending state for this version of Bar because a successful verification result for Bar has not yet been published. If this verification fails, it will not cause the overall build to fail. Read more at https://pact.io/pending"
            },
            {
              "when": "after_verification:success_true_published_false",
              "text": "This pact is still in pending state for Bar as the successful verification results have not yet been published."
            },
            {
              "when": "after_verification:success_false_published_false",
              "text": "This pact is still in pending state for Bar as a successful verification result has not yet been published"
            },
            {
              "when": "after_verification:success_true_published_true",
              "text": "This pact is no longer in pending state for Bar, as a successful verification result has been published. If a verification for a version with  fails in the future, it will fail the build. Read more at https://pact.io/pending"
            },
            {
              "when": "after_verification:success_false_published_true",
              "text": "This pact is still in pending state for Bar as a successful verification result has not yet been published"
            }
          ],
        },
        "_links": {
          "self": {
            "href": "http://localhost:9292/pacts/provider/Bar/consumer/Foo/pact-version/0e3369199f4008231946e0245474537443ccda2a",
            "name": "Pact between Foo (v1.0.0) and Bar"
          }
        }
      }
    ]
  },
  "_links": {
    "self": {
      "href": "http://localhost:9292/pacts/provider/Bar/for-verification",
      "title": "Pacts to be verified"
    }
  }
}

Testing

This feature is only released in the Pactflow code, until we have tested and finalised the API. You can use this instance to test with.

Broker base URL: https://test.pact.dius.com.au
username: dXfltyFMgNOFZAxr8io9wJ37iUpY42M
password: O5AIZWxelWbLvqMd8PkAVycBJh2Psyg1

If you're writing a native implementation, please write me a pact and I'll add it to the verifications! Here's an example: https://github.com/pact-foundation/pact-ruby/blob/master/spec/service_providers/pact_ruby_fetch_pacts_for_verification_test.rb