No way to configure webhooks for setting github commit statuses when multiple provider verifications are performed

[vgrigoruk]

We're using a free beta of hosted pact broker at https://pact.dius.com.au/

Our delivery pipelines for consumer and provider are designed in a way that github master branch is automatically deployed to staging environment and release branch deployed to production. We use git commit SHA-1 for provider and consumer versioning. Also, consumer contracts are tagged with git branch name. Provider application versions are tagged with either staging or production tag (depends on which version of provider code was used to run contract verification).

Here are the webhooks we've configured:

• 2 webhooks for triggering provider CI builds (from master and release branches). Webhook payload example (note, ORG, BRANCH and TOKEN are just placeholders):

{
  "consumer": {
    "name": "client"
  },
  "provider": {
    "name": "api"
  },
  "request": {
    "method": "POST",
    "url": "https://circleci.com/api/v1.1/project/github/ORG/api/tree/BRANCH?circle-token=TOKEN",
    "headers": {
      "Content-Type": "application/json"
    },
    "body": {
      "build_parameters": {
        "CIRCLE_JOB": "test_pact",
        "PACT_URL": "${pactbroker.pactUrl}"
      }
    }
  },
  "events": [
    {
      "name": "contract_content_changed"
    }
  ]
}

• webhook for creating github commit statuses:

{
  "consumer": {
    "name": "client"
  },
  "request": {
    "method": "POST",
    "url": "https://api.github.com/repos/ORG/${pactbroker.consumerName}/statuses/${pactbroker.consumerVersionNumber}",
    "headers": {
      "Content-Type": "application/json"
    },
    "body": {
      "state": "${pactbroker.githubVerificationStatus}",
      "description": "Pact Verification Tests",
      "context": "${pactbroker.providerName} ${pactbroker.providerVersionTags}",
      "target_url": "${pactbroker.verificationResultUrl}"
    },
    "username": "machine-user",
    "password": "**********"
  },
  "events": [
    {
      "name": "provider_verification_published"
    },
    {
      "name": "contract_published"
    }
  ]
}

But, every time we create commits in consumer codebase, we see that commit statuses are not working as expected.

Scenario 1

1. Developer makes a change in codebase that leads to contract change and pushes commits to github.
2. Consumer CI executes tests and publishes new contract to pact broker. Pact broker executes both webhooks (trigger provider CI and create github commit status)
   Expected result: 2 statuses are created on consumer github commit: context: api staging and context: api production
   Actual result: only 1 commit status is created on consumer github commit context: api[space], as ${pactbroker.providerVersionTags} is not available at this moment.
3. Provider CI publishes verification results for both master and release code branches (provider versions are tagged with staging and production tags).
4. Pact broker executes github commit status webhook twice and create 2 more github commit statuses
   

Scenario 2

1. Developer makes a change in codebase that doesn't lead to contract change and pushes commits to github.
2. Consumer CI executes tests and publishes new contract to pact broker. Pact broker executes only a webhook that sets github commit status
   Expected result: 2 statuses are created on consumer github commit context: api staging and context: api production
   Actual result: only 1 status is created on consumer github commit (either context: api staging or context: api production, based on which verification was latest)
   

[bethesque]

Hi @vgrigoruk.

Unfortunately, I don't think that there is any way to make the github statuses do what you want using the information that's available to the Pact Broker. In scenario 1, how is the contract_published webhook supposed to know that you have two builds being kicked off, and what the names of the provider version tags will be when they don't exist yet?

What you could do is send a pending status update from the actual triggered CI build, before it starts. That way you know what the provider version tag is supposed to be, and the pending status will only be sent when there is going to be a result coming soon.

In scenario 2, what underlying logic would you want to make the statuses work the way you'd want it, and how would you make it backwards compatible for existing users?

[bethesque]

Ok! I've had a think while I was walking to work from the station (my best thinking time!) When declarative logic is not solving a problem, it's time to move to procedural code. Let's use the right tool for the right problem.

Here's what I'd do.

I'd have a single contract_published webhook trigger a "github status update" CI build. That build would take the consumer version number from the params, and would then call can-i-deploy --pacticipant CONSUMER --version CONSUMER_VERSION --to staging --output json and use the JSON returned to send the correct status to github. Then, repeat with --to prod.

You may be able to reuse this for the provider_verification_published webhook as well, and keep all the logic in one place.

[vgrigoruk]

Thanks for feedback and suggestions @bethesque!
I've also spent some time studying pact broker codebase and now I do realize that it is not possible to achieve what I want with the info that is available to pact broker.
I'll try to implement a custom CI job as per your suggestion and will share results here.

[bethesque]

Interested to hear how this went @vgrigoruk