Simplify provider testing against multiple pact files

[cxong]

I'm looking into provider CI testing against a folder of pact files. So far I have something like this:

import os

from pactman.verifier.broker_pact import BrokerPact
from pactman.verifier.result import PytestResult


def test_pacts():
    # Start provider

    try:
        pact_files = (
            f for f in os.listdir("path/to/pact/files")
            if os.path.isfile(f) and os.path.splitext(f)[1] == '.json'
        )
        for pact_file in pact_files:
            pact = BrokerPact.load_file(pact_file, PytestResult)
            for interaction in pact.interactions:
                interaction.verify("provider_url", "provider_pact_url")
                assert interaction.result.success
    finally:
        # Stop provider

I was looking into simplifying this code. Would it be ok to provide something like this:

import pytest

from pactman.test import load_interactions


@pytest.mark.parametrize("interaction", load_interactions("path/to/pact/files"))
def test_interaction(interaction):
    interaction.verify("provider_url", "provider_pact_url")
    assert interaction.result.success

[richard-reece]

I have some code very much like that which I used to use to run our pact verification. I stopped using that approach for a number of reasons, which will only be shared over beer , and they've largely gone away with my implementing of pact mocking in pure Python. The following is probably very close to likely to work still, pulling in pacts for a provider from a pact broker, and running verification against a Django live_server:

from pactman.verifier.broker_pact import BrokerPacts, pact_id
from pactman.verifier.results import PytestResult

@pytest.mark.parametrize('pact', BrokerPacts('MyProvider', result_factory=PytestResult), ids=pact_id)
def test_pacts(live_server, pact):
    pact.verify(live_server.url, live_server.url + SETUP_STATE_URL)

Adding code to, or near to, BrokerPacts (which already implements BrokerPact.load_file), to load pacts from files on disk instead, should be quite possible.

And now I've re-read your first message, I see you found that 😅

[MatthewMiele]

@richard-reece I didnt realise pactman provided this functionality! Its pretty cool. Im trying to rewrite our scripts that currently use the Pact ruby tools for verification to use Pactman. Its almost there. It all seems to work (based of your example). Except im not quite sure how to publish. I know pactman can publish as I have seen the CLI tool code, but im not sure how to adapt your example to also publish the results.

Here's the code I have that is working (without publishing).

pacts = BrokerPacts(PACT_PROVIDER_NAME, result_factory=PytestResult, pact_broker_url=PACT_BROKER_URL)

@pytest.mark.parametrize('interaction', pacts.all_interactions(), ids=pact_id)
def test_pacts(mocked_provider, interaction):
    # Hack to prevent setup url being called. Need to raise a ticket with 
    # pactman to see if we can make setup url in verify() optional. 
    # Not sure if this hack is safe!
    interaction.providerState = None
    interaction.providerStates = None

    interaction.verify(service_url=mocked_provider.url, setup_url="junk")

p.s There's a comment about our setup URL hack. We dont have setup urls as we dont need to do any data setup. I couldnt find a way ignore the setup url failed response or to skip the setup url. Should I raise a ticket for this?

[richard-reece]

@MatthewMiele yeah, it's pretty old functionality that hasn't been used in a while (it existed when pactman was still a wrapper around pact-python and hence the Ruby server, and ran into many issues which resulted in my desire to write a pure-python implementation of mocking/verification).

If I understand your comment correctly, the above verification doesn't publish the results back to the broker and you'd like it to? That's actually at a level above the interaction, unfortunately. For example in the command-line tool, it effectively does:

for pact in pacts:
    for interaction in pact.interactions:
        interaction.verify(args.provider_url, args.provider_setup_url)
    pact.publish_result(provider_version)

whereas the pytest code does:

for pact in pacts:
    for interaction in pact.interactions:
        invoke(test_pacts)

There needs to be something between BrokerPacts and the unit test that handles the publishing. Not sure what that should look like, exactly, but perhaps instead of:

pacts = BrokerPacts(PACT_PROVIDER_NAME, result_factory=PytestResult, 
    pact_broker_url=PACT_BROKER_URL)

we might have:

pacts = pytest_pacts(PACT_PROVIDER_NAME, pact_broker_url=PACT_BROKER_URL,
    publish_results=True, provider_version=VERSION)

And then the thing created by pytest_pacts() can handle the publishing (and make the overall API UX nicer to boot).

In terms of making things like the setup URL work more nicely, yes, please file bugs :)

[MatthewMiele]

Yep that's right. The code I pasted works great and verifies the pacts, just no publishing of the results, and I couldn't see a way to publish without looping myself like you described. Its a shame as your example is really nice and clean with the @pytest.mark.parametrize and result_factory=PytestResult. Was hoping there was someway to keep this format and still be able to publish.

How about like the all_interatctions() there could be an all_pacts() on the BrokerPacts? Then it would be something like..

pacts = BrokerPacts(PACT_PROVIDER_NAME, result_factory=PytestResult, pact_broker_url=PACT_BROKER_URL)

@pytest.mark.parametrize('pact', pacts.all_pacts(), ids=pact_id)
def test_pacts(mocked_provider, pact):
   for interaction in pact.interactions:
       interaction.verify(service_url=mocked_provider.url, setup_url="junk")
   pact.publish_result()

Still its not as nice as the non publishing code. Your idea seems like it would solve the need for the extra manual loop.

Will raise a new bug for the setup URL :)

[MatthewMiele]

...just thinking out loud. Would something like this work?

class PublishingBrokerPacts():
     def all_interactions(self):
        for pact in self:
            for interaction in pact.interactions:
               yield interaction
        pact.publish()

pacts = PublishingBrokerPacts(PACT_PROVIDER_NAME, result_factory=PytestResult, pact_broker_url=PACT_BROKER_URL)

@pytest.mark.parametrize('interaction', pacts.all_interactions(), ids=pact_id)
def test_pacts(mocked_provider, interaction):
    interaction.verify(service_url=mocked_provider.url, setup_url="junk")

[richard-reece]

That's definitely an alternative, yeah. Needs the provider version tho :)

[richard-reece]

I'm put up the above PR which greatly improves the pytest experience. Please have a look, @cxong and @MatthewMiele!