(How/should) we we use the mock server as a stub?

[bethesque]

A new thread for a conversation started here: DiUS/pact-consumer-js-dsl#27 (comment)

[bethesque]

Goals:

• Allow unit tests to be strict about their matching and use a mock server (with verification that all expected interactions have taken place)
• Allow integration tests to be loose in their matching, and use a stub server (no verification that all the expected interactions have taken place)
• Share the interactions between the unit tests and the integration tests, so that we know that every interaction that takes place in the integration test has been verified during the unit test phase, so we can't get a sense of false confidence (otherwise, you could just stub anything in your integration tests, and you'd have no more confidence that it would work in real life than if you'd used a regular stub).

Before we talk implementation, @BenSayers, @mboudreau - do these requirements sound right?

[mboudreau]

@bethesque I'm not certain that we need to specify the difference between a mock and a stub server. In my approach, they are exactly the same, the only difference is if I'm developing with it or testing.

As for the request/respond description, all of my mocks will have path and data regex since I don't care about exactness of variables as long as it's in the same format.

[mboudreau]

@bethesque Replying to your thoughts on the other thread:

I don't see a problem with making the methods to set up and tear down interactions public so that they can be used independently of run method. That should allow @mboudreau to continue his integration testing approach.

The idea of setting up pact to work as a stub server from an existing pact file could work, but we'd need to work out an approach to a certain issue. One of pact's strengths is that you can test the same request, with different responses (eg. 200, 404, when x has a y, when x has no y etc...). This works because you set up the interactions on the mock server on a test-by-test basis - you only have one response mocked for a given request at a time, so it always knows which response to give.

If we used the generated pact file to set up a stub server, it is inevitable that there would be more than one response found for a given request - we'd need to work out how to handle this situation. We can't just take the first response found, because the order of the unit tests determines the order in which the interactions are written to the file, so it would not be predictable.

The other subtlety is that in a unit test, we're at a fine grained level, and we generally want to check exact values of an outgoing request - that's what tells us that "orderId: 345" on our model is being converted to "{order_id: 345}" in our request body. If we use pact for stubbing for integration tests, we want to be much less strict about that - we don't care what the value of the order id is, we just want the mock server to respond to any PUT order request that has the right document structure. We already have this type of matching implemented, we'd just need a way to turn it on when in "stub" mode.

Sorry this is a bit long winded. What we need is:

1. A way to choose which of the interactions that we used in a unit test end up being used in a stub test.
2. A way to relax the matching so the integration tests don't become brittle.

I will give these some thought.

I've given this some thought on how this would work for our current use case.

Ideally, the mock server should only have one response per request, but I can see that it would be easily possible to have more than one by messing up the matching. I believe that the mock service should give back a warning when this happens and that it should only return the 'newest' matching response so that test 'children' can highjack a request from a potential parent suite test.

As for the matching, in our application, we have a restful api, which is where a lot of matching will happen based on the url, then we expect a json back of a specific format. For the URL matching, I quite like how angular ui-router does it mostly because it gives me the flexibility to simply add a variable (up to the next '/') or create a more complex regex for specificity. For every 'matcher', we need to also specify a fake object that gets returned/used when in need (ie. fake url for provider testing, a fake json object for consumer) that matches the requirement.

If we can get this going, our pact tests would be a lot simpler and development much stronger.

[bethesque]

I'm not certain that we need to specify the difference between a mock and a stub server

There is an important difference between a mock and a stub - you don't verify a stub. I believe a stub is what you are asking for - the mock service without verification. Correct me if this is not what you're asking for. I think @BenSayers said it well:

I say stubbed instead of mocked intentionally here as in these tests it is important that the stub behaves the same way as the provider api when it is called, but we do not care how many times it was called or if it was even called at all.

Ideally, the mock server should only have one response per request

In fact, ideally, there should be at least two responses per request - one success, and one failure. If you're not testing the failure scenario, then you have untested paths inside your code. I would typically test the failure scenario with a unit test though, not an integrated test.

Using the common route declaration syntax '/user/:id' will help us match a route, but we still need a concrete example to replay against the provider, so we would still need to give the path a "generate" (example) value. The route syntax is a bit clearer to read, but does not provide any extra functionality over the term. You could, however, write a function that turned a route into a regular expression behind the scenes, so that the tests were more readable.

[bethesque]

@BenSayers, I've been giving the idea of creating a stub server from the pact file some thought and while I really like the idea (it would solve a lot of problems), I'm not sure if I see a good way to share the interactions between unit tests and integration tests as their concerns are different. In a unit test, I want to assert that when I save model with ID 100, that the path it goes to is exactly "/models/100". However, in an integration test, I just care that /models/\d+/ has a stubbed response, and that the document structures match - exact values make the tests brittle and painful.

The difference is in the setup.

Unit test:

helloProvider
      .given("a thing with ID 1234 exists")
      .uponReceiving("a request to update a thing")
      .withRequest("put", "/things/1234", {name: 'Updated thing'})
      .willRespondWith(200, {
        "Content-Type": "application/json"
      }, {
        name: 'Updated thing'
      });

Integration tests:

helloProvider
      .given("a thing with ID 1234 exists")
      .uponReceiving("a request to update a thing")
      .withRequest(
          "put", 
          term("/\/things\/\d+/", "/things/1234"), # This should be a term
          somethingLike({name: 'Updated thing'})) # This should be a somethingLike (type based matching)
      .willRespondWith(200, {
        "Content-Type": "application/json"
      }, {
        name: 'Updated thing'
      });

Do you see the issue? Any ideas?

[BenSayers]

I see the issue you are raising and agree its a concern. Let me ponder on it.

Another question that was raised is how do you tell the server which provider state it should be serving for a particular endpoint. My first thought to solve this is to expose a url to allow you to change provider states from within the integration tests. But that may quickly become cumbersome if you have many providers with many endpoints. There may be a need for the ability to tag states to allow the integration test to change all endpoints from all providers to the states tagged with a particular tag.

@bethesque I assume you have projects that use Pact that have the need for a stub server of this kind for integration testing. I'd like to understand how you solve this problem today, let me bombard you with questions:

• How do you create this server? Do you use a tool or library to help?
• The Pact mock is a mock that gets validated against the real server. How do you maintain this quality in the stub and ensure it stays consistent with the real implementation?
• In the best practices page you mention to use a shared fixture to ensure consistency, but wouldn't you run into the same brittleness issues by taking that approach?
• Does the shared fixture solution help you validate the structure of data sent as part of a http post or put? If not how do you solve this problem?

[bethesque]

Hi Ben,

How do you create this server? Do you use a tool or library to help?

Most of the projects I have worked with Pact have been Ruby microservices, and I have done what I suggested to do - use Pact for unit tests, and use something else for the "top-to-bottom" tests of the service.

The Pact mock is a mock that gets validated against the real server. How do you maintain this quality in the stub and ensure it stays consistent with the real implementation?

We haven't. It hasn't been an issue for us, because the services have been so small, they only had one codepath, and all those codepaths went through the Client class (responsible for making the HTTP calls to the provider) and all the requests that client could make were tested with Pact in unit tests. I don't think this will work for the Javascript integration tests.

In the best practices page you mention to use a shared fixture to ensure consistency, but wouldn't you run into the same brittleness issues by taking that approach?

The fixtures are not necessarily shared JSON fixtures. You could have a shared instance of a model, where one test checks that the ProviderClient class returns some json that can be deserialised to that model, and another test uses that same model to stub the response from the ProviderClient class. This works fine in Ruby, but it's obviously not going to work for Javascript integration tests.

I have recently added a feature to Pact which was missing before, which I think will make me change my position on using Pact at the "top-to-bottom" level of testing - the ability to use a Pact::Term in the path. This means it can be used in integration tests, allowing multiple requests to the same endpoint with different actual IDs without bloating the number of interactions that would need to be verified. You are right that using a shared JSON fixture can be brittle, but this approach would let you use the same fixture in a unit test with exact matching, and then use it in integrated tests with a something_like and term. This approach would still mean you'd need to verify the interactions that had taken place in the integration phase though. I'm still giving this some thought.

Does the shared fixture solution help you validate the structure of data sent as part of a http post or put? If not how do you solve this problem?

Yes.

[mboudreau]

I say the term and something_like is definitely needed. Not sure I'm a fan of the 'something_like' function name, but still, just a wait to validate a json schema (or XML if you're so inclined) is something that would solve many (if not all) of the contract issues we have in our project.

[bethesque]

SomethingLike was never meant to be public! It was a hack that we added that actually turned out to be super useful.

@BenSayers , what are your thoughts on this? https://gist.github.com/bethesque/94df51376c11aeeb6b71

I do not believe that JSON schema would solve our problems. Please see https://github.com/realestate-com-au/pact/wiki/FAQ#why-doesnt-pact-use-json-schema
What will solve our problems is the v2 matching, which is a WIP. This will allow us to say things like "match by type", or "an array where all elements are like this, I don't care what length". Please read: https://gist.github.com/bethesque/39c42c8feff308481449 and https://gist.github.com/bethesque/5a35a3c1cb9fdab6dce7 if you are interested in how v2 matching will work.

[mboudreau]

That's what I mean by json schema. I don't care about values, I just want to know that certain keys are required, some are optional, and all of them have a type (array, object, string, number) and the ability to be hierarchical.

I don't care about the actual values themselves, just the structure.

[bethesque]

Yup, that's the "something like" matching. That's why it became so useful!

[bethesque]

I'm replying to @BenSayers comment on https://gist.github.com/bethesque/94df51376c11aeeb6b71 because gists do not have notifications for comments. (I just happen to have set up my own notifications script, otherwise I'd never have seen it).

Sorry its taken me 3 months to get to this. Is there a canonical place where big picture pact stuff is discussed? I struggle to keep track of the interesting discussions we have on where things are going scattered through github.

I'd like this to be the central place to talk about this issue, as there are proper notifications.

Here are some random thoughts I have on this:

If this is what the unit test looks like, how do you imagine the integration tests would look? Are you thinking that the integration tests would just say "give me all the interactions tagged with integration"?

Yes, it could work that way. I still don't know if this will work, it's just an idea I'm kicking around.

How far would you go with the fussy matching? The above example creates a "thing" with the name "Updated thing". Would you want the fussy matching to support passing any name (so the integration tests just care that name is specified but don't care about the value)? Given there is sometimes a relationship between the request body and the response body that might get complicated.

The fussy matching would be the current, exact matching. The flexible matching would just expect there to be a string. It makes sense in our brains that the response should correspond to the request, however, for the purposes of testing a UI, what we actually care about is that our UI displays what came back from the server, not that the response data matches the request data we sent. That assertion is actually the responsibility of the functional test in the provider.

There is also sometimes a relationship between different requests, e.g. you might have a test that wants to do a GET request for /things/1234 in the unit tests. How would you represent this interaction so that the integration stub behaved correctly. You don't want the stub server to respond to a GET on /things/abc if the PUT you did was /things/123.

You could specify the path in the andSave to the degree you wanted.

.andSaveAsStub({path: "/things/:id", tags: ['integration']});

for all paths matching /things/:id, or for specific ids:

.andSaveAsStub({path: "/things/abc", tags: ['integration']});
...
.andSaveAsStub({path: "/things/123", tags: ['integration']});

Stepping back from this, having the integration test stub server match responses in a different way from the unit test mock server brings with it a bit of complexity, so I'm starting to question the assumptions that it should work this way. You seem to be a lot more concerned then I am about having the integration stub mock server care about exact values. I assume you've been burned by something that I haven't. Can you explain in a bit more detail why you think this is important?

I think you might have it the wrong way around. I prefer the mock server to use "lax matching" for integration tests, because otherwise the tests are very brittle and hard to maintain. I've been burnt by that. I prefer the unit tests to use the fussy exact matching, because they are much easier to maintain, and can be very focussed.

I do not know if this idea would work at all, it might just be better to have two completely separate sets of interactions, the specific unit test ones, and the flexible integration tests ones.