Proposal: use consumer driven contract testing

[ringods]

Hello @ddelnano & @olivierlambert,

The current tests for xo-sdk-go are based on having a Xen Orchestra instance running. This makes it very hard to test the SDK in isolation as well as under CI. I took some time today to come up with an alternative and more lightweight approach, implemented as a proof of concept (POC) in this PR.

In previous years, I have used Consumer Driven Contract Testing (CDCT) a few times on different professional assignments with positive outcome. This approach decouples a consumer of an API from its provider by testing against the messages sent back and forth over the network.

A lot of people have used a mocking/stubbing approach like Wiremock when testing a consumer of an API where Wiremock acts as the replacement for the real service. But it only goes as far as being a solution for testing the consumer side. It is still decoupled from testing the real implementation.

One of the frameworks which supports a coupled way of testing for both consumer and provider is Pact. Pact supports testing both synchronous (http) and asynchronous APIs (messaging). A good intro article is this intro page:

https://docs.pact.io/

When implementing tests with Pact, we start implementing the expected request and response message in our test method using the Pact libraries. This request/response pair is sent to the Pact mock server. The second part of our tests invokes functionality on the consumer, which as part of the implementation, generates the real request. The SDK is in our case connected to our Pact mock server. If the message which is actually sent matches the expected message, then the configured response is sent back to the SDK where we handle the response as part of our SDK implementation. When the result is OK, our test succeeds, otherwise we fail the test. What is common in how Pact works is that pact files with all these request/response pairs are written to disk when running the consumer tests. These pact files can be "replayed" against a provider of the API to test if the expectations of a consumer hold against the real implementation. The exchange of these pact files usually happens via the Pact Broker.

In this PR, I wrote a single pact based test as an example of how this mechanism works: TestCreateUser. I replaced the real jsonrpc2 client with my own version which talks to the Pact Broker, a process which serves the pact files, and in our cases is a stub for the real Xen Orchestra.

This Pact Broker process is part of the Pact CLI tools, which need to be installed separately. Once these tools can be found in your PATH, you can run the pact test like this (with the output added):

$ go test -v ./... -run TestCreateUser
=== RUN   TestCreateUser
2021/11/20 20:46:48 [INFO] checking pact-mock-service within range >= 3.5.0, < 4.0.0
2021/11/20 20:46:49 [INFO] checking pact-provider-verifier within range >= 1.36.1, < 2.0.0
2021/11/20 20:46:49 [INFO] checking pact-broker within range >= 1.22.3
2021/11/20 20:46:49 [INFO] INFO  WEBrick 1.3.1
2021/11/20 20:46:49 [INFO] INFO  ruby 2.2.2 (2015-04-13) [x86_64-darwin13]
2021/11/20 20:46:49 [INFO] INFO  WEBrick::HTTPServer#start: pid=95238 port=51482
2021/11/20 20:46:50 [INFO] INFO: Writing pact before shutting down
2021/11/20 20:46:50 [INFO] 
2021/11/20 20:46:50 [INFO] Writing pact for xenorchestra to /Users/ringods/Projects/xenorchestra/xo-sdk-go/client/pacts/xo-sdk-go-xenorchestra.json
2021/11/20 20:46:50 [INFO] INFO  going to shutdown ...
2021/11/20 20:46:50 [INFO] INFO  WEBrick::HTTPServer#start done.
--- PASS: TestCreateUser (1.72s)
PASS
ok      github.com/vatesfr/xo-sdk-go/client     1.981s

What are your first impressions of this approach?

Note: this is the most straight forward thing I could get working in a day. While working on it I found out that I would better use the message based approach rather then the current synchronous approach using the http based mock server. If this doesn't say much to you at the moment, no problem. I can always explain more later.

Signed-off-by: Ringo De Smet ringo@de-smet.name

[olivierlambert]

Adding @julien-f to get his feeling on this. My first gut feeling is that any change on the API will need to be reflected in the tests too.

[ddelnano]

I have been interested in pact and understand it as a concept, but don't have practical experience with it. I appreciate you starting the discussion on improving the testing workflow. It's been something that has been on my mind as well for the terraform provider (in addition to the client testing).

Unfortunately I'm not too optimistic in this approach. I am concerned that there will be too much overhead with keeping pact's view of the world consistent with XO development. While the XO api has proven to be stable from a backwards compatibility perspective, the terraform provider and this client move significantly slower since this work is a part time effort. Also XO development happens in a different time zone and is disjoint from the xo-sdk-go and provider's development. While I don't believe this disjoint development has caused a problem, I think the more coupling we do between the projects the higher the chance for our part time work to be spent on things that don't push the provider or xo-sdk-go forward.

Therefore I would prefer to invest in lowering the bar on integration testing against a real XO server. This would also help out the terraform provider significantly. I currently run those tests against the Vate's "lab environment", which works great for me but is inaccessible to anyone without internal company access (like those that have contributed to the provider).

What about if we applied for open source Equinix bare metal credit (or something similar) and ran a CI job running against a XO cluster running on it? I know the Vates team has discussed using Equinix bare metal as a platform for XO and so it would also build out a real use case on how to deploy an XO cluster this way (not sure if there has been development here since @olivierlambert and I talked about it months ago).

[olivierlambert]

@ddelnano we successfully integrated Equinix Metal Open Source program. We are planning to use it for our own XCP-ng CI, but we are willing to see how we can integrate other testing within the project (we have to take a look on how we can do that properly in Equinix console).

Adding @stormi in the loop too (but I would also love the @julien-f point of view)

[julien-f]

Hello all,

All that I can say is that xo-server's API has been developed for internal use and even though we try not to break things for external users, its stability is not guarantee. The XO team does not have the resources to monitor and maintain the tests of this repo but we'll do what we can to fix things on our side if you have issues 🙂

We have a (badly maintained TBH) test client for the API, that you may want to take a look at: https://github.com/vatesfr/xen-orchestra/tree/master/packages/xo-server-test

We have recently discussed about the possibility to create a better-designed public API in the future, that would be:

• based on simple HTTP requests (maybe REST) instead of WebSocket for easier fire and forget use
• built incrementally based on the community needs.

But it's very hypothetical for now and will depend on the community interest.

[ringods]

I find it a bit disappointing that we come to this conclusion about the API only at this point. When I initially discussed this on the forum, the answer I got from @olivierlambert was this:

To be fair, I would prefer a Xen Orchestra provider, to provide a better abstraction, as XO is becoming a real middleware for various things nowadays (and overcome some XAPI limitations).

But now, the message from you, @julien-f, is that the XO API isn't intended as a public API. Not the answer I was hoping for.

I hope Vates can come to a decision soon, but I hope you can understand that I will pause any further contributions for now.

[olivierlambert]

I think there's a misunderstanding. What I said on the forum is the target we want to achieve. But @julien-f was telling you about the current state of things. The API wasn't meant for that originally, but it's becoming more and more used anyway. And we are aware of it, that's why we almost never broke it.

In the end, it doesn't mean you shouldn't use it. It means we couldn't maintain both the API and the tests made by a 3rd party tool. That's the point we had initially: it's better to rely on tests based on a real XO that on a mock up API that will be hard to maintain anyway.

It's just a matter of being realistic at this point, and using CI with a real XO seems far more relevant now.

This will be also a very interesting in terms of feedback and what could be done on a future/completely independent public API.