Return Verification Errors in VerificationError object

[lon-io]

Feature description

Currently, verification errors are only implicitly logged to console, meaning I cannot access them programmatically after the tests have been run.

Use case

Consider a verification flow like in the example below, where I run my verification behind a HTTP method. I would like to return the errors to the client calling the HTTP endpoint, for example, from error.data in the catch block, where error is likely an instance of VerificationError

const verifier = new Verifier(options);

try {
  await verifier.verifyProvider();

  res.send('successful verification');
} catch (error) {
  const errors = error?.data;

  res.send(errors);
}

[TimothyJones]

What is the use case for running verification like this?

[mefellows]

I was thinking the same, I'd love to know. The example given doesn't look to be very useful.

One thought could be if it were wrapped into another library/framework and the information needed to be pulled into another UI / interface / report etc

[TimothyJones]

Sorry, I think I wasn’t clear- I’m advocating not putting the redirect in the contract- since the behaviour you want a contract for (usually) isn’t “redirect the client”, it’s “get a response in the following format”.

Whether or not there is a redirect on the way to that response is an implementation detail.

The same argument applies to the URL- probably the specific URL is an implementation detail, and not something the client is relying on.

Remember that the contract is for recording the behaviour the client relies on, not the behaviour of the provider. Unless the client needs the redirect as part of its logic (which is possible, but unlikely) it’s best to have it invisible from the perspective of the consumer test.

[lon-io]

Hi all, thank you for your responses.

The use-case stemmed from the difficulty in reading the default logs to stdout in pact V2 due to the size and verbosity. So we put the verification behind an existing Node.js server as a POST method and tried to read the errors and return them in a JSON response. Afaik, the log readability was improved in pact V3, but I believe there are still advantages to gain from being able to access the error results.

One key example I can mention is for KPIs tracking on endpoints that fail verification repeatedly. Another is piping the verification to consolidated clients like Spotify backstage.

Perhaps, there's a broker API that can be called to re-fetch the results/errors and I probably missed it from the documentation?

In the end, my goal is to be able to access the verification errors outside of the default log.

[TimothyJones]

What problem are you trying to solve?

[lon-io]

@TimothyJones I thought I had explained it in my last comment. I am trying to get the verification errors programmatically so as to use them for further analysis

[TimothyJones]

Apologies, perhaps I wasn't clear. Let me try again.

I understand that you are trying to get the verification errors programmatically, but I'm interested to know why you would want this.

In your further explanation you mentioned issues with the log verbosity, but this has (I believe) been addressed with the upgrade to v10. You also mentioned KPIs for tracking verification results, which would be a pretty custom use of both Pact and KPIs (which is fine). However, I don't think programmatic verification results would be the right way to go about it - it would be better achieved by asking the broker directly, or building the tracking you need into your pipeline.

To me, this feels like an xy problem, where the request is about how to achieve some part of a solution to a problem - I'm asking what the underlying problem is, so that I can advise best and help you find a solution.

Of course, potentially this is a feature request for a widely applicable use case that I haven't thought of - if that's the case, I'm definitely keen to know what it is.

[TimothyJones]

Also, sorry about the unrelated message about redirects. I don't know which issue that was supposed to be on :/

[mefellows]

Also, sorry about the unrelated message about redirects. I don't know which issue that was supposed to be on :/

I think maybe it was intended for pact-foundation/pact-net#441?

[mefellows]

Perhaps, there's a broker API that can be called to re-fetch the results/errors and I probably missed it from the documentation?

There almost certainly is, and if it doesn't exist it wouldn't be a difficult thing to create. I think you can definitely get the most recent (latest) verification result.

I'm also keen to hear more about the use case.