Multi-artifacts: Secondary artifact not increasing conformance score

[goll-michael]

Hello everyone, I have uploaded two artifacts for a service. The main artifact is a JSON file without examples, and the second artifact is a YAML file specifically for the examples. The goal is to be able to create the two files separately.

To create them, the main artifact is first uploaded using curl via the endpoint /api/artifact/upload?mainArtifact=true. The ID created is then determined and the second artifact with the examples is uploaded using /api/artifact/upload?mainArtifact=false&mainArtifactId=${$primaryId}.

A test is performed via /api/tests:
{
“serviceId”: “‘ + $SERVICE_PRODUCT + ’---‘ + $SERVICE_NAME + ’:' + $SERVICE_VERSION + '”,
“testEndpoint”: “‘ + $MICROCKS_BASE_URL + ’/rest/‘ + $SERVICE_PRODUCT + ’---‘ + $SERVICE_NAME + ’/' + $SERVICE_VERSION + '”,
“runnerType”: “OPEN_API_SCHEMA”,
“timeout”: 30000,
“operationsHeaders”: {
“globals”: []
}
}

In this case, the conformance score is 50%. The cause appears to be missing examples in the main artifact. If one example per endpoint is entered in the main artifact, the conformance score increases to 100%.

Is there a way to perform the test in such a way that the conformance score is 100%, even if the examples are only provided in the secondary artifact and not in the primary artifact?

[lbroudoux]

Hi @goll-michael

First, a remark on your curl command: the mainArtifactId is actually unknown from Microcks. Matching between secondary and primary artifacts is done only on the service name and service version extracted from the artifacts themselves.

I think you're mixing a bit of 2 things: the conformance index and the conformance score.

The conformance index is a grade that estimates how your API contract is actually covered by the samples you’ve attached to it. If your primary artifact has no examples, the conformance index will still be 0 after importing it. But if your secondary artifacts include examples, they are counted toward this index's progress. So you may end up with something like 50% if half of your API paths and responses are covered (if you only describe a 200 response, we expect to have 2 samples to possibly illustrate a success and a failure). It represents the maximum possible conformance score you may achieve if all your tests are successful.

The conformance score is the current score computed during your last test execution. And so, it can never exceed the conformance index.

Those explanations come from https://microcks.io/documentation/explanations/conformance-testing/#conformance-metrics.

Hope it helps,

[goll-michael]

Hello @lbroudoux, thank you for your feedback. My idea was to use the Conformance Index as a quality gate (similar to unit test coverage) to ensure that there are enough examples for consumers. For very simple endpoints that only return random text (fictitious example), for example, I don't think there is a need for an error case.

Primary Artifact (json)
“/example”: {
“get”: {
“responses”: {
“200”: {
“content”: {
“text/plain”: {
“schema”: {
“type”: “string”
}
}
}
}
}
}
}

Secondary Artifact (yaml):
/example:
get:
responses:
200:
content:
text/plain:
examples:
Ok:
value: result

Are there any best practices for defining an error case here?