Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add test rule to Makefile and integrate with CI #68

Merged
merged 3 commits into from
Aug 28, 2020

Conversation

idoqo
Copy link
Collaborator

@idoqo idoqo commented Aug 26, 2020

This adds support for testing the documentation against the API.
The data sent in the requests are based on the examples we define in our OpenAPI document, for example, given the OpenAPI spec below:

ContactRequest:
  type: object
  description: "Format of the request body for POST/PUT requests to `/contacts` and related sub-paths."
  properties:
    type:
      type: string
      example: "mail"
    value:
      type: string
      example: "devops@example.com"

Schemathesis will generate this request payload:

{
  "type": "mail", 
  "value": "devops@example.com"
}

Schemathesis checks the response code, content type, and the schema of the JSON response.
The test fails if none of it matches what is in the OpenAPI spec.

The request and response body is logged in build/cassette.yml,
and the output is set to verbose mode so all the data can be seen on stdout.

This adds support for testing the documentation against the API.
The data sent in the requests are based on the examples we define in openapi.yml.

Schemathesis checks the response code, content type and the schema of the json response.
The test fails if none of it matches what is in the OpenAPI spec.

The request and response body is logged in build/cassette.yml,
and the output is set to verbose mode so all the data can be seen on stdout.
Currently, the document is uploaded before running schemathesis,
this makes the upload pipeline dependent on the test pipeline so they are only
uploaded after tests are successful.
Copy link
Contributor

@litleleprikon litleleprikon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a doubts that this tests check the response schemas right. Because in Moira master branch Trigger schema is little bit different than in OpenAPI description. Can you please provide a PR that will proof that schemathesis will fail on incorrect response schema

steps:
- uses: actions/checkout@v2
with:
repository: moira-alert/moira
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to use this repo instead https://github.com/moira-alert/docker-compose/blob/master/docker-compose.yml

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know if it makes sense, but I was thinking we need to test against the same branch name on main Moira API, for example:
feature/add-teams branch on doc repo should test on feature/add-teams on moira repository.
Because if an API endpoint doesn't exist on master but is documented, then the test will fail (with 404 error)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature/add-teams branch on doc repo should test on feature/add-teams on moira repository.

On second thoughts, this brings up the case when the feature is for doc only and doesn't concern the main Moira repo. I think I'd just go ahead and change it to the one you suggested.

@litleleprikon litleleprikon merged commit 6a0355f into master Aug 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants