-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add test rule to Makefile and integrate with CI #68
Conversation
This adds support for testing the documentation against the API. The data sent in the requests are based on the examples we define in openapi.yml. Schemathesis checks the response code, content type and the schema of the json response. The test fails if none of it matches what is in the OpenAPI spec. The request and response body is logged in build/cassette.yml, and the output is set to verbose mode so all the data can be seen on stdout.
Currently, the document is uploaded before running schemathesis, this makes the upload pipeline dependent on the test pipeline so they are only uploaded after tests are successful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a doubts that this tests check the response schemas right. Because in Moira master branch Trigger
schema is little bit different than in OpenAPI description. Can you please provide a PR that will proof that schemathesis will fail on incorrect response schema
steps: | ||
- uses: actions/checkout@v2 | ||
with: | ||
repository: moira-alert/moira |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's better to use this repo instead https://github.com/moira-alert/docker-compose/blob/master/docker-compose.yml
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't know if it makes sense, but I was thinking we need to test against the same branch name on main Moira API, for example:
feature/add-teams
branch on doc
repo should test on feature/add-teams
on moira
repository.
Because if an API endpoint doesn't exist on master but is documented, then the test will fail (with 404 error)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
feature/add-teams
branch ondoc
repo should test onfeature/add-teams
onmoira
repository.
On second thoughts, this brings up the case when the feature
is for doc
only and doesn't concern the main Moira repo. I think I'd just go ahead and change it to the one you suggested.
This adds support for testing the documentation against the API.
The data sent in the requests are based on the examples we define in our OpenAPI document, for example, given the OpenAPI spec below:
Schemathesis will generate this request payload:
Schemathesis checks the response code, content type, and the schema of the JSON response.
The test fails if none of it matches what is in the OpenAPI spec.
The request and response body is logged in build/cassette.yml,
and the output is set to
verbose
mode so all the data can be seen on stdout.