-
-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stateful testing #520
Stateful testing #520
Conversation
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
Hi @psegedy ! Thank you for making this PR, it will take some time to review it and I will have some questions :) P.S. I really like the idea of stateful testing |
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
I am very sorry for a long silence on this PR :( I read your master's thesis about this feature (found on the VUT website) :) Impressive and interesting read.
The general design of Open API links was somehow inspired by this PR and I tried to be forward compatible with changes in this PR. In the end, it could be At first glance, the changes you proposed looked quite invasive to me, but probably it was the most straightforward way without major refactorings in the Schemathesis core. However, now, after adding some stateful capabilities to Schemathesis itself I hope that your PR could be integrated easier. Let me know what do you think :) If you are willing to continue working on this PR, or I can take it over and re-implement, or we can collaborate somehow. Cheers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After reviewing the current implementation, I think that the most efficient approach would be splitting this implementation by chunks and adapt them to the current API & existing facilities. I'll work on it soon, not sure how much it might take, but the idea is excellent. I need to add stateful testing capabilities to the pytest plugin first and support Open API links. Then I'll add dependency resolving somewhere isolated and then connect it to the runner and the pytest plugin.
I'd like to keep this PR open as a reference and when everything will be ported I'll close it
for dependent_endpoint in self.schemathesis_case.sort_by_requirements(dependencies): | ||
for item in self._gen_items(dependent_endpoint): | ||
items.append(item) | ||
for item in self._gen_items(endpoint): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current approach goes another way around - after each endpoint test, its dependencies are checked and executed if needed, and then it goes recursively (it is implemented only inside the runner, though). This one adds dependencies first and then the endpoint itself. It feels like in both cases results are reused and therefore they can hit more lines of the application under test and they both feel equally capable, but I am not completely sure about it - will do some experiments (however with this one the number of tests is known before the execution)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, the idea was to execute tests against dependencies to gather required values (and possibly modify the same resources) and then execute tests for the (target) endpoint. You are right, results are reused. Correct me if I'm wrong, but I think your recursive approach might test even deeper state-space thanks to testing transitive dependencies. As you say, some experiments are needed, also for different recursion depth limits.
@Stranger6667 While implementing these changes I tried to just add more functionality and maintain the current API without major refactorings. However, some changes could have been invasive, it was a while and I don't remember everything I changed 😄. I really like the idea with OpenAPI links, it adds determinism to dependency inference, and thank you for making it forward compatible. If you are still open to collaboration, I would like to work on it further and port it to the current Schemathesis code base, even if it might take some time to get acquainted with the current implementation. Linking here master's thesis text - Fuzz testing of REST API and slide deck for a quicker overview. |
Cool! Thank you for the link! It will take ~week for me to finish the planned refactoring - I want to make schemas more generic to support GraphQL tests via Generally, I see two areas where we can start:
I need some time to plan these things in more detail. I will appreciate your feedback - you could expect the plan around the end of the week |
76b7fd5
to
9f3cd95
Compare
I am going to close this, as stateful testing is implemented. The inference part is tracked separately in #1084 |
Hi,
this PR adds an option to test REST application statefully.
Features:
PATCH /systems/{id}
with a generated body will modify the resource incorrectlyGET /systems/{id}
fails because of the previous API call--stateful
or from pytest by addingstateful=True
parameter while initializing the schema or by test parametrization.