Stateful testing

[psegedy]

Hi,
this PR adds an option to test REST application statefully.

Features:

• determine from schema which endpoints are required to call before the tested endpoint and endpoints that have the same required parameters
• minimize 404s by getting required body/path parameters for the tested endpoint, determine which parameters are really required and which could be fuzzed.
• find issues caused by incorrect modification of a resource e.g.:
  • PATCH /systems/{id} with a generated body will modify the resource incorrectly
  • GET /systems/{id} fails because of the previous API call
• modify the schema of an endpoint to add values for required parameters:
  • if the parameter is required, add an enum to the endpoint's schema, e.g.:

    {
      "type": "string",
      "enum": ["red", "amber", "green"]
    }
    

  • if the endpoint is not required, add oneOf with an enum to the schema, e.g.:

    {
      "oneOf": [
        {"type": "string"},
        {"type": "string", "enum": ["red", "amber", "green"]}
    }
    

• hold state of previous tests - previous test results and update list of values for required parameters
• can be executed from CLI using --stateful or from pytest by adding stateful=True parameter while initializing the schema or by test parametrization.
• add an example from dependent endpoints while creating tests.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
7 Code Smells

No Coverage information
0.0% Duplication

[Stranger6667]

Hi @psegedy !

Thank you for making this PR, it will take some time to review it and I will have some questions :)

P.S. I really like the idea of stateful testing

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities (and 0 Security Hotspots to review)
11 Code Smells

No Coverage information
0.0% Duplication

[Stranger6667]

@psegedy

I am very sorry for a long silence on this PR :(

I read your master's thesis about this feature (found on the VUT website) :) Impressive and interesting read.
This week I am going to review these changes in detail, and since the last push there are some changes in Schemathesis that relate to stateful testing:

• I added Open API links support that gives some stateful capabilities if these links are present in the tested schema;
• There is --stateful option in CLI with the only possible input value - links;
• There are some structures & functions implemented to support different approaches for stateful testing;
• The current approach is recursive and endpoints could be tested multiple times, which might be beneficial for your implementation as well (in regard to section 6.3.2 of your thesis)

The general design of Open API links was somehow inspired by this PR and I tried to be forward compatible with changes in this PR. In the end, it could be --stateful=randomized (or maybe some other name variant to reflect this approach better) CLI option to switch between these two approaches or even auto to use one or another per endpoint depending on if links are present or no, or maybe both to combine them.

At first glance, the changes you proposed looked quite invasive to me, but probably it was the most straightforward way without major refactorings in the Schemathesis core. However, now, after adding some stateful capabilities to Schemathesis itself I hope that your PR could be integrated easier.

Let me know what do you think :) If you are willing to continue working on this PR, or I can take it over and re-implement, or we can collaborate somehow.

Cheers

[Stranger6667]

After reviewing the current implementation, I think that the most efficient approach would be splitting this implementation by chunks and adapt them to the current API & existing facilities. I'll work on it soon, not sure how much it might take, but the idea is excellent. I need to add stateful testing capabilities to the pytest plugin first and support Open API links. Then I'll add dependency resolving somewhere isolated and then connect it to the runner and the pytest plugin.

I'd like to keep this PR open as a reference and when everything will be ported I'll close it

[Stranger6667]

The current approach goes another way around - after each endpoint test, its dependencies are checked and executed if needed, and then it goes recursively (it is implemented only inside the runner, though). This one adds dependencies first and then the endpoint itself. It feels like in both cases results are reused and therefore they can hit more lines of the application under test and they both feel equally capable, but I am not completely sure about it - will do some experiments (however with this one the number of tests is known before the execution)

[psegedy]

yeah, the idea was to execute tests against dependencies to gather required values (and possibly modify the same resources) and then execute tests for the (target) endpoint. You are right, results are reused. Correct me if I'm wrong, but I think your recursive approach might test even deeper state-space thanks to testing transitive dependencies. As you say, some experiments are needed, also for different recursion depth limits.

[psegedy]

@Stranger6667
Thank you for the nice feedback! I wanted to send you my master's thesis, but I didn't know it is public already.

While implementing these changes I tried to just add more functionality and maintain the current API without major refactorings. However, some changes could have been invasive, it was a while and I don't remember everything I changed 😄.

I really like the idea with OpenAPI links, it adds determinism to dependency inference, and thank you for making it forward compatible. --stateful==randomized and auto options sound great.

If you are still open to collaboration, I would like to work on it further and port it to the current Schemathesis code base, even if it might take some time to get acquainted with the current implementation.

Linking here master's thesis text - Fuzz testing of REST API and slide deck for a quicker overview.

[Stranger6667]

Cool! Thank you for the link!

It will take ~week for me to finish the planned refactoring - I want to make schemas more generic to support GraphQL tests via pytest and CLI. I will also try to prepare some interfaces for these changes.

Generally, I see two areas where we can start:

1. Adding "stateful" support to the pytest runner without introducing the "inferred" approach (after some time I think that this work fits better than "randomized") to make possible running any stateful tests. The major problem here is that due to recursion it is unknown how many tests will be performed during the pytest's collection phase, but we can entirely skip it and count all recursive test cases as part of the first one. However, it will still require some additional care, because these new tests will need proper pytest_runtest_protocol invocation. In other words, the current pytest plugin should learn how to properly execute additional tests in a recursive manner. Also, I am not sure about the output - probably we can grab the CLI output style and indent the pytest output in the same way by using some other hooks

2. Adding the inferred approach separately to CLI. It would be cool to implement inference separately and then connect it to the BaseOpenAPISchema.get_stateful_tests method, in general, if StatefulTest instances will be generated, then the rest should work automatically

I need some time to plan these things in more detail. I will appreciate your feedback - you could expect the plan around the end of the week

[Stranger6667]

I am going to close this, as stateful testing is implemented. The inference part is tracked separately in #1084