[Question] Only Execute Examples from the OpenAPI Schema

[mohit-sentieo]

I have read all the documentation and a big part of the code and I am unable to find a way to do this.

Only execute examples from the schema.

It keeps returning a FilteredStrategy and I am not sure how to filter out only the case examples.

It's quite surprising that such a simple case is not natively supported.

[Stranger6667]

Hello @mohit-sentieo !
It sounds like a nice feature, indeed. Will work on it for the next release

[Stranger6667]

It is actually possible at the moment with configuring Hypothesis in a certain way:

In code:

from hypothesis import Phase, settings

schema = ...

@schema.parametrize()
@settings(phases=[Phase.explicit])
def test_something(case):
    ...

For CLI, I'll add it soon as a shortcut, currently, it might work like this:

schemathesis run --hypothesis-phase=explicit ...

[Stranger6667]

Let me know any of these options works for you

[mohit-sentieo]

Thank You for the prompt reply. However this doesn't seem to be working.

I'll be back after some debugging :)

[mohit-sentieo]

Hi @Stranger6667

After debugging, I can confirm that hypothesis is working as expected. However schemathesis is not properly adding the examples to the hypothesis strategy.

My OPENAPI3 YAML Schema is below:-

/documents/get:
    get:
      tags:
        - documents
      summary: Find document by ID
      description: >-
        Returns html for a document e.g. 5cff7ca9668114405a001036 ,
        5d40b7385cf17cc7f1000cf3
      operationId: getDocsById
      parameters:
        - name: id
          in: query
          description: ID of the document
          required: true
          schema:
            type: string
          example: 5cff7ca9668114405a001036
      responses:
        '200':
          description: successful operation
          content: {}

I have also tried with examples.

I have tracked the problem to the schemas.py file and these two functions:-

def add_parameter(self, container: Optional[Dict[str, Any]], parameter: Dict[str, Any]) -> Dict[str, Any]:
        """Add parameter object to the container."""
        name = parameter["name"]
        container = container or empty_object()
        container["properties"][name] = self.parameter_to_json_schema(parameter)
        if parameter.get("required", False):
            container["required"].append(name)
        return container

    def parameter_to_json_schema(self, data: Dict[str, Any]) -> Dict[str, Any]:
        """Convert Parameter object to a JSON schema."""
        return {
            key: value
            for key, value in data.items()
            # Do not include keys not supported by JSON schema
            if not (key == "required" and not isinstance(value, list))
        }

They completely filter out the example instead of adding them to the container object.

Can you please help with this ?

[Stranger6667]

Thank you very much for your investigation! Definitely a bug that I missed.

I'll fix it - it happened that the examples testing is done only for examples in "body", but they are processed slightly differently.

[mohit-sentieo]

No worries. I am forking it for my own use case meanwhile. Cheers for the wonderful library.

[Stranger6667]

Thank you :) Let me know if we can do something to support your use case better :)

#382 should fix the issue - "example" field in Open API 3.0 ParameterObject was missed (https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.md#fixed-fields-10)

[Stranger6667]

And supporting "examples" field is probably worth another issue.

Feel free to reopen this issue, if it doesn't solve the problem. Released in 0.23.7 - https://pypi.org/project/schemathesis/0.23.7/