Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Response schema conformance failures are not properly deduplicated #907

Closed
Stranger6667 opened this issue Dec 4, 2020 · 0 comments · Fixed by #1210
Closed

[BUG] Response schema conformance failures are not properly deduplicated #907

Stranger6667 opened this issue Dec 4, 2020 · 0 comments · Fixed by #1210
Assignees
Labels
Difficulty: Intermediate Requires some experience Priority: Low Cosmetic or low-impact changes Specification: OpenAPI Specific to OpenAPI Type: Bug Errors or unexpected behavior

Comments

@Stranger6667
Copy link
Member

Stranger6667 commented Dec 4, 2020

Describe the bug
When validating response conformance, the failures might not be deduplicated which creates a lot of noise.

To Reproduce
Steps to reproduce the behavior:

  1. Build https://github.com/ajnisbet/opentopodata
  2. Run Schemathesis against the following hand-written schema and supply base URL pointing to the running server:
schemathesis run schema.json --base-url=http://0.0.0.0:3000/v1 -c response_schema_conformance --hypothesis-phases=generate

Note the --hypothesis-phases=generate option, otherwise the test might take much longer

Schema:

{
  "openapi": "3.0.2",
  "info": {
    "title": "Test",
    "description": "Test",
    "version": "0.1.0"
  },
  "servers": [
    {
      "url": "http://0.0.0.0:5000/{basePath}",
      "variables": {
        "basePath": {
          "default": "v1"
        }
      }
    }
  ],
  "paths": {
    "/{dataset_name}": {
      "get": {
        "parameters": [
          {
            "name": "dataset_name",
            "in": "path",
            "required": true,
            "schema": {
              "type": "string",
              "enum": [
                "test-dataset"
              ]
            }
          },
          {
            "name": "locations",
            "in": "query",
            "required": true,
            "style": "pipeDelimited",
            "explode": false,
            "schema": {
              "type": "array",
              "minItems": 2,
              "maxItems": 2,
              "items": {
                "type": "number",
                "minimum": -180.0,
                "maximum": 180.0
              }
            }
          },
          {
            "name": "interpolation",
            "in": "query",
            "required": false,
            "schema": {
              "enum": [
                "nearest",
                "bilinear",
                "cubic"
              ]
            }
          },
          {
            "name": "nodata_value",
            "in": "query",
            "required": false,
            "schema": {
              "anyOf": [
                {
                  "type": "integer"
                },
                {
                  "enum": [
                    "null",
                    "nan"
                  ]
                }
              ]
            }
          }
        ],
        "responses": {
          "200": {
            "description": "OK",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "additionalProperties": false,
                  "properties": {
                    "status": {
                      "enum": ["OK"]
                    },
                    "results": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "additionalProperties": false,
                        "properties": {
                          "elevation": {"type": "integer", "nullable":  true}
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
  1. Observe a lot of very similar failures in the output:
154. The received response does not conform to the defined schema!

Details: 

Additional properties are not allowed ('location' was unexpected)

Failed validating 'additionalProperties' in schema['properties']['results']['items']

Path parameters : {'dataset_name': 'test-dataset'}
Query           : {'locations': '99.92297954590147|-171.28327659808383', 'nodata_value': -1791173676829156192, 'interpolation': 'cubic'}

Run this Python code to reproduce this failure: 

    requests.get('http://0.0.0.0:3000/v1/test-dataset', headers={'User-Agent': 'schemathesis/2.8.4'}, params={'locations': '99.92297954590147|-171.28327659808383', 'nodata_value': -1791173676829156192, 'interpolation': 'cubic'})

Or add this option to your command line parameters: --hypothesis-seed=140907943817362640799165644218694011587


155. The received response does not conform to the defined schema!

Details: 

Additional properties are not allowed ('location' was unexpected)

Failed validating 'additionalProperties' in schema['properties']['results']['items']

Path parameters : {'dataset_name': 'test-dataset'}
Query           : {'locations': '98.18156716347|98.18156716347', 'interpolation': 'cubic', 'nodata_value': 17155926026267753522784395817154392287}

Expected behavior
Only one failure with the given jsonschema-level output is expected

Environment:

  • OS: Linux
  • Python version: 3.8.6
  • Schemathesis version: 2.8.4
  • Spec version: Open API 3.0.2

This issue is connected with #834, after implementing which this behavior should be fixed as well.

As an idea - difflib could help here

@Stranger6667 Stranger6667 added Type: Bug Errors or unexpected behavior Status: Needs Triage Requires initial assessment to categorize and prioritize labels Dec 4, 2020
@Stranger6667 Stranger6667 self-assigned this Dec 4, 2020
@Stranger6667 Stranger6667 added Difficulty: Intermediate Requires some experience Priority: Low Cosmetic or low-impact changes Specification: OpenAPI Specific to OpenAPI Status: Need More Info and removed Status: Needs Triage Requires initial assessment to categorize and prioritize labels Dec 5, 2020
Stranger6667 added a commit that referenced this issue Jun 16, 2021
…ailures that happen to have the same failing validator but different input values

Ref: #907
Stranger6667 added a commit that referenced this issue Jun 16, 2021
…ailures that happen to have the same failing validator but different input values

Ref: #907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Difficulty: Intermediate Requires some experience Priority: Low Cosmetic or low-impact changes Specification: OpenAPI Specific to OpenAPI Type: Bug Errors or unexpected behavior
Projects
None yet
1 participant