Skip to content

Latest commit

 

History

History
2374 lines (1665 loc) · 94.9 KB

changelog.rst

File metadata and controls

2374 lines (1665 loc) · 94.9 KB
Raw

Changelog

Unreleased - TBD

Changed

  • ExecutionEvent.asdict adds the event_type field which is the event class name.

3.9.0 - 2021-06-07

Added

  • GraphQL support in CLI. #746
  • A way to stop the Schemathesis runner's event stream manually via events.stop() / events.finish() methods. #1202

Changed

  • Avoid pytest warnings when internal Schemathesis classes are in the test module scope.

3.8.0 - 2021-06-03

Added

  • Negative testing. #65
  • Case.data_generation_method attribute that provides the information of the underlying data generation method (e.g. positive or negative)

Changed

  • Raise UsageError if schema.parametrize or schema.given are applied to the same function more than once. #1194
  • Python values of True, False and None are converted to their JSON equivalents when generated for path parameters or query. #1166
  • Bump hypothesis-jsonschema version. It allows the end-user to override known string formats.
  • Bump hypothesis version.
  • APIOperation.make_case behavior. If no media_type is passed along with body, then it tries to infer the proper media type and raises an error if it is not possible. #1094

Fixed

  • Compatibility with hypothesis>=6.13.3.

3.7.8 - 2021-06-02

Fixed

  • Open API style & explode for parameters derived from security definitions.

3.7.7 - 2021-06-01

Fixed

  • Apply the Open API's style & explode keywords to explicit examples. #1190

3.7.6 - 2021-05-31

Fixed

  • Disable filtering optimization for headers when there are keywords other than type. #1189

3.7.5 - 2021-05-31

Fixed

  • Too much filtering in headers that have schemas with the pattern keyword. #1189

3.7.4 - 2021-05-28

Changed

  • Internal: SerializedCase.path_template returns path templates as they are in the schema, without base path.

3.7.3 - 2021-05-28

Fixed

  • Invalid multipart payload generated for unusual schemas for the multipart/form-data media type.

Performance

  • Reduce the amount of filtering needed to generate valid headers and cookies.

3.7.2 - 2021-05-27

Added

  • SerializedCase.media_type that stores the information about what media type was used for a particular case.

Fixed

  • Internal error on unusual schemas for the multipart/form-data media type. #1152
  • Ignored explicit Content-Type override in Case.as_requests_kwargs.

3.7.1 - 2021-05-23

Added

  • Internal: FailureContext.title attribute that gives a short failure description.
  • Internal: FailureContext.message attribute that gives a longer failure description.

Changed

  • Rename JSONDecodeErrorContext.message to JSONDecodeErrorContext.validation_message for consistency.
  • Store the more precise schema & instance in ValidationErrorContext.
  • Rename ResponseTimeout to RequestTimeout.

3.7.0 - 2021-05-23

Added

  • Additional context for each failure coming from the runner. It allows the end-user to customize failure formatting.

Changed

  • Use different exception classes for not_a_server_error and status_code_conformance checks. It improves the variance of found errors.
  • All network requests (not WSGI) now have the default timeout of 10 seconds. If the response is time-outing, Schemathesis will report it as a failure. It also solves the case when the tested app hangs. #1164
  • The default test duration deadline is extended to 15 seconds.

3.6.11 - 2021-05-20

Added

  • Internal: BeforeExecution.verbose_name & SerializedCase.verbose_name that reflect specification-specific API operation name.

3.6.10 - 2021-05-17

Changed

  • Explicitly add colorama to project's dependencies.
  • Bump hypothesis-jsonschema version.

3.6.9 - 2021-05-14

Fixed

  • Ignored $ref keyword in schemas with deeply nested references. #1167
  • Ignored Open API specific keywords & types in schemas with deeply nested references. #1162

3.6.8 - 2021-05-13

Changed

  • Relax dependency on starlette to >=0.13,<1. #1160

3.6.7 - 2021-05-12

Fixed

  • Missing support for the date string format (only full-date was supported).

3.6.6 - 2021-05-07

Changed

  • Improve error message for failing Hypothesis deadline healthcheck in CLI. #880

3.6.5 - 2021-05-07

Added

  • Support for disabling ANSI color escape codes via the NO_COLOR <https://no-color.org/> environment variable or the --no-color CLI option. #1153

Changed

  • Generate valid header values for Bearer auth by construction rather than by filtering.

3.6.4 - 2021-04-30

Changed

  • Bump minimum hypothesis-graphql version to 0.5.0. It brings support for interfaces and unions and fixes a couple of bugs in query generation.

3.6.3 - 2021-04-20

Fixed

  • Bump minimum hypothesis-graphql version to 0.4.1. It fixes a problem with generating queries with surrogate characters.
  • UnicodeEncodeError when sending application/octet-stream payloads that have no format: binary in their schemas. #1134

3.6.2 - 2021-04-15

Fixed

  • Windows: UnicodeDecodeError during schema loading via the from_path loader if it contains certain Unicode symbols. from_path loader defaults to UTF-8 from now on.

3.6.1 - 2021-04-09

Fixed

  • Using parametrized pytest fixtures with the from_pytest_fixture loader. #1121

3.6.0 - 2021-04-04

Added

  • Custom keyword arguments to schemathesis.graphql.from_url that are proxied to requests.post.
  • from_wsgi, from_asgi, from_path and from_file loaders for GraphQL apps. #1097, #1100
  • Support for data_generation_methods and code_sample_style in all GraphQL loaders.
  • Support for app & base_url arguments for the from_pytest_fixture runner.
  • Initial support for GraphQL schemas in the Schemathesis runner.
import schemathesis

# Load schema
schema = schemathesis.graphql.from_url("http://localhost:8000/graphql")
# Initialize runner
runner = schemathesis.runner.from_schema(schema)
# Emit events
for event in runner.execute():
    ...

Breaking

  • Loaders' signatures are unified. Most of the arguments became keyword-only. All except the first two for ASGI/WSGI, all except the first one for the others. It forces loader calls to be more consistent.
# BEFORE
schema = schemathesis.from_uri(
    "http://example.com/openapi.json", "http://127.0.0.1:8000/", "GET"
)
# NOW
schema = schemathesis.from_uri(
    "http://example.com/openapi.json", base_url="http://127.0.0.1:8000/", method="GET"
)

Changed

  • Schemathesis generates separate tests for each field defined in the GraphQL Query type. It makes the testing process unified for both Open API and GraphQL schemas.
  • IDs for GraphQL tests use the corresponding Query field instead of HTTP method & path.
  • Do not show overly verbose raw schemas in Hypothesis output for failed GraphQL tests.
  • The schemathesis.graphql.from_url loader now uses the usual Schemathesis User-Agent.
  • The Hypothesis database now uses separate entries for each API operation when executed via CLI. It increases its effectiveness when tests are re-run.
  • Module schemathesis.loaders is moved to schemathesis.specs.openapi.loaders.
  • Show a more specific exception on incorrect usage of the from_path loader in the Schemathesis runner.

Deprecated

  • schemathesis.runner.prepare will be removed in Schemathesis 4.0. Use schemathesis.runner.from_schema instead. With this change, the schema loading part goes to your code, similar to using the regular Schemathesis Python API. It leads to a unified user experience where the starting point is API schema loading, which is much clearer than passing a callback & keyword arguments to the prepare function.

Fixed

  • Add the missing @schema.given implementation for schemas created via the from_pytest_fixture loader. #1093
  • Silently ignoring some incorrect usages of @schema.given.
  • Fixups examples were using the incorrect fixup name.
  • Return type of make_case for GraphQL schemas.
  • Missed operation_id argument in from_asgi loader.

Removed

  • Undocumented way to install fixups via the fixups argument for schemathesis.runner.prepare is removed.

3.5.3 - 2021-03-27

Fixed

3.5.2 - 2021-03-24

Changed

  • Prefix worker thread names with schemathesis_.

3.5.1 - 2021-03-23

Fixed

  • Encoding for response payloads displayed in the CLI output. #1073
  • Use actual charset (from flask.Response.mimetype_params) when storing WSGI responses rather than defaulting to flask.Response.charset.

3.5.0 - 2021-03-22

Added

  • before_generate_case hook, that allows the user to modify or filter generated Case instances. #1067

Fixed

  • Missing body parameters during Open API links processing in CLI. #1069
  • Output types for evaluation results of $response.body and $request.body runtime expressions. #1068

3.4.1 - 2021-03-21

Added

  • event_type field to the debug output.

3.4.0 - 2021-03-20

Added

  • --debug-output-file CLI option to enable storing the underlying runner events in the JSON Lines format in a separate file for debugging purposes. #1059

Changed

  • Make Request.body, Response.body and Response.encoding internal attributes optional. For Request, it means that absent body will lead to Request.body to be None. For Response, body will be None if the app response did not have any payload. Previously these values were empty strings, which was not distinguishable from the cases described above. For the end-user, it means that in VCR cassettes, fields request.body and response.body may be absent.
  • models.Status enum now has string values for more readable representation.

3.3.1 - 2021-03-18

Fixed

  • Displaying wrong headers in the FAILURES block of the CLI output. #792

3.3.0 - 2021-03-17

Added

  • Display failing response payload in the CLI output, similarly to the pytest plugin output. #1050
  • A way to control which code sample style to use - Python or cURL. #908

Fixed

  • UnicodeDecodeError when generating cURL commands for failed test case reproduction if the request's body contains non-UTF8 characters.

Internal

  • Extra information to events, emitted by the Schemathesis runner.

3.2.2 - 2021-03-11

Added

  • Support for Hypothesis 6. #1013

3.2.1 - 2021-03-10

Fixed

  • Wrong test results in some cases when the tested schema contains a media type that Schemathesis doesn't know how to work with. #1046

3.2.0 - 2021-03-09

Performance

  • Add an internal caching layer for data generation strategies. It relies on the fact that the internal BaseSchema structure is not mutated over time. It is not directly possible through the public API and is discouraged from doing through hook functions.

Changed

  • APIOperation and subclasses of Parameter are now compared by their identity rather than by value.

3.1.3 - 2021-03-08

Added

  • count_operations boolean flag to runner.prepare. In case of False value, Schemathesis won't count the total number of operations upfront. It improves performance for the direct runner usage, especially on large schemas. Schemathesis CLI will still use these calculations to display the progress during execution, but this behavior may become configurable in the future.

3.1.2 - 2021-03-08

Fixed

  • Percent-encode the generated . and .. strings in path parameters to avoid resolving relative paths and changing the tested path structure. #1036

3.1.1 - 2021-03-05

Fixed

  • Loosen importlib-metadata version constraint and update pyproject.toml #1039

3.1.0 - 2021-02-11

Added

  • Support for external examples via the externalValue keyword. #884

Fixed

  • Prevent a small terminal width causing a crash (due to negative length used in an f-string) when printing percentage
  • Support the latest cryptography version in Docker images. #1033

3.0.9 - 2021-02-10

Fixed

  • Return a default terminal size to prevent crashes on systems with zero-width terminals (some CI/CD servers).

3.0.8 - 2021-02-04

  • This release updates the documentation to be in-line with the current state.

3.0.7 - 2021-01-31

Fixed

  • Docker tags for Buster-based images.

3.0.6 - 2021-01-31

  • Packaging-only release for Docker images based on Debian Buster. #1028

3.0.5 - 2021-01-30

Fixed

  • Allow to use any iterable type for checks and additional_checks arguments to Case.validate_response.

3.0.4 - 2021-01-19

Fixed

  • Generating stateful tests, with common parameters behind a reference. #1020
  • Programmatic addition of Open API links via add_link when schema validation is disabled and response status codes are noted as integers. #1022

Changed

  • When operations are resolved by operationId then the same reference resolving logic is applied as in other cases. This change leads to less reference inlining and lower memory consumption for deeply nested schemas. #945

3.0.3 - 2021-01-18

Fixed

  • Flaky Hypothesis error during explicit examples generation. #1018

3.0.2 - 2021-01-15

Fixed

  • Processing parameters common for multiple API operations if they are behind a reference. #1015

3.0.1 - 2021-01-15

Added

  • YAML serialization for text/yaml, text/x-yaml, application/x-yaml and text/vnd.yaml media types. #1010.

3.0.0 - 2021-01-14

Added

  • Support for sending text/plain payload as test data. Including variants with non-default charset. #850, #939
  • Generating data for all media types defined for an operation. #690
  • Support for user-defined media types serialization. You can define how Schemathesis should handle media types defined in your schema or customize existing (like application/json).
  • The response_schema_conformance check now runs on media types that are encoded with JSON. For example, application/problem+json. #920
  • Base URL for GraphQL schemas. It allows you to load the schema from one place but send test requests to another one. #934
  • A helpful error message when an operation is not found during the direct schema access. #812
  • --dry-run CLI option. When applied, Schemathesis won't send any data to the server and won't perform any response checks. #963
  • A better error message when the API schema contains an invalid regular expression syntax. #1003

Changed

  • Open API parameters parsing to unblock supporting multiple media types per operation. Their definitions aren't converted to JSON Schema equivalents right away but deferred instead and stored as-is.
  • Missing required: true in path parameters definition is now automatically enforced if schema validation is disabled. According to the Open API spec, the required keyword value should be true for path parameters. This change allows Schemathesis to generate test cases even for endpoints containing optional path parameters (which is not compliant with the spec). #941
  • Using --auth together with --header that sets the Authorization header causes a validation error. Before, the --header value was ignored in such cases, and the basic auth passed in --auth was used. #911
  • When hypothesis-jsonschema fails to resolve recursive references, the test is skipped with an error message that indicates why it happens.
  • Shorter error messages when API operations have logical errors in their schema. For example, when the maximum is less than the minimum - {"type": "integer", "minimum": 5, "maximum": 4}.
  • If multiple non-check related failures happens during a test of a single API operation, they are displayed as is, instead of Hypothesis-level error messages about multiple found failures or flaky tests. #975
  • Catch schema parsing errors, that are caused by YAML parsing.
  • The built-in test server now accepts --operations instead of --endpoints.
  • Display Collected API operations instead of collected endpoints in the CLI. #869
  • --skip-deprecated-endpoints is renamed to --skip-deprecated-operations. #869
  • Rename various internal API methods that contained endpoint in their names. #869
  • Bump hypothesis-jsonschema version to 0.19.0. This version improves the handling of unsupported regular expression syntax and can generate data for a subset of schemas containing such regular expressions.
  • Schemathesis doesn't stop testing on errors during schema parsing. These errors are handled the same way as other errors during the testing process. It allows Schemathesis to test API operations with valid definitions and report problematic operations instead of failing the whole run. #999

Fixed

  • Allow generating requests without payload if the schema does not require it. #916
  • Allow sending null as request payload if the schema expects it. #919
  • CLI failure if the tested operation is GET and has payload examples. #925
  • Excessive reference inlining that leads to out-of-memory for large schemas with deep references. #945, #671
  • --exitfirst CLI option trims the progress bar output when a failure occurs. #951
  • Internal error if filling missing explicit examples led to Unsatisfiable errors. #904
  • Do not suggest to disable schema validation if it is already disabled. #914
  • Skip explicit examples generation if this phase is disabled via config. #905
  • Unsatisfiable error in stateful testing caused by all API operations having inbound links. #965, #822
  • A possibility to override APIStateMachine.step. #970
  • TypeError on nullable parameters during Open API specific serialization. #980
  • Invalid types in x-examples. #982
  • CLI crash on schemas with operation names longer than the current terminal width. #990
  • Handling of API operations that contain reserved characters in their paths. #992
  • CLI execution stops on errors during example generation. #994
  • Fill missing properties in incomplete explicit examples for non-body parameters. #1007

Deprecated

  • HookContext.endpoint. Use HookContext.operation instead.
  • Case.endpoint. Use Case.operation instead.

Performance

  • Use compiled versions of Open API spec validators.
  • Decrease CLI memory usage. #987
  • Various improvements relevant to processing of API operation definitions. It gives ~20% improvement on large schemas with many references.

Removed

  • Case.form_data. Use Case.body instead.
  • Endpoint.form_data. Use Endpoint.body instead.
  • before_generate_form_data hook. Use before_generate_body instead.
  • Deprecated stateful testing integration from our pytest plugin.

Note

This release features multiple backward-incompatible changes. The first one is removing form_data and hooks related to it -all payload related actions can be done via body and its hooks. The second one involves renaming the so-called "endpoint" to "operation". The main reason for this is to generalize terminology and make it applicable to GraphQL schemas, as all Schemathesis internals are more suited to work with semantically different API operations rather than with endpoints that are often connected with URLs and HTTP methods. It brings the possibility to reuse the same concepts for Open API and GraphQL - in the future, unit tests will cover individual API operations in GraphQL, rather than everything available under the same "endpoint".

2.8.5 - 2020-12-15

Added

  • auto variant for the --workers CLI option that automatically detects the number of available CPU cores to run tests on. #917

2.8.4 - 2020-11-27

Fixed

  • Use --request-tls-verify during schema loading as well. #897

2.8.3 - 2020-11-27

Added

  • Display failed response payload in the error output for the pytest plugin. #895

Changed

  • In pytest plugin output, Schemathesis error classes use the CheckFailed name. Before, they had not readable "internal" names.
  • Hypothesis falsifying examples. The code does not include Case attributes with default values to improve readability. #886

2.8.2 - 2020-11-25

Fixed

  • Internal error in CLI, when the base_url is an invalid IPv6. #890
  • Internal error in CLI, when a malformed regex is passed to -E / -M / -T / -O CLI options. #889

2.8.1 - 2020-11-24

Added

  • --force-schema-version CLI option to force Schemathesis to use the specific Open API spec version when parsing the schema. #876

Changed

  • The content_type_conformance check now raises a well-formed error message when encounters a malformed media type value. #877

Fixed

  • Internal error during verifying explicit examples if an example has no value key. #882

2.8.0 - 2020-11-24

Added

  • --request-tls-verify CLI option, that controls whether Schemathesis verifies the server's TLS certificate. You can also pass the path to a CA_BUNDLE file for private certs. #830

Changed

  • In CLI, if an endpoint contains an invalid schema, show a message about the --validate-schema CLI option. #855

Fixed

  • Handling of 204 responses in the response_schema_conformance check. Before, all responses were required to have the Content-Type header. #844
  • Catch OverflowError when an invalid regex is passed to -E / -M / -T / -O CLI options. #870
  • Internal error in CLI, when the schema location is an invalid IPv6. #872
  • Collecting Open API links behind references via CLI. #874

Deprecated

  • Using of Case.form_data and Endpoint.form_data. In the 3.0 release, you'll need to use relevant body attributes instead. This change includes deprecation of the before_generate_form_data hook, use before_generate_body instead. The reason for this is the upcoming unification of parameter handling and their serialization.
  • --stateful-recursion-limit. It will be removed in 3.0 as a part of removing the old stateful testing approach. This parameter is no-op.

2.7.7 - 2020-11-13

Fixed

  • Missed headers in Endpoint.partial_deepcopy.

2.7.6 - 2020-11-12

Added

  • An option to set data generation methods. At the moment, it includes only "positive", which means that Schemathesis will generate data that matches the schema.

Fixed

  • Pinned dependency on attrs that caused an error on fresh installations. #858

2.7.5 - 2020-11-09

Fixed

  • Invalid keyword in code samples that Schemathesis suggests to run to reproduce errors. #851

2.7.4 - 2020-11-07

Added

  • New relative_path property for BeforeExecution and AfterExecution events. It represents an operation path as it is in the schema definition.

2.7.3 - 2020-11-05

Fixed

  • Internal error on malformed JSON when the response_conformance check is used. #832

2.7.2 - 2020-11-05

Added

  • Shortcut for response validation when Schemathesis's data generation is not used. #485

Changed

  • Improve the error message when the application can not be loaded from the value passed to the --app command-line option. #836
  • Security definitions are now serialized as other parameters. At the moment, it means that the generated values will be coerced to strings, which is a no-op. However, types of security definitions might be affected by the "Negative testing" feature in the future. Therefore this change is mostly for future-compatibility. #841

Fixed

  • Internal error when a "header" / "cookie" parameter were not coerced to a string before filtration. #839

2.7.1 - 2020-10-22

Fixed

  • Adding new Open API links via the add_link method, when the related PathItem contains a reference. #824

2.7.0 - 2020-10-21

Added

  • New approach to stateful testing, based on the Hypothesis's RuleBasedStateMachine. #737
  • Case.validate_response accepts the new additional_checks argument. It provides a way to execute additional checks in addition to existing ones.

Changed

  • The response_schema_conformance and content_type_conformance checks fail unconditionally if the input response has no Content-Type header. #816

Fixed

  • Failure reproduction code missing values that were explicitly passed to call_* methods during testing. #814

Deprecated

  • Using stateful=Stateful.links in schema loaders and parametrize. Use schema.as_state_machine().TestCase instead. The old approach to stateful testing will be removed in 3.0. See the Stateful testing section of our documentation for more information.

2.6.1 - 2020-10-19

Added

  • New method as_curl_command added to the Case class. #689

2.6.0 - 2020-10-06

Added

  • Support for passing Hypothesis strategies to tests created with schema.parametrize by using schema.given decorator. #768
  • Support for PEP561. #748
  • Shortcut for calling & validation. #738
  • New hook to pre-commit, rstcheck, as well as updates to documentation based on rstcheck. #734
  • New check for maximum response time and corresponding CLI option --max-response-time. #716
  • New response_headers_conformance check that verifies the presence of all headers defined for a response. #742
  • New field with information about executed checks in cassettes. #702
  • New port parameter added to from_uri() method. #706
  • A code snippet to reproduce a failed check when running Python tests. #793
  • Python 3.9 support. #731
  • Ability to skip deprecated endpoints with --skip-deprecated-endpoints CLI option and skip_deprecated_operations=True argument to schema loaders. #715

Fixed

  • User-Agent header overriding the passed one. #757
  • Default User-Agent header in Case.call. #717
  • Status of individual interactions in VCR cassettes. Before this change, all statuses were taken from the overall test outcome, rather than from the check results for a particular response. #695
  • Escaping header values in VCR cassettes. #783
  • Escaping HTTP response message in VCR cassettes. #788

Changed

  • Case.as_requests_kwargs and Case.as_werkzeug_kwargs now return the User-Agent header. This change also affects code snippets for failure reproduction - all snippets will include the User-Agent header.

Performance

  • Speed up generation of headers, cookies, and formData parameters when their schemas do not define the type keyword. #795

2.5.1 - 2020-09-30

This release contains only documentation updates which are necessary to upload to PyPI.

2.5.0 - 2020-09-27

Added

  • Stateful testing via Open API links for the pytest runner. #616
  • Support for GraphQL tests for the pytest runner. #649

Fixed

  • Progress percentage in the terminal output for "lazy" schemas. #636

Changed

  • Check name is no longer displayed in the CLI output, since its verbose message is already displayed. This change also simplifies the internal structure of the runner events.
  • The stateful argument type in the runner.prepare is Optional[Stateful] instead of Optional[str]. Use schemathesis.Stateful enum.

2.4.1 - 2020-09-17

Changed

  • Hide Case.endpoint from representation. Its representation decreases the usability of the pytest's output. #719
  • Return registered functions from register_target and register_check decorators. #721

Fixed

  • Possible IndexError when a user-defined check raises an exception without a message. #718

2.4.0 - 2020-09-15

Added

  • Ability to register custom targets for targeted testing. #686

Changed

  • The AfterExecution event now has path and method fields, similar to the BeforeExecution one. The goal is to make these events self-contained, which improves their usability.

2.3.4 - 2020-09-11

Changed

  • The default Hypothesis's deadline setting for tests with schema.parametrize is set to 500 ms for consistency with the CLI behavior. #705

Fixed

  • Encoding error when writing a cassette on Windows. #708

2.3.3 - 2020-08-04

Fixed

  • KeyError during the content_type_conformance check if the response has no Content-Type header. #692

2.3.2 - 2020-08-04

Added

  • Run checks conditionally.

2.3.1 - 2020-07-28

Fixed

  • IndexError when examples list is empty.

2.3.0 - 2020-07-26

Added

  • Possibility to generate values for in: formData parameters that are non-bytes or contain non-bytes (e.g., inside an array). #665

Changed

  • Error message for cases when a path parameter is in the template but is not defined in the parameters list or missing required: true in its definition. #667
  • Bump minimum required hypothesis-jsonschema version to 0.17.0. This allows Schemathesis to use the custom_formats argument in from_schema calls and avoid using its private API. #684

Fixed

  • ValueError during sending a request with test payload if the endpoint defines a parameter with type: array and in: formData. #661
  • KeyError while processing a schema with nullable parameters and in: body. #660
  • StopIteration during requestBody processing if it has empty "content" value. #673
  • AttributeError during generation of "multipart/form-data" parameters that have no "type" defined. #675
  • Support for properties named "$ref" in object schemas. Previously, it was causing TypeError. #672
  • Generating illegal Unicode surrogates in the path. #668
  • Invalid development dependency on graphql-server-core package. #658

2.2.1 - 2020-07-22

Fixed

  • Possible UnicodeEncodeError during generation of Authorization header values for endpoints with basic security scheme. #656

2.2.0 - 2020-07-14

Added

  • schemathesis.graphql.from_dict loader allows you to use GraphQL schemas represented as a dictionary for testing.
  • before_load_schema hook for GraphQL schemas.

Fixed

  • Serialization of non-string parameters. #651

2.1.0 - 2020-07-06

Added

  • Support for property-level examples. #467

Fixed

  • Content-type conformance check for cases when Open API 3.0 schemas contain "default" response definitions. #641
  • Handling of multipart requests for Open API 3.0 schemas. #640
  • Sending non-file form fields in multipart requests. #647

Removed

  • Deprecated skip_validation argument to HookDispatcher.apply.
  • Deprecated _accepts_context internal function.

2.0.0 - 2020-07-01

Changed

  • BREAKING. Base URL handling. base_url now is treated as one with a base path included. You should pass a full base URL now instead:
schemathesis run --base-url=http://127.0.0.1:8080/api/v2 ...

This value will override basePath / servers[0].url defined in your schema if you use Open API 2.0 / 3.0 respectively. Previously if you pass a base URL like the one above, it was concatenated with the base path defined in the schema, which leads to a lack of ability to redefine the base path. #511

Fixed

  • Show the correct URL in CLI progress when the base URL is overridden, including the path part. #511
  • Construct valid URL when overriding base URL with base path. #511

Example:

Base URL in the schema         : http://0.0.0.0:8081/api/v1
`--base-url` value in CLI      : http://0.0.0.0:8081/api/v2
Full URLs before this change   : http://0.0.0.0:8081/api/v2/api/v1/users/  # INVALID!
Full URLs after this change    : http://0.0.0.0:8081/api/v2/users/         # VALID!

Removed

  • Support for hooks without context argument in the first position.
  • Hooks registration by name and function. Use register decorators instead. For more details, see the "Customization" section in our documentation.
  • BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

1.10.0 - 2020-06-28

Added

  • loaders.from_asgi supports making calls to ASGI-compliant application (For example: FastAPI). #521
  • Support for GraphQL strategies.

Fixed

  • Passing custom headers to schema loader for WSGI / ASGI apps. #631

1.9.1 - 2020-06-21

Fixed

  • Schema validation error on schemas containing numeric values in scientific notation without a dot. #629

1.9.0 - 2020-06-20

Added

  • Pass the original case's response to the add_case hook.
  • Support for multiple examples with OpenAPI examples. #589
  • --verbosity CLI option to minimize the error output. #598
  • Allow registering function-level hooks without passing their name as the first argument to apply. #618
  • Support for hook usage via LazySchema / from_pytest_fixture. #617

Changed

  • Tests with invalid schemas marked as errors, instead of failures. #622

Fixed

  • Crash during the generation of loosely-defined headers. #621
  • Show exception information for test runs on invalid schemas with --validate-schema=false command-line option. Before, the output sections for invalid endpoints were empty. #622

1.8.0 - 2020-06-15

Fixed

  • Tests with invalid schemas are marked as failed instead of passed when hypothesis-jsonschema>=0.16 is installed. #614
  • KeyError during creating an endpoint strategy if it contains a reference. #612

Changed

  • Require hypothesis-jsonschema>=0.16. #614
  • Pass original InvalidSchema text to pytest.fail call.

1.7.0 - 2020-05-30

Added

  • Support for YAML files in references via HTTPS & HTTP schemas. #600
  • Stateful testing support via Open API links syntax. #548
  • New add_case hook. #458
  • Support for parameter serialization formats in Open API 2 / 3. For example pipeDelimited or deepObject. #599
  • Support serializing parameters with application/json content-type. #594

Changed

  • The minimum required versions for Hypothesis and hypothesis-jsonschema are 5.15.0 and 0.11.1 respectively. The main reason is this fix that is required for stability of Open API links feature when it is executed in multiple threads.

1.6.3 - 2020-05-26

Fixed

  • Support for a colon symbol (:) inside of a header value passed via CLI. #596

1.6.2 - 2020-05-15

Fixed

  • Partially generated explicit examples are always valid and can be used in requests. #582

1.6.1 - 2020-05-13

Changed

  • Look at the current working directory when loading hooks for CLI. #586

1.6.0 - 2020-05-10

Added

  • New before_add_examples hook. #571
  • New after_init_cli_run_handlers hook. #575

Fixed

  • Passing workers_num to ThreadPoolRunner leads to always using 2 workers in this worker kind. #579

1.5.1 - 2020-05-08

Fixed

  • Display proper headers in reproduction code when headers are overridden. #566

1.5.0 - 2020-05-06

Added

  • Display a suggestion to disable schema validation on schema loading errors in CLI. #531
  • Filtration of endpoints by operationId via operation_id parameter to schema.parametrize or -O command-line option. #546
  • Generation of security-related parameters. They are taken from securityDefinitions / securitySchemes and injected to the generated data. It supports generating API keys in headers or query parameters and generating data for HTTP authentication schemes. #540

Fixed

  • Overriding header values in CLI and runner when headers provided explicitly clash with ones defined in the schema. #559
  • Nested references resolving in response_schema_conformance check. #562
  • Nullable parameters handling when they are behind a reference. #542

1.4.0 - 2020-05-03

Added

  • context argument for hook functions to provide an additional context for hooks. A deprecation warning is emitted for hook functions that do not accept this argument.
  • A new hook system that allows generic hook dispatching. It comes with new hook locations. For more details, see the "Customization" section in our documentation.
  • New before_process_path hook.
  • Third-party compatibility fixups mechanism. Currently, there is one fixup for FastAPI. #503

Deprecated

  • Hook functions that do not accept context as their first argument. They will become not be supported in Schemathesis 2.0.
  • Registering hooks by name and function. Use register decorators instead. For more details, see the "Customization" section in our documentation.
  • BaseSchema.with_hook and BaseSchema.register_hook. Use BaseSchema.hooks.apply and BaseSchema.hooks.register instead.

Fixed

  • Add missing validate_schema argument to loaders.from_pytest_fixture.
  • Reference resolving during response schema conformance check. #539

1.3.4 - 2020-04-30

Fixed

  • Validation of nullable properties in response_schema_conformance check introduced in 1.3.0. #542

1.3.3 - 2020-04-29

Changed

  • Update pytest-subtests pin to >=0.2.1,<1.0. #537

1.3.2 - 2020-04-27

Added

  • Show exceptions if they happened during loading a WSGI application. Option --show-errors-tracebacks will display a full traceback.

1.3.1 - 2020-04-27

Fixed

  • Packaging issue

1.3.0 - 2020-04-27

Added

  • Storing network logs with --store-network-log=<filename.yaml>. The stored cassettes are based on the VCR format and contain extra information from the Schemathesis internals. #379
  • Replaying of cassettes stored in VCR format. #519
  • Targeted property-based testing in CLI and runner. It only supports the response_time target at the moment. #104
  • Export CLI test results to JUnit.xml with --junit-xml=<filename.xml>. #427

Fixed

  • Code samples for schemas where body is defined as {"type": "string"}. #521
  • Showing error causes on internal jsonschema errors during input schema validation. #513
  • Recursion error in response_schema_conformance check. Because of this change, Endpoint.definition contains a definition where references are not resolved. In this way, it makes it possible to avoid recursion errors in jsonschema validation. #468

Changed

  • Added indentation & section name to the SUMMARY CLI block.
  • Use C-extension for YAML loading when it is possible. It can cause more than 10x speedup on schema parsing. Do not show Click's "Aborted!" message when an error occurs during CLI schema loading.
  • Add a help message to the CLI output when an internal exception happens. #529

1.2.0 - 2020-04-15

Added

  • Per-test hooks for modification of data generation strategies. #492
  • Support for x-example vendor extension in Open API 2.0. #504
  • Sanity validation for the input schema & loader in runner.prepare. #499

1.1.2 - 2020-04-14

Fixed

  • Support for custom loaders in runner. Now all built-in loaders are supported as an argument to runner.prepare. #496
  • from_wsgi loader accepts custom keyword arguments that will be passed to client.get when accessing the schema. #497

1.1.1 - 2020-04-12

Fixed

  • Mistakenly applied Open API -> JSON Schema Draft 7 conversion. It should be Draft 4. #489
  • Using wrong validator in response_schema_conformance check. It should be Draft 4 validator. #468

1.1.0 - 2020-04-08

Fixed

  • Response schema check for recursive schemas. #468

Changed

  • App loading in runner. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute a runner in a subprocess. Otherwise, apps can't be easily serialized and transferred into another process.
  • Runner events structure. All data in events is static from now. There are no references to BaseSchema, Endpoint or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object, which decouples any runner consumer from implementation details. It will help make runner usable in more cases (e.g., web application) since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with the InternalError event that should be handled accordingly.

1.0.5 - 2020-04-03

Fixed

  • Open API 3. Handling of endpoints that contain multipart/form-data media types. Previously only file upload endpoints were working correctly. #473

1.0.4 - 2020-04-03

Fixed

  • OpenApi30.get_content_types behavior, introduced in 8aeee1a. #469

1.0.3 - 2020-04-03

Fixed

  • Precedence of produces keywords for Swagger 2.0 schemas. Now, operation-level produces overrides schema-level produces as specified in the specification. #463
  • Content-type conformance check for Open API 3.0 schemas. #461
  • Pytest 5.4 warning for test functions without parametrization. #451

1.0.2 - 2020-04-02

Fixed

  • Handling of fields in paths that are not operations, but allowed by the Open API spec. #457
  • Pytest 5.4 warning about deprecated Node initialization usage. #451

1.0.1 - 2020-04-01

Fixed

  • Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g. path) contain example value. They are properly combined now. #450

1.0.0 - 2020-03-31

Changed

  • Move processing of runner parameters to runner.prepare. This change will provide better code reuse since all users of runner (e.g., if you extended it in your project) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn't use runner directly, then this change should not have a visible effect on your use-case.

0.28.0 - 2020-03-31

Fixed

  • Handling of schemas that use x-* custom properties. #448

Removed

  • Deprecated runner.execute. Use runner.prepare instead.

0.27.0 - 2020-03-31

Deprecated

  • runner.execute should not be used, since runner.prepare provides a more flexible interface to test execution.

Removed

  • Deprecated Parametrizer class. Use schemathesis.from_path as a replacement for Parametrizer.from_path.

0.26.1 - 2020-03-24

Fixed

  • Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435

0.26.0 - 2020-03-19

Fixed

  • Filter problematic path template variables containing "/", or "%2F" url encoded. #440
  • Filter invalid empty "" path template variables. #439
  • Typo in a help message in the CLI output. #436

0.25.1 - 2020-03-09

Changed

  • Allow werkzeug >= 1.0.0. #433

0.25.0 - 2020-02-27

Changed

  • Handling of explicit examples from schemas. Now, if there are examples for multiple locations (e.g., for body and query) then they will be combined into a single example. #424

0.24.5 - 2020-02-26

Fixed

  • Error during pytest collection on objects with custom __getattr__ method and therefore pass is_schemathesis check. #429

0.24.4 - 2020-02-22

Fixed

  • Resolving references when the schema is loaded from a file on Windows. #418

0.24.3 - 2020-02-10

Fixed

  • Not copied validate_schema parameter in BaseSchema.parametrize. Regression after implementing #383
  • Missing app, location and hooks parameters in schema when used with BaseSchema.parametrize. #416

0.24.2 - 2020-02-09

Fixed

  • Crash on invalid regular expressions in method, endpoint and tag CLI options. #403
  • Crash on a non-latin-1 encodable value in the auth CLI option. #404
  • Crash on an invalid value in the header CLI option. #405
  • Crash on some invalid URLs in the schema CLI option. #406
  • Validation of --request-timeout parameter. #407
  • Crash with --hypothesis-deadline=0 CLI option. #410
  • Crash with --hypothesis-max-examples=0 CLI option. #412

0.24.1 - 2020-02-08

Fixed

  • CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400

0.24.0 - 2020-02-07

Added

  • Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
  • --show-error-tracebacks CLI option to display errors' tracebacks in the output. #391
  • Support for schema behind auth. #115

Changed

  • Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via --validate-schema=false for example). The use-case is for tools like ElasticSearch that use GET requests with non-empty bodies. #383

Fixed

  • CLI crash when an explicit example is specified in the endpoint definition. #386

0.23.7 - 2020-01-30

Added

  • -x/--exitfirst CLI option to exit after the first failed test. #378

Fixed

  • Handling examples of parameters in Open API 3.0. #381

0.23.6 - 2020-01-28

Added

  • all variant for --checks CLI option to use all available checks. #374

Changed

  • Use built-in importlib.metadata on Python 3.8. #376

0.23.5 - 2020-01-24

Fixed

  • Generation of invalid values in Case.cookies. #371

0.23.4 - 2020-01-22

Fixed

  • Converting exclusiveMinimum & exclusiveMaximum fields to JSON Schema. #367

0.23.3 - 2020-01-21

Fixed

  • Filter out surrogate pairs from the query string.

0.23.2 - 2020-01-16

Fixed

  • Prevent KeyError when the response does not have the "Content-Type" header. #365

0.23.1 - 2020-01-15

Fixed

  • Dockerfile entrypoint was not working as per docs. #361

0.23.0 - 2020-01-15

Added

  • Hooks for strategy modification. #313
  • Input schema validation. Use --validate-schema=false to disable it in CLI and validate_schema=False argument in loaders. #110

0.22.0 - 2020-01-11

Added

  • Show multiple found failures in the CLI output. #266 & #207
  • Raise a proper exception when the given schema is invalid. #308
  • Support for None as a value for --hypothesis-deadline. #349

Fixed

  • Handling binary request payloads in Case.call. #350
  • Type of the second argument to all built-in checks set to proper Case instead of TestResult. The error was didn't affect built-in checks since both Case and TestResult had endpoint attribute, and only it was used. However, this fix is not backward-compatible with 3rd party checks.

0.21.0 - 2019-12-20

Added

  • Support for AioHTTP applications in CLI. #329

0.20.5 - 2019-12-18

Fixed

  • Compatibility with the latest release of hypothesis-jsonschema and setting its minimal required version to 0.9.13. #338

0.20.4 - 2019-12-17

Fixed

  • Handling nullable attribute in Open API schemas. #335

0.20.3 - 2019-12-17

Fixed

  • Usage of the response status code conformance check with old requests version. #330

0.20.2 - 2019-12-14

Fixed

  • Response schema conformance check for Open API 3.0. #332

0.20.1 - 2019-12-13

Added

  • Support for response code ranges. #330

0.20.0 - 2019-12-12

Added

  • WSGI apps support. #31
  • Case.validate_response for running built-in checks against app's response. #319

Changed

  • Checks receive Case instance as a second argument instead of TestResult. This was done for making checks usable in Python tests via Case.validate_response. Endpoint and schema are accessible via case.endpoint and case.endpoint.schema.

0.19.1 - 2019-12-11

Fixed

  • Compatibility with Hypothesis >= 4.53.2. #322

0.19.0 - 2019-12-02

Added

  • Concurrent test execution in CLI / runner. #91
  • update importlib_metadata pin to ^1.1. #315

0.18.1 - 2019-11-28

Fixed

  • Validation of the base-url CLI parameter. #311

0.18.0 - 2019-11-27

Added

  • Resolving references in PathItem objects. #301

Fixed

  • Resolving of relative paths in schemas. #303
  • Loading string dates as datetime.date objects in YAML loader. #305

0.17.0 - 2019-11-21

Added

  • Resolving references that point to different files. #294

Changed

  • Keyboard interrupt is now handled during the CLI run, and the summary is displayed in the output. #295

0.16.0 - 2019-11-19

Added

  • Display RNG seed in the CLI output to allow test reproducing. #267
  • Allow specifying seed in CLI.
  • Ability to pass custom kwargs to the requests.get call in loaders.from_uri.

Changed

  • Refactor case generation strategies: strategy is not used to generate empty value. #253
  • Improved error message for invalid path parameter declaration. #255

Fixed

  • Pytest fixture parametrization via pytest_generate_tests. #280
  • Support for tests defined as methods. #282
  • Unclosed requests.Session on calling Case.call without passing a session explicitly. #286

0.15.0 - 2019-11-15

Added

  • Support for OpenAPI 3.0 server variables (base_path). #40
  • Support for format: byte. #254
  • Response schema conformance check in CLI / Runner. #256
  • Docker image for CLI. #268
  • Pre-run hooks for CLI. #147
  • A way to register custom checks for CLI via schemathesis.register_check. #270

Fixed

  • Not encoded path parameters. #272

Changed

  • Verbose messages are displayed in the CLI on failed checks. #261

0.14.0 - 2019-11-09

Added

  • CLI: Support file paths in the schema argument. #119
  • Checks to verify response status & content type in CLI / Runner. #101

Fixed

  • Custom base URL handling in CLI / Runner. #248

Changed

  • Raise an error if the schema has a body for GET requests. #218
  • Method names are case insensitive during direct schema access. #246

0.13.2 - 2019-11-05

Fixed

  • IndexError when Hypothesis found inconsistent test results during the test execution in the runner. #236

0.13.1 - 2019-11-05

Added

  • Support for binary format #197

Fixed

  • Error that happens when there are no success checks in the statistic in CLI. #237

0.13.0 - 2019-11-03

Added

  • An option to configure request timeout for CLI / Runner. #204
  • A help snippet to reproduce errors caught by Schemathesis. #206
  • Total running time to the CLI output. #181
  • Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
  • Extra information to the CLI output: schema address, spec version, and base URL. #188

Fixed

  • Compatibility with Hypothesis 4.42.4+ . #212
  • Display flaky errors only in the "ERRORS" section and improve CLI output. #215
  • Handling formData parameters in Case.call. #196
  • Handling cookies in Case.call. #211

Changed

  • More readable falsifying examples output. #127
  • Show exceptions in a separate section of the CLI output. #203
  • Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
  • Do not stop on schema errors related to a single endpoint. #139
  • Display a proper error message when the schema is not available in CLI / Runner. #214

0.12.2 - 2019-10-30

Fixed

  • Wrong handling of the base_url parameter in runner and Case.call if it has a trailing slash. #194 and #199
  • Do not send any payload with GET requests. #200

0.12.1 - 2019-10-28

Fixed

  • Handling for errors other than AssertionError and HypothesisException in the runner. #189
  • CLI failing on the case when there are tests, but no checks were performed. #191

Changed

  • Display the "SUMMARY" section in the CLI output for empty test suites.

0.12.0 - 2019-10-28

Added

  • Display progress during the CLI run. #125

Fixed

  • Test server-generated wrong schema when the endpoints option is passed via CLI. #173
  • Error message if the schema is not found in CLI. #172

Changed

  • Continue running tests on hypothesis error. #137

0.11.0 - 2019-10-22

Added

  • LazySchema accepts filters. #149
  • Ability to register strategies for custom string formats. #94
  • Generator-based events in the runner module to improve control over the execution flow.
  • Filtration by tags. #134

Changed

  • Base URL in schema instances could be reused when it is defined during creation. Now on, the base_url argument in Case.call is optional in such cases. #153
  • Hypothesis deadline is set to 500ms by default. #138
  • Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
  • Disallow empty username in CLI --auth option.

Fixed

  • User-agent during schema loading. #144
  • Generation of invalid values in Case.headers. #167

Removed

  • Undocumented support for file:// URI schema

0.10.0 - 2019-10-14

Added

  • HTTP Digest Auth support. #106
  • Support for Hypothesis settings in CLI & Runner. #107
  • Case.call and Case.as_requests_kwargs convenience methods. #109
  • Local development server. #126

Removed

  • Autogenerated runner.StatsCollector.__repr__ to make Hypothesis output more readable.

0.9.0 - 2019-10-09

Added

  • Test executor collects results of execution. #29
  • CLI option --base-url for specifying base URL of API. #118
  • Support for coroutine-based tests. #121
  • User Agent to network requests in CLI & runner. #130

Changed

  • CLI command schemathesis run prints result in a more readable way with a summary of passing checks.
  • Empty header names are forbidden for CLI.
  • Suppressed hypothesis exception about using example non-interactively. #92

0.8.1 - 2019-10-04

Fixed

  • Wrap each test in suppress so the runner doesn't stop after the first test failure.

0.8.0 - 2019-10-04

Added

  • CLI tool invoked by the schemathesis command. #30
  • New arguments api_options, loader_options and loader for test executor. #90
  • A mapping interface for schemas & convenience methods for direct strategy access. #98

Fixed

  • Runner stopping on the first falsifying example. #99

0.7.3 - 2019-09-30

Fixed

  • Filtration in lazy loaders.

0.7.2 - 2019-09-30

Added

  • Support for type "file" for Swagger 2.0. #78
  • Support for filtering in loaders. #75

Fixed

  • Conflict for lazy schema filtering. #64

0.7.1 - 2019-09-27

Added

  • Support for x-nullable extension. #45

0.7.0 - 2019-09-26

Added

  • Support for the cookie parameter in OpenAPI 3.0 schemas. #21
  • Support for the formData parameter in Swagger 2.0 schemas. #6
  • Test executor. #28

Fixed

  • Using hypothesis.settings decorator with test functions created from from_pytest_fixture loader. #69

0.6.0 - 2019-09-24

Added

  • Parametrizing tests from a pytest fixture via pytest-subtests. #58

Changed

  • Rename module readers to loaders.
  • Rename parametrize parameters. filter_endpoint to endpoint and filter_method to method.

Removed

  • Substring match for method/endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.

0.5.0 - 2019-09-16

Added

  • Generating explicit examples from the schema. #17

Changed

  • Schemas are loaded eagerly from now on. Using schemathesis.from_uri implies network calls.

Deprecated

  • Using Parametrizer.from_{path,uri} is deprecated, use schemathesis.from_{path,uri} instead.

Fixed

  • Body resolving during test collection. #55

0.4.1 - 2019-09-11

Fixed

  • Possibly unhandled exception during hasattr check in is_schemathesis_test.

0.4.0 - 2019-09-10

Fixed

  • Resolving all inner references in objects. #34

Changed

  • jsonschema.RefResolver is now used for reference resolving. #35

0.3.0 - 2019-09-06

Added

  • Parametrizer.from_uri method to construct parametrizer instances from URIs. #24

Removed

  • Possibility to use Parametrizer.parametrize and custom Parametrizer kwargs for passing config options to hypothesis.settings. Use hypothesis.settings decorators on tests instead.

0.2.0 - 2019-09-05

Added

  • Open API 3.0 support. #10
  • "header" parameters. #7

Changed

  • Handle errors during collection / executions as failures.
  • Use re.search for pattern matching in filter_method/filter_endpoint instead of fnmatch. #18
  • Case.body contains properties from the target schema, without the extra level of nesting.

Fixed

  • KeyError on collection when "basePath" is absent. #16

0.1.0 - 2019-06-28

  • Initial public release