changelog.rst

Changelog

Unreleased

Fixed

• Support for custom loaders in runner. Now all built-in loaders are supported as an argument to runner.prepare. #496
• from_wsgi loader accept custom keyword arguments that will be passed to client.get when accessing the schema. #497

1.1.1 - 2020-04-12

Fixed

• Mistakenly applied Open API -> JSON Schema Draft 7 conversion. It should be Draft 4. #489
• Using wrong validator in response_schema_conformance check. It should be Draft 4 validator. #468

1.1.0 - 2020-04-08

Fixed

• Response schema check for recursive schemas. #468

Changed

• App loading in runner. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute runner in a subprocess. Otherwise apps can't be easily serialized and transferred into another process.
• Runner events structure. All data in events is static from now, there are no references to BaseSchema, Endpoint or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object which decouples any runner consumer from implementation details and will help make runner usable in more cases (e.g. web application), since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with InternalError event that should be handled accordingly.

1.0.5 - 2020-04-03

Fixed

• Open API 3. Handling of endpoints that contain multipart/form-data media types. Previously only file upload endpoints were working correctly. #473

1.0.4 - 2020-04-03

Fixed

• OpenApi30.get_content_types behavior, introduced in 8aeee1a. #469

1.0.3 - 2020-04-03

Fixed

• Precedence of produces keywords for Swagger 2.0 schemas. Now, operation-level produces overrides schema-level produces as specified in the specification. #463
• Content Type conformance check for Open API 3.0 schemas. #461
• Pytest 5.4 warning for test functions without parametrization. #451

1.0.2 - 2020-04-02

Fixed

• Handling of fields in paths that are not operations, but allowed by the Open API spec. #457
• Pytest 5.4 warning about deprecated Node initialization usage. #451

1.0.1 - 2020-04-01

Fixed

• Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g. path) contain example value. They are properly combined now. #450

1.0.0 - 2020-03-31

Changed

• Move processing of runner parameters to runner.prepare. This change will provide better code reusage, since all users of runner (e.g. if you extended it in your project`) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn't use runner directly, then this change should not have a visible effect for your use-case.

0.28.0 - 2020-03-31

Fixed

• Handling of schemas, that use x-* custom properties. #448

Removed

• Deprecated runner.execute. Use runner.prepare instead.

0.27.0 - 2020-03-31

Deprecated

• runner.execute should not be used, since runner.prepare provides a more flexible interface to test execution.

Removed

• Deprecated Parametrizer class. Use schemathesis.from_path as a replacement for Parametrizer.from_path.

0.26.1 - 2020-03-24

Fixed

• Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435

0.26.0 - 2020-03-19

Fixed

• Filter problematic path template variables containing "/", or "%2F" url encoded. #440
• Filter invalid empty "" path template variables. #439
• Typo in a help message in the CLI output. #436

0.25.1 - 2020-03-09

Changed

• Allow werkzeug >= 1.0.0. #433

0.25.0 - 2020-02-27

Changed

• Handling of explicit examples from schemas. Now if there are examples for multiple locations (e.g. for body and for query) then they will be combined into a single example. #424

0.24.5 - 2020-02-26

Fixed

• Error during pytest collection on objects that have custom __getattr__ method and therefore pass is_schemathesis check. #429

0.24.4 - 2020-02-22

Fixed

• Resolving references when schema is loaded from a file on Windows. #418

0.24.3 - 2020-02-10

Fixed

• Not copied validate_schema parameter in BaseSchema.parametrize. Regression after implementing #383
• Missing app, location and hooks parameters in schema when used with BaseSchema.parametrize. #416

0.24.2 - 2020-02-09

Fixed

• Crash on invalid regular expressions in method, endpoint and tag CLI options. #403
• Crash on non latin-1 encodable value in auth CLI option. #404
• Crash on invalid value in header CLI options. #405
• Crash on some invalid URLs in schema CLI option. #406
• Validation of --request-timeout parameter. #407
• Crash with --hypothesis-deadline=0 CLI option. #410
• Crash with --hypothesis-max-examples=0 CLI option. #412

0.24.1 - 2020-02-08

Fixed

• CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400

0.24.0 - 2020-02-07

Added

• Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
• --show-error-tracebacks CLI option to display errors' tracebacks in the output. #391
• Support for schema behind auth. #115

Changed

• Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via --validate-schema=false for example). The usecase is for tools like ElasticSearch that use GET requests with non empty bodies. #383

Fixed

• CLI crash when an explicit example is specified in endpoint definition. #386

0.23.7 - 2020-01-30

Added

• -x/--exitfirst CLI option to exit after first failed test. #378

Fixed

• Handling examples of parameters in Open API 3.0. #381

0.23.6 - 2020-01-28

Added

• all variant for --checks CLI option to use all available checks. #374

Changed

• Use built-in importlib.metadata on Python 3.8. #376

0.23.5 - 2020-01-24

Fixed

• Generation of invalid values in Case.cookies. #371

0.23.4 - 2020-01-22

Fixed

• Converting exclusiveMinimum & exclusiveMaximum fields to JSON Schema. #367

0.23.3 - 2020-01-21

Fixed

• Filter out surrogate pairs from query string.

0.23.2 - 2020-01-16

Fixed

• Prevent KeyError when response does not have Content-Type. #365

0.23.1 - 2020-01-15

Fixed

• Dockerfile entrypoint was not working as per docs. #361

0.23.0 - 2020-01-15

Added

• Hooks for strategy modification. #313
• Input schema validation. Use --validate-schema=false to disable it in CLI and validate_schema=False argument in loaders. #110

0.22.0 - 2020-01-11

Added

• Show multiple found failures in the CLI output. #266 & #207
• Raise proper exception when the given schema is invalid. #308
• Support for None as a value for --hypothesis-deadline. #349

Fixed

• Handling binary request payloads in Case.call. #350
• Type of the second argument to all built-in checks set to proper Case instead of TestResult. The error was didn't affect built-in checks since both Case and TestResult had endpoint attribute and only it was used. However this fix is not backward-compatible with 3rd party checks.

0.21.0 - 2019-12-20

Added

• Support for AioHTTP applications in CLI. #329

0.20.5 - 2019-12-18

Fixed

• Compatibility with the latest release of hypothesis-jsonschema and setting its minimal required version to 0.9.13. #338

0.20.4 - 2019-12-17

Fixed

• Handling nullable attribute in Open API schemas. #335

0.20.3 - 2019-12-17

Fixed

• Response status code conformance check applicability for old requests version. #330

0.20.2 - 2019-12-14

Fixed

• Response schema conformance check for Open API 3.0. #332

0.20.1 - 2019-12-13

Added

• Support for response code ranges. #330

0.20.0 - 2019-12-12

Added

• WSGI apps support. #31
• Case.validate_response for running built-in checks against app's response. #319

Changed

• Checks receive Case instance as a second argument instead of TestResult. This was done for making checks usable in Python tests via Case.validate_response. Endpoint and schema are accessible via case.endpoint and case.endpoint.schema.

0.19.1 - 2019-12-11

Fixed

• Compatibility with Hypothesis >= 4.53.2. #322

0.19.0 - 2019-12-02

Added

• Concurrent test execution in CLI / runner. #91
• update importlib_metadata pin to ^1.1. #315

0.18.1 - 2019-11-28

Fixed

• Validation of base-url CLI parameter. #311

0.18.0 - 2019-11-27

Added

• Resolving references in PathItem objects. #301

Fixed

• Resolving of relative paths in schemas. #303
• Loading string dates as datetime.date objects in YAML loader. #305

0.17.0 - 2019-11-21

Added

• Resolving references that point to different files. #294

Changed

• Keyboard interrupt is now handled during the CLI run and the summary is displayed in the output. #295

0.16.0 - 2019-11-19

Added

• Display RNG seed in the CLI output to allow test reproducing. #267
• Allow to specify seed in CLI.
• Ability to pass custom kwargs to the requests.get call in loaders.from_uri.

Changed

• Refactor case generation strategies: strategy is not used to generate empty value. #253
• Improved error message for invalid path parameter declaration. #255

Fixed

• Pytest fixture parametrization via pytest_generate_tests. #280
• Support for tests defined as methods. #282
• Unclosed requests.Session on calling Case.call without passing a session explicitly. #286

0.15.0 - 2019-11-15

Added

• Support for OpenAPI 3.0 server variables (base_path). #40
• Support for format: byte. #254
• Response schema conformance check in CLI / Runner. #256
• Docker image for CLI. #268
• Pre-run hooks for CLI. #147
• A way to register custom checks for CLI via schemathesis.register_check. #270

Fixed

• Not encoded path parameters. #272

Changed

• Verbose messages are displayed in the CLI on failed checks. #261

0.14.0 - 2019-11-09

Added

• CLI: Support file paths in schema argument. #119
• Checks to verify response status & content type in CLI / Runner. #101

Fixed

• Custom base URL handling in CLI / Runner. #248

Changed

• Raise an error if schema has body for GET requests. #218
• Method names are case insensitive during direct schema access. #246

0.13.2 - 2019-11-05

Fixed

• IndexError when Hypothesis found inconsistent test results during the test execution in runner. #236

0.13.1 - 2019-11-05

Added

• Support for binary format #197

Fixed

• Error that happens when there are no success checks in the statistic in CLI. #237

0.13.0 - 2019-11-03

Added

• An option to configure request timeout for CLI / Runner. #204
• A help snippet to reproduce errors caught by Schemathesis. #206
• Total running time to the CLI output. #181
• Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
• Extra information to the CLI output: schema address, spec version and base url. #188

Fixed

• Compatibility with Hypothesis 4.42.4+ . #212
• Display flaky errors only in the "ERRORS" section and improve CLI output. #215
• Handling formData parameters in Case.call. #196
• Handling cookies in Case.call. #211

Changed

• More readable falsifying examples output. #127
• Show exceptions in a separate section of the CLI output. #203
• Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
• Do not stop on schema errors related to single endpoint. #139
• Display a proper error message when schema is not available in CLI / Runner. #214

0.12.2 - 2019-10-30

Fixed

• Wrong handling of base_url parameter in runner and Case.call if it has a trailing slash. #194 and #199
• Do not send any payload with GET requests. #200

0.12.1 - 2019-10-28

Fixed

• Handling for errors other than AssertionError and HypothesisException in the runner. #189
• CLI failing on the case when there are tests, but no checks were performed. #191

Changed

• Display "SUMMARY" section in the CLI output for empty test suites.

0.12.0 - 2019-10-28

Added

• Display progress during the CLI run. #125

Fixed

• Test server generated wrong schema when endpoints option is passed via CLI. #173
• Error message if schema is not found in CLI. #172

Changed

• Continue running tests on hypothesis error. #137

0.11.0 - 2019-10-22

Added

• LazySchema accepts filters. #149
• Ability to register strategies for custom string formats. #94
• Generator-based events in runner module to improve control over the execution flow.
• Filtration by tags. #134

Changed

• Base URL in schema instances could be reused when it is defined during creation. Now on, base_url argument in Case.call is optional in such cases. #153
• Hypothesis deadline is set to 500ms by default. #138
• Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
• Disallow empty username in CLI --auth option.

Fixed

• User agent during schema loading. #144
• Generation of invalid values in Case.headers. #167

Removed

• Undocumented support for file:// uri schema

0.10.0 - 2019-10-14

Added

• HTTP Digest Auth support. #106
• Support for Hypothesis settings in CLI & Runner. #107
• Case.call and Case.as_requests_kwargs convenience methods. #109
• Local development server. #126

Removed

• Autogenerated runner.StatsCollector.__repr__ to make Hypothesis output more readable.

0.9.0 - 2019-10-09

Added

• Test executor collects results of execution. #29
• CLI option --base-url for specifying base URL of API. #118
• Support for coroutine-based tests. #121
• User Agent to network requests in CLI & runner. #130

Changed

• CLI command schemathesis run prints results in a more readable way with a summary of passing checks.
• Empty header names are forbidden for CLI.
• Suppressed hypothesis exception about using example non-interactively. #92

0.8.1 - 2019-10-04

Fixed

• Wrap each individual test in suppress so the runner doesn't stop after the first test failure.

0.8.0 - 2019-10-04

Added

• CLI tool invoked by the schemathesis command. #30
• New arguments api_options, loader_options and loader for test executor. #90
• A mapping interface for schemas & convenience methods for direct strategies access. #98

Fixed

• Runner stopping on the first falsifying example. #99

0.7.3 - 2019-09-30

Fixed

• Filtration in lazy loaders.

0.7.2 - 2019-09-30

Added

• Support for type "file" for Swagger 2.0. #78
• Support for filtering in loaders. #75

Fixed

• Conflict for lazy schema filtering. #64

0.7.1 - 2019-09-27

Added

• Support for x-nullable extension. #45

0.7.0 - 2019-09-26

Added

• Support for cookie parameter in OpenAPI 3.0 schemas. #21
• Support for formData parameter in Swagger 2.0 schemas. #6
• Test executor. #28

Fixed

• Using hypothesis.settings decorator with test functions created from from_pytest_fixture loader. #69

0.6.0 - 2019-09-24

Added

• Parametrizing tests from a pytest fixture via pytest-subtests. #58

Changed

• Rename module readers to loaders.
• Rename parametrize parameters. filter_endpoint to endpoint and filter_method to method.

Removed

• Substring match for method / endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.

0.5.0 - 2019-09-16

Added

• Generating explicit examples from schema. #17

Changed

• Schemas are loaded eagerly from now on. Using schemathesis.from_uri implies network calls.

Deprecated

• Using Parametrizer.from_{path,uri} is deprecated, use schemathesis.from_{path,uri} instead.

Fixed

• Body resolving during test collection. #55

0.4.1 - 2019-09-11

Fixed

• Possibly unhandled exception during hasattr check in is_schemathesis_test.

0.4.0 - 2019-09-10

Fixed

• Resolving all inner references in objects. #34

Changed

• jsonschema.RefResolver is now used for reference resolving. #35

0.3.0 - 2019-09-06

Added

• Parametrizer.from_uri method to construct parametrizer instances from URIs. #24

Removed

• Possibility to use Parametrizer.parametrize and custom Parametrizer kwargs for passing config options to hypothesis.settings. Use hypothesis.settings decorators on tests instead.

0.2.0 - 2019-09-05

Added

• Open API 3.0 support. #10
• "header" parameters. #7

Changed

• Handle errors during collection / executions as failures.
• Use re.search for pattern matching in filter_method/filter_endpoint instead of fnmatch. #18
• Case.body contains properties from the target schema, without extra level of nesting.

Fixed

• KeyError on collection when "basePath" is absent. #16

0.1.0 - 2019-06-28

• Initial public release