changelog.rst

Changelog

Unreleased

0.24.3 - 2020-02-10

Fixed

• Not copied validate_schema parameter in BaseSchema.parametrize. Regression after implementing #383
• Missing app, location and hooks parameters in schema when used with BaseSchema.parametrize. #416

0.24.2 - 2020-02-09

Fixed

• Crash on invalid regular expressions in method, endpoint and tag CLI options. #403
• Crash on non latin-1 encodable value in auth CLI option. #404
• Crash on invalid value in header CLI options. #405
• Crash on some invalid URLs in schema CLI option. #406
• Validation of --request-timeout parameter. #407
• Crash with --hypothesis-deadline=0 CLI option. #410
• Crash with --hypothesis-max-examples=0 CLI option. #412

0.24.1 - 2020-02-08

Fixed

• CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400

0.24.0 - 2020-02-07

Added

• Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
• --show-error-tracebacks CLI option to display errors' tracebacks in the output. #391
• Support for schema behind auth. #115

Changed

• Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via --validate-schema=false for example). The usecase is for tools like ElasticSearch that use GET requests with non empty bodies. #383

Fixed

• CLI crash when an explicit example is specified in endpoint definition. #386

0.23.7 - 2020-01-30

Added

• -x/--exitfirst CLI option to exit after first failed test. #378

Fixed

• Handling examples of parameters in Open API 3.0. #381

0.23.6 - 2020-01-28

Added

• all variant for --checks CLI option to use all available checks. #374

Changed

• Use built-in importlib.metadata on Python 3.8. #376

0.23.5 - 2020-01-24

Fixed

• Generation of invalid values in Case.cookies. #371

0.23.4 - 2020-01-22

Fixed

• Converting exclusiveMinimum & exclusiveMaximum fields to JSON Schema. #367

0.23.3 - 2020-01-21

Fixed

• Filter out surrogate pairs from query string.

0.23.2 - 2020-01-16

Fixed

• Prevent KeyError when response does not have Content-Type. #365

0.23.1 - 2020-01-15

Fixed

• Dockerfile entrypoint was not working as per docs. #361

0.23.0 - 2020-01-15

Added

• Hooks for strategy modification. #313
• Input schema validation. Use --validate-schema=false to disable it in CLI and validate_schema=False argument in loaders. #110

0.22.0 - 2020-01-11

Added

• Show multiple found failures in the CLI output. #266 & #207
• Raise proper exception when the given schema is invalid. #308
• Support for None as a value for --hypothesis-deadline. #349

Fixed

• Handling binary request payloads in Case.call. #350
• Type of the second argument to all built-in checks set to proper Case instead of TestResult. The error was didn't affect built-in checks since both Case and TestResult had endpoint attribute and only it was used. However this fix is not backward-compatible with 3rd party checks.

0.21.0 - 2019-12-20

Added

• Support for AioHTTP applications in CLI. #329

0.20.5 - 2019-12-18

Fixed

• Compatibility with the latest release of hypothesis-jsonschema and setting its minimal required version to 0.9.13. #338

0.20.4 - 2019-12-17

Fixed

• Handling nullable attribute in Open API schemas. #335

0.20.3 - 2019-12-17

Fixed

• Response status code conformance check applicability for old requests version. #330

0.20.2 - 2019-12-14

Fixed

• Response schema conformance check for Open API 3.0. #332

0.20.1 - 2019-12-13

Added

• Support for response code ranges. #330

0.20.0 - 2019-12-12

Added

• WSGI apps support. #31
• Case.validate_response for running built-in checks against app's response. #319

Changed

• Checks receive Case instance as a second argument instead of TestResult. This was done for making checks usable in Python tests via Case.validate_response. Endpoint and schema are accessible via case.endpoint and case.endpoint.schema.

0.19.1 - 2019-12-11

Fixed

• Compatibility with Hypothesis >= 4.53.2. #322

0.19.0 - 2019-12-02

Added

• Concurrent test execution in CLI / runner. #91
• update importlib_metadata pin to ^1.1. #315

0.18.1 - 2019-11-28

Fixed

• Validation of base-url CLI parameter. #311

0.18.0 - 2019-11-27

Added

• Resolving references in PathItem objects. #301

Fixed

• Resolving of relative paths in schemas. #303
• Loading string dates as datetime.date objects in YAML loader. #305

0.17.0 - 2019-11-21

Added

• Resolving references that point to different files. #294

Changed

• Keyboard interrupt is now handled during the CLI run and the summary is displayed in the output. #295

0.16.0 - 2019-11-19

Added

• Display RNG seed in the CLI output to allow test reproducing. #267
• Allow to specify seed in CLI.
• Ability to pass custom kwargs to the requests.get call in loaders.from_uri.

Changed

• Refactor case generation strategies: strategy is not used to generate empty value. #253
• Improved error message for invalid path parameter declaration. #255

Fixed

• Pytest fixture parametrization via pytest_generate_tests. #280
• Support for tests defined as methods. #282
• Unclosed requests.Session on calling Case.call without passing a session explicitly. #286

0.15.0 - 2019-11-15

Added

• Support for OpenAPI 3.0 server variables (base_path). #40
• Support for format: byte. #254
• Response schema conformance check in CLI / Runner. #256
• Docker image for CLI. #268
• Pre-run hooks for CLI. #147
• A way to register custom checks for CLI via schemathesis.register_check. #270

Fixed

• Not encoded path parameters. #272

Changed

• Verbose messages are displayed in the CLI on failed checks. #261

0.14.0 - 2019-11-09

Added

• CLI: Support file paths in schema argument. #119
• Checks to verify response status & content type in CLI / Runner. #101

Fixed

• Custom base URL handling in CLI / Runner. #248

Changed

• Raise an error if schema has body for GET requests. #218
• Method names are case insensitive during direct schema access. #246

0.13.2 - 2019-11-05

Fixed

• IndexError when Hypothesis found inconsistent test results during the test execution in runner. #236

0.13.1 - 2019-11-05

Added

• Support for binary format #197

Fixed

• Error that happens when there are no success checks in the statistic in CLI. #237

0.13.0 - 2019-11-03

Added

• An option to configure request timeout for CLI / Runner. #204
• A help snippet to reproduce errors caught by Schemathesis. #206
• Total running time to the CLI output. #181
• Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
• Extra information to the CLI output: schema address, spec version and base url. #188

Fixed

• Compatibility with Hypothesis 4.42.4+ . #212
• Display flaky errors only in the "ERRORS" section and improve CLI output. #215
• Handling formData parameters in Case.call. #196
• Handling cookies in Case.call. #211

Changed

• More readable falsifying examples output. #127
• Show exceptions in a separate section of the CLI output. #203
• Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
• Do not stop on schema errors related to single endpoint. #139
• Display a proper error message when schema is not available in CLI / Runner. #214

0.12.2 - 2019-10-30

Fixed

• Wrong handling of base_url parameter in runner and Case.call if it has a trailing slash. #194 and #199
• Do not send any payload with GET requests. #200

0.12.1 - 2019-10-28

Fixed

• Handling for errors other than AssertionError and HypothesisException in the runner. #189
• CLI failing on the case when there are tests, but no checks were performed. #191

Changed

• Display "SUMMARY" section in the CLI output for empty test suites.

0.12.0 - 2019-10-28

Added

• Display progress during the CLI run. #125

Fixed

• Test server generated wrong schema when endpoints option is passed via CLI. #173
• Error message if schema is not found in CLI. #172

Changed

• Continue running tests on hypothesis error. #137

0.11.0 - 2019-10-22

Added

• LazySchema accepts filters. #149
• Ability to register strategies for custom string formats. #94
• Generator-based events in runner module to improve control over the execution flow.
• Filtration by tags. #134

Changed

• Base URL in schema instances could be reused when it is defined during creation. Now on, base_url argument in Case.call is optional in such cases. #153
• Hypothesis deadline is set to 500ms by default. #138
• Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
• Disallow empty username in CLI --auth option.

Fixed

• User agent during schema loading. #144
• Generation of invalid values in Case.headers. #167

Removed

• Undocumented support for file:// uri schema

0.10.0 - 2019-10-14

Added

• HTTP Digest Auth support. #106
• Support for Hypothesis settings in CLI & Runner. #107
• Case.call and Case.as_requests_kwargs convenience methods. #109
• Local development server. #126

Removed

• Autogenerated runner.StatsCollector.__repr__ to make Hypothesis output more readable.

0.9.0 - 2019-10-09

Added

• Test executor collects results of execution. #29
• CLI option --base-url for specifying base URL of API. #118
• Support for coroutine-based tests. #121
• User Agent to network requests in CLI & runner. #130

Changed

• CLI command schemathesis run prints results in a more readable way with a summary of passing checks.
• Empty header names are forbidden for CLI.
• Suppressed hypothesis exception about using example non-interactively. #92

0.8.1 - 2019-10-04

Fixed

• Wrap each individual test in suppress so the runner doesn't stop after the first test failure.

0.8.0 - 2019-10-04

Added

• CLI tool invoked by the schemathesis command. #30
• New arguments api_options, loader_options and loader for test executor. #90
• A mapping interface for schemas & convenience methods for direct strategies access. #98

Fixed

• Runner stopping on the first falsifying example. #99

0.7.3 - 2019-09-30

Fixed

• Filtration in lazy loaders.

0.7.2 - 2019-09-30

Added

• Support for type "file" for Swagger 2.0. #78
• Support for filtering in loaders. #75

Fixed

• Conflict for lazy schema filtering. #64

0.7.1 - 2019-09-27

Added

• Support for x-nullable extension. #45

0.7.0 - 2019-09-26

Added

• Support for cookie parameter in OpenAPI 3.0 schemas. #21
• Support for formData parameter in Swagger 2.0 schemas. #6
• Test executor. #28

Fixed

• Using hypothesis.settings decorator with test functions created from from_pytest_fixture loader. #69

0.6.0 - 2019-09-24

Added

• Parametrizing tests from a pytest fixture via pytest-subtests. #58

Changed

• Rename module readers to loaders.
• Rename parametrize parameters. filter_endpoint to endpoint and filter_method to method.

Removed

• Substring match for method / endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.

0.5.0 - 2019-09-16

Added

• Generating explicit examples from schema. #17

Changed

• Schemas are loaded eagerly from now on. Using schemathesis.from_uri implies network calls.

Deprecated

• Using Parametrizer.from_{path,uri} is deprecated, use schemathesis.from_{path,uri} instead.

Fixed

• Body resolving during test collection. #55

0.4.1 - 2019-09-11

Fixed

• Possibly unhandled exception during hasattr check in is_schemathesis_test.

0.4.0 - 2019-09-10

Fixed

• Resolving all inner references in objects. #34

Changed

• jsonschema.RefResolver is now used for reference resolving. #35

0.3.0 - 2019-09-06

Added

• Parametrizer.from_uri method to construct parametrizer instances from URIs. #24

Removed

• Possibility to use Parametrizer.parametrize and custom Parametrizer kwargs for passing config options to hypothesis.settings. Use hypothesis.settings decorators on tests instead.

0.2.0 - 2019-09-05

Added

• Open API 3.0 support. #10
• "header" parameters. #7

Changed

• Handle errors during collection / executions as failures.
• Use re.search for pattern matching in filter_method/filter_endpoint instead of fnmatch. #18
• Case.body contains properties from the target schema, without extra level of nesting.

Fixed

• KeyError on collection when "basePath" is absent. #16

0.1.0 - 2019-06-28

• Initial public release