- Check name is no longer displayed in the CLI output, since its verbose message is already displayed. This change also simplifies the internal structure of the runner events.
2.4.1 - 2020-09-17
- Hide
Case.endpoint
from representation. Its representation decreases the usability of the pytest's output. #719 - Return registered functions from
register_target
andregister_check
decorators. #721
- Possible
IndexError
when a user-defined check raises an exception without a message. #718
2.4.0 - 2020-09-15
- Ability to register custom targets for targeted testing. #686
- The
AfterExecution
event now haspath
andmethod
fields, similar to theBeforeExecution
one. The goal is to make these events self-contained, which improves their usability.
2.3.4 - 2020-09-11
- The default Hypothesis's
deadline
setting for tests withschema.parametrize
is set to 500 ms for consistency with the CLI behavior. #705
- Encoding error when writing a cassette on Windows. #708
2.3.3 - 2020-08-04
KeyError
during thecontent_type_conformance
check if the response has noContent-Type
header. #692
2.3.2 - 2020-08-04
- Run checks conditionally.
2.3.1 - 2020-07-28
IndexError
whenexamples
list is empty.
2.3.0 - 2020-07-26
- Possibility to generate values for
in: formData
parameters that are non-bytes or contain non-bytes (e.g., inside an array). #665
- Error message for cases when a path parameter is in the template but is not defined in the parameters list or missing
required: true
in its definition. #667 - Bump minimum required
hypothesis-jsonschema
version to 0.17.0. This allows Schemathesis to use thecustom_formats
argument infrom_schema
calls and avoid using its private API. #684
ValueError
during sending a request with test payload if the endpoint defines a parameter withtype: array
andin: formData
. #661KeyError
while processing a schema with nullable parameters andin: body
. #660StopIteration
duringrequestBody
processing if it has empty "content" value. #673AttributeError
during generation of "multipart/form-data" parameters that have no "type" defined. #675- Support for properties named "$ref" in object schemas. Previously, it was causing
TypeError
. #672 - Generating illegal Unicode surrogates in the path. #668
- Invalid development dependency on
graphql-server-core
package. #658
2.2.1 - 2020-07-22
- Possible
UnicodeEncodeError
during generation ofAuthorization
header values for endpoints withbasic
security scheme. #656
2.2.0 - 2020-07-14
schemathesis.graphql.from_dict
loader allows you to use GraphQL schemas represented as a dictionary for testing.before_load_schema
hook for GraphQL schemas.
- Serialization of non-string parameters. #651
2.1.0 - 2020-07-06
- Support for property-level examples. #467
- Content-type conformance check for cases when Open API 3.0 schemas contain "default" response definitions. #641
- Handling of multipart requests for Open API 3.0 schemas. #640
- Sending non-file form fields in multipart requests. #647
- Deprecated
skip_validation
argument toHookDispatcher.apply
. - Deprecated
_accepts_context
internal function.
2.0.0 - 2020-07-01
- BREAKING. Base URL handling.
base_url
now is treated as one with a base path included. You should pass a full base URL now instead:
schemathesis run --base-url=http://127.0.0.1:8080/api/v2 ...
This value will override basePath
/ servers[0].url
defined in your schema if you use Open API 2.0 / 3.0 respectively. Previously if you pass a base URL like the one above, it was concatenated with the base path defined in the schema, which leads to a lack of ability to redefine the base path. #511
- Show the correct URL in CLI progress when the base URL is overridden, including the path part. #511
- Construct valid URL when overriding base URL with base path. #511
Example:
Base URL in the schema : http://0.0.0.0:8081/api/v1
`--base-url` value in CLI : http://0.0.0.0:8081/api/v2
Full URLs before this change : http://0.0.0.0:8081/api/v2/api/v1/users/ # INVALID!
Full URLs after this change : http://0.0.0.0:8081/api/v2/users/ # VALID!
- Support for hooks without context argument in the first position.
- Hooks registration by name and function. Use
register
decorators instead. For more details, see the "Customization" section in our documentation. BaseSchema.with_hook
andBaseSchema.register_hook
. UseBaseSchema.hooks.apply
andBaseSchema.hooks.register
instead.
1.10.0 - 2020-06-28
loaders.from_asgi
supports making calls to ASGI-compliant application (For example: FastAPI). #521- Support for GraphQL strategies.
- Passing custom headers to schema loader for WSGI / ASGI apps. #631
1.9.1 - 2020-06-21
- Schema validation error on schemas containing numeric values in scientific notation without a dot. #629
1.9.0 - 2020-06-20
- Pass the original case's response to the
add_case
hook. - Support for multiple examples with OpenAPI
examples
. #589 --verbosity
CLI option to minimize the error output. #598- Allow registering function-level hooks without passing their name as the first argument to
apply
. #618 - Support for hook usage via
LazySchema
/from_pytest_fixture
. #617
- Tests with invalid schemas marked as errors, instead of failures. #622
- Crash during the generation of loosely-defined headers. #621
- Show exception information for test runs on invalid schemas with
--validate-schema=false
command-line option. Before, the output sections for invalid endpoints were empty. #622
1.8.0 - 2020-06-15
- Tests with invalid schemas are marked as failed instead of passed when
hypothesis-jsonschema>=0.16
is installed. #614 KeyError
during creating an endpoint strategy if it contains a reference. #612
- Require
hypothesis-jsonschema>=0.16
. #614 - Pass original
InvalidSchema
text topytest.fail
call.
1.7.0 - 2020-05-30
- Support for YAML files in references via HTTPS & HTTP schemas. #600
- Stateful testing support via
Open API links
syntax. #548 - New
add_case
hook. #458 - Support for parameter serialization formats in Open API 2 / 3. For example
pipeDelimited
ordeepObject
. #599 - Support serializing parameters with
application/json
content-type. #594
- The minimum required versions for
Hypothesis
andhypothesis-jsonschema
are5.15.0
and0.11.1
respectively. The main reason is this fix that is required for stability of Open API links feature when it is executed in multiple threads.
1.6.3 - 2020-05-26
- Support for a colon symbol (
:
) inside of a header value passed via CLI. #596
1.6.2 - 2020-05-15
- Partially generated explicit examples are always valid and can be used in requests. #582
1.6.1 - 2020-05-13
- Look at the current working directory when loading hooks for CLI. #586
1.6.0 - 2020-05-10
- Passing
workers_num
toThreadPoolRunner
leads to always using 2 workers in this worker kind. #579
1.5.1 - 2020-05-08
- Display proper headers in reproduction code when headers are overridden. #566
1.5.0 - 2020-05-06
- Display a suggestion to disable schema validation on schema loading errors in CLI. #531
- Filtration of endpoints by
operationId
viaoperation_id
parameter toschema.parametrize
or-O
command-line option. #546 - Generation of security-related parameters. They are taken from
securityDefinitions
/securitySchemes
and injected to the generated data. It supports generating API keys in headers or query parameters and generating data for HTTP authentication schemes. #540
- Overriding header values in CLI and runner when headers provided explicitly clash with ones defined in the schema. #559
- Nested references resolving in
response_schema_conformance
check. #562 - Nullable parameters handling when they are behind a reference. #542
1.4.0 - 2020-05-03
context
argument for hook functions to provide an additional context for hooks. A deprecation warning is emitted for hook functions that do not accept this argument.- A new hook system that allows generic hook dispatching. It comes with new hook locations. For more details, see the "Customization" section in our documentation.
- New
before_process_path
hook. - Third-party compatibility fixups mechanism. Currently, there is one fixup for FastAPI. #503
- Hook functions that do not accept
context
as their first argument. They will become not be supported in Schemathesis 2.0. - Registering hooks by name and function. Use
register
decorators instead. For more details, see the "Customization" section in our documentation. BaseSchema.with_hook
andBaseSchema.register_hook
. UseBaseSchema.hooks.apply
andBaseSchema.hooks.register
instead.
- Add missing
validate_schema
argument toloaders.from_pytest_fixture
. - Reference resolving during response schema conformance check. #539
1.3.4 - 2020-04-30
- Validation of nullable properties in
response_schema_conformance
check introduced in1.3.0
. #542
1.3.3 - 2020-04-29
- Update
pytest-subtests
pin to>=0.2.1,<1.0
. #537
1.3.2 - 2020-04-27
- Show exceptions if they happened during loading a WSGI application. Option
--show-errors-tracebacks
will display a full traceback.
1.3.1 - 2020-04-27
- Packaging issue
1.3.0 - 2020-04-27
- Storing network logs with
--store-network-log=<filename.yaml>
. The stored cassettes are based on the VCR format and contain extra information from the Schemathesis internals. #379 - Replaying of cassettes stored in VCR format. #519
- Targeted property-based testing in CLI and runner. It only supports the
response_time
target at the moment. #104 - Export CLI test results to JUnit.xml with
--junit-xml=<filename.xml>
. #427
- Code samples for schemas where
body
is defined as{"type": "string"}
. #521 - Showing error causes on internal
jsonschema
errors during input schema validation. #513 - Recursion error in
response_schema_conformance
check. Because of this change,Endpoint.definition
contains a definition where references are not resolved. In this way, it makes it possible to avoid recursion errors injsonschema
validation. #468
- Added indentation & section name to the
SUMMARY
CLI block. - Use C-extension for YAML loading when it is possible. It can cause more than 10x speedup on schema parsing. Do not show Click's "Aborted!" message when an error occurs during CLI schema loading.
- Add a help message to the CLI output when an internal exception happens. #529
1.2.0 - 2020-04-15
- Per-test hooks for modification of data generation strategies. #492
- Support for
x-example
vendor extension in Open API 2.0. #504 - Sanity validation for the input schema & loader in
runner.prepare
. #499
1.1.2 - 2020-04-14
- Support for custom loaders in
runner
. Now all built-in loaders are supported as an argument torunner.prepare
. #496 from_wsgi
loader accepts custom keyword arguments that will be passed toclient.get
when accessing the schema. #497
1.1.1 - 2020-04-12
- Mistakenly applied Open API -> JSON Schema Draft 7 conversion. It should be Draft 4. #489
- Using wrong validator in
response_schema_conformance
check. It should be Draft 4 validator. #468
1.1.0 - 2020-04-08
- Response schema check for recursive schemas. #468
- App loading in
runner
. Now it accepts application as an importable string, rather than an instance. It is done to make it possible to execute a runner in a subprocess. Otherwise, apps can't be easily serialized and transferred into another process. - Runner events structure. All data in events is static from now. There are no references to
BaseSchema
,Endpoint
or similar objects that may calculate data dynamically. This is done to make events serializable and not tied to Python object, which decouples anyrunner
consumer from implementation details. It will help makerunner
usable in more cases (e.g., web application) since events can be serialized to JSON and used in any environment. Another related change is that Python exceptions are not propagated anymore - they are replaced with theInternalError
event that should be handled accordingly.
1.0.5 - 2020-04-03
- Open API 3. Handling of endpoints that contain
multipart/form-data
media types. Previously only file upload endpoints were working correctly. #473
1.0.4 - 2020-04-03
1.0.3 - 2020-04-03
- Precedence of
produces
keywords for Swagger 2.0 schemas. Now, operation-levelproduces
overrides schema-levelproduces
as specified in the specification. #463 - Content-type conformance check for Open API 3.0 schemas. #461
- Pytest 5.4 warning for test functions without parametrization. #451
1.0.2 - 2020-04-02
- Handling of fields in
paths
that are not operations, but allowed by the Open API spec. #457 - Pytest 5.4 warning about deprecated
Node
initialization usage. #451
1.0.1 - 2020-04-01
- Processing of explicit examples in Open API 3.0 when there are multiple parameters in the same location (e.g.
path
) containexample
value. They are properly combined now. #450
1.0.0 - 2020-03-31
- Move processing of
runner
parameters torunner.prepare
. This change will provide better code reuse since all users ofrunner
(e.g., if you extended it in your project`) need some kind of input parameters handling, which was implemented only in Schemathesis CLI. It is not backward-compatible. If you didn't userunner
directly, then this change should not have a visible effect on your use-case.
0.28.0 - 2020-03-31
- Handling of schemas that use
x-*
custom properties. #448
- Deprecated
runner.execute
. Userunner.prepare
instead.
0.27.0 - 2020-03-31
runner.execute
should not be used, sincerunner.prepare
provides a more flexible interface to test execution.
- Deprecated
Parametrizer
class. Useschemathesis.from_path
as a replacement forParametrizer.from_path
.
0.26.1 - 2020-03-24
- Limit recursion depth while resolving JSON schema to handle recursion without breaking. #435
0.26.0 - 2020-03-19
- Filter problematic path template variables containing
"/"
, or"%2F"
url encoded. #440 - Filter invalid empty
""
path template variables. #439 - Typo in a help message in the CLI output. #436
0.25.1 - 2020-03-09
- Allow
werkzeug
>= 1.0.0. #433
0.25.0 - 2020-02-27
- Handling of explicit examples from schemas. Now, if there are examples for multiple locations (e.g., for body and query) then they will be combined into a single example. #424
0.24.5 - 2020-02-26
- Error during
pytest
collection on objects with custom__getattr__
method and therefore passis_schemathesis
check. #429
0.24.4 - 2020-02-22
- Resolving references when the schema is loaded from a file on Windows. #418
0.24.3 - 2020-02-10
- Not copied
validate_schema
parameter inBaseSchema.parametrize
. Regression after implementing #383 - Missing
app
,location
andhooks
parameters in schema when used withBaseSchema.parametrize
. #416
0.24.2 - 2020-02-09
- Crash on invalid regular expressions in
method
,endpoint
andtag
CLI options. #403 - Crash on a non-latin-1 encodable value in the
auth
CLI option. #404 - Crash on an invalid value in the
header
CLI option. #405 - Crash on some invalid URLs in the
schema
CLI option. #406 - Validation of
--request-timeout
parameter. #407 - Crash with
--hypothesis-deadline=0
CLI option. #410 - Crash with
--hypothesis-max-examples=0
CLI option. #412
0.24.1 - 2020-02-08
- CLI crash on Windows and Python < 3.8 when the schema path contains characters unrepresentable at the OS level. #400
0.24.0 - 2020-02-07
- Support for testing of examples in Parameter & Media Type objects in Open API 3.0. #394
--show-error-tracebacks
CLI option to display errors' tracebacks in the output. #391- Support for schema behind auth. #115
- Schemas with GET endpoints accepting body are allowed now if schema validation is disabled (via
--validate-schema=false
for example). The use-case is for tools like ElasticSearch that use GET requests with non-empty bodies. #383
- CLI crash when an explicit example is specified in the endpoint definition. #386
0.23.7 - 2020-01-30
-x
/--exitfirst
CLI option to exit after the first failed test. #378
- Handling examples of parameters in Open API 3.0. #381
0.23.6 - 2020-01-28
all
variant for--checks
CLI option to use all available checks. #374
- Use built-in
importlib.metadata
on Python 3.8. #376
0.23.5 - 2020-01-24
- Generation of invalid values in
Case.cookies
. #371
0.23.4 - 2020-01-22
- Converting
exclusiveMinimum
&exclusiveMaximum
fields to JSON Schema. #367
0.23.3 - 2020-01-21
- Filter out surrogate pairs from the query string.
0.23.2 - 2020-01-16
- Prevent
KeyError
when the response does not have the "Content-Type" header. #365
0.23.1 - 2020-01-15
- Dockerfile entrypoint was not working as per docs. #361
0.23.0 - 2020-01-15
- Hooks for strategy modification. #313
- Input schema validation. Use
--validate-schema=false
to disable it in CLI andvalidate_schema=False
argument in loaders. #110
0.22.0 - 2020-01-11
- Show multiple found failures in the CLI output. #266 & #207
- Raise a proper exception when the given schema is invalid. #308
- Support for
None
as a value for--hypothesis-deadline
. #349
- Handling binary request payloads in
Case.call
. #350 - Type of the second argument to all built-in checks set to proper
Case
instead ofTestResult
. The error was didn't affect built-in checks since bothCase
andTestResult
hadendpoint
attribute, and only it was used. However, this fix is not backward-compatible with 3rd party checks.
0.21.0 - 2019-12-20
- Support for AioHTTP applications in CLI. #329
0.20.5 - 2019-12-18
- Compatibility with the latest release of
hypothesis-jsonschema
and setting its minimal required version to0.9.13
. #338
0.20.4 - 2019-12-17
- Handling
nullable
attribute in Open API schemas. #335
0.20.3 - 2019-12-17
- Usage of the response status code conformance check with old
requests
version. #330
0.20.2 - 2019-12-14
- Response schema conformance check for Open API 3.0. #332
0.20.1 - 2019-12-13
- Support for response code ranges. #330
0.20.0 - 2019-12-12
- WSGI apps support. #31
Case.validate_response
for running built-in checks against app's response. #319
- Checks receive
Case
instance as a second argument instead ofTestResult
. This was done for making checks usable in Python tests viaCase.validate_response
. Endpoint and schema are accessible viacase.endpoint
andcase.endpoint.schema
.
0.19.1 - 2019-12-11
- Compatibility with Hypothesis >= 4.53.2. #322
0.19.0 - 2019-12-02
0.18.1 - 2019-11-28
- Validation of the
base-url
CLI parameter. #311
0.18.0 - 2019-11-27
- Resolving references in
PathItem
objects. #301
- Resolving of relative paths in schemas. #303
- Loading string dates as
datetime.date
objects in YAML loader. #305
0.17.0 - 2019-11-21
- Resolving references that point to different files. #294
- Keyboard interrupt is now handled during the CLI run, and the summary is displayed in the output. #295
0.16.0 - 2019-11-19
- Display RNG seed in the CLI output to allow test reproducing. #267
- Allow specifying seed in CLI.
- Ability to pass custom kwargs to the
requests.get
call inloaders.from_uri
.
- Refactor case generation strategies: strategy is not used to generate empty value. #253
- Improved error message for invalid path parameter declaration. #255
- Pytest fixture parametrization via
pytest_generate_tests
. #280 - Support for tests defined as methods. #282
- Unclosed
requests.Session
on callingCase.call
without passing a session explicitly. #286
0.15.0 - 2019-11-15
- Support for OpenAPI 3.0 server variables (base_path). #40
- Support for
format: byte
. #254 - Response schema conformance check in CLI / Runner. #256
- Docker image for CLI. #268
- Pre-run hooks for CLI. #147
- A way to register custom checks for CLI via
schemathesis.register_check
. #270
- Not encoded path parameters. #272
- Verbose messages are displayed in the CLI on failed checks. #261
0.14.0 - 2019-11-09
- CLI: Support file paths in the
schema
argument. #119 - Checks to verify response status & content type in CLI / Runner. #101
- Custom base URL handling in CLI / Runner. #248
- Raise an error if the schema has a body for GET requests. #218
- Method names are case insensitive during direct schema access. #246
0.13.2 - 2019-11-05
IndexError
when Hypothesis found inconsistent test results during the test execution in the runner. #236
0.13.1 - 2019-11-05
- Support for binary format #197
- Error that happens when there are no success checks in the statistic in CLI. #237
0.13.0 - 2019-11-03
- An option to configure request timeout for CLI / Runner. #204
- A help snippet to reproduce errors caught by Schemathesis. #206
- Total running time to the CLI output. #181
- Summary line in the CLI output with the number of passed / failed / errored endpoint tests. #209
- Extra information to the CLI output: schema address, spec version, and base URL. #188
- Compatibility with Hypothesis 4.42.4+ . #212
- Display flaky errors only in the "ERRORS" section and improve CLI output. #215
- Handling
formData
parameters inCase.call
. #196 - Handling cookies in
Case.call
. #211
- More readable falsifying examples output. #127
- Show exceptions in a separate section of the CLI output. #203
- Error message for cases when it is not possible to satisfy schema parameters. It should be more clear now. #216
- Do not stop on schema errors related to a single endpoint. #139
- Display a proper error message when the schema is not available in CLI / Runner. #214
0.12.2 - 2019-10-30
- Wrong handling of the
base_url
parameter in runner andCase.call
if it has a trailing slash. #194 and #199 - Do not send any payload with GET requests. #200
0.12.1 - 2019-10-28
- Handling for errors other than
AssertionError
andHypothesisException
in the runner. #189 - CLI failing on the case when there are tests, but no checks were performed. #191
- Display the "SUMMARY" section in the CLI output for empty test suites.
0.12.0 - 2019-10-28
- Display progress during the CLI run. #125
- Test server-generated wrong schema when the
endpoints
option is passed via CLI. #173 - Error message if the schema is not found in CLI. #172
- Continue running tests on hypothesis error. #137
0.11.0 - 2019-10-22
- LazySchema accepts filters. #149
- Ability to register strategies for custom string formats. #94
- Generator-based events in the
runner
module to improve control over the execution flow. - Filtration by tags. #134
- Base URL in schema instances could be reused when it is defined during creation. Now on, the
base_url
argument inCase.call
is optional in such cases. #153 - Hypothesis deadline is set to 500ms by default. #138
- Hypothesis output is captured separately, without capturing the whole stdout during CLI run.
- Disallow empty username in CLI
--auth
option.
- Undocumented support for
file://
URI schema
0.10.0 - 2019-10-14
- HTTP Digest Auth support. #106
- Support for Hypothesis settings in CLI & Runner. #107
Case.call
andCase.as_requests_kwargs
convenience methods. #109- Local development server. #126
- Autogenerated
runner.StatsCollector.__repr__
to make Hypothesis output more readable.
0.9.0 - 2019-10-09
- Test executor collects results of execution. #29
- CLI option
--base-url
for specifying base URL of API. #118 - Support for coroutine-based tests. #121
- User Agent to network requests in CLI & runner. #130
- CLI command
schemathesis run
prints result in a more readable way with a summary of passing checks. - Empty header names are forbidden for CLI.
- Suppressed hypothesis exception about using
example
non-interactively. #92
0.8.1 - 2019-10-04
- Wrap each test in
suppress
so the runner doesn't stop after the first test failure.
0.8.0 - 2019-10-04
- CLI tool invoked by the
schemathesis
command. #30 - New arguments
api_options
,loader_options
andloader
for test executor. #90 - A mapping interface for schemas & convenience methods for direct strategy access. #98
- Runner stopping on the first falsifying example. #99
0.7.3 - 2019-09-30
- Filtration in lazy loaders.
0.7.2 - 2019-09-30
- Conflict for lazy schema filtering. #64
0.7.1 - 2019-09-27
- Support for
x-nullable
extension. #45
0.7.0 - 2019-09-26
- Support for the
cookie
parameter in OpenAPI 3.0 schemas. #21 - Support for the
formData
parameter in Swagger 2.0 schemas. #6 - Test executor. #28
- Using
hypothesis.settings
decorator with test functions created fromfrom_pytest_fixture
loader. #69
0.6.0 - 2019-09-24
- Parametrizing tests from a pytest fixture via
pytest-subtests
. #58
- Rename module
readers
toloaders
. - Rename
parametrize
parameters.filter_endpoint
toendpoint
andfilter_method
tomethod
.
- Substring match for method/endpoint filters. To avoid clashing with escaped chars in endpoints keys in schemas.
0.5.0 - 2019-09-16
- Generating explicit examples from the schema. #17
- Schemas are loaded eagerly from now on. Using
schemathesis.from_uri
implies network calls.
- Using
Parametrizer.from_{path,uri}
is deprecated, useschemathesis.from_{path,uri}
instead.
- Body resolving during test collection. #55
0.4.1 - 2019-09-11
- Possibly unhandled exception during
hasattr
check inis_schemathesis_test
.
0.4.0 - 2019-09-10
- Resolving all inner references in objects. #34
jsonschema.RefResolver
is now used for reference resolving. #35
0.3.0 - 2019-09-06
Parametrizer.from_uri
method to construct parametrizer instances from URIs. #24
- Possibility to use
Parametrizer.parametrize
and customParametrizer
kwargs for passing config options tohypothesis.settings
. Usehypothesis.settings
decorators on tests instead.
0.2.0 - 2019-09-05
- Handle errors during collection / executions as failures.
- Use
re.search
for pattern matching infilter_method
/filter_endpoint
instead offnmatch
. #18 Case.body
contains properties from the target schema, without the extra level of nesting.
KeyError
on collection when "basePath" is absent. #16
- Initial public release