Feedback: Experimental Open API 3.1 support

[Stranger6667]

We've introduced experimental support for OpenAPI 3.1, focusing initially on compatible JSON Schema validation for responses. Data generation still adheres to OpenAPI 3.0.

How to Enable

• CLI: --experimental=openapi-3.1
• Python API: schemathesis.experimental.openapi_3_1.enable()

We'd love to hear your feedback:

• Did it work as you expected?
• Did you encounter any issues?
• Any other thoughts or suggestions?

Your input is vital for refining this feature. Thank you for contributing to its development!

[hutchins]

CLI --experimental=openapi-3.1 works great, however I'm not clear how to implement the schemathesis.experimental.openapi_3_1.enable() in pytest code. Can you give a little more detail or an example?

[Stranger6667]

@graipher Can you, please, share your code, pls? Maybe I didn't cover some usage pattern

[graipher]

I get the error with the same three lines shown above:

Python 3.11.4 (main, Jun  9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import schemathesis
>>> print(schemathesis.__version__)
3.20.2
>>> schemathesis.experimental.openapi_3_1.enable()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'schemathesis.experimental' has no attribute 'openapi_3_1'

(and I get the same error when trying to add this to my test files as first line after the imports).

[Stranger6667]

Thanks! The original doc was incorrect, the attribute is named OPEN_API_3_1, I updated the docs but forgot to mention it here:

import schemathesis

# Globally enable OpenAPI 3.1 experimental feature
schemathesis.experimental.OPEN_API_3_1.enable()

[hutchins]

I have found what appears to be an issue or it could be that I'm not using the python module correctly.
When using the module in pytest and OpenAPI 3.1 it reports failures that the CLI doesn't.

The consistent error found appears to be related to:

• Received a response with 'text/plain' Content-Type, but it is not declared in the schema.
• Defined content types: application/json
• Response payload: Bad Request

This is a different behavior then when using the CLI.

Code used to reproduce:

schemathesis.experimental.OPEN_API_3_1.enable()
schema = schemathesis.from_uri(
    url
    skip_deprecated_operations=True,
    headers={
        "Authorization": "Bearer " + token,
        "data-partition-id": data_partition,
    },
    code_sample_style="curl",
)
@schema.parametrize()
def test_openapi(case):
    headers = {
        "Authorization": "Bearer " + token,
        "data-partition-id": data_partition,
    }
    case.call_and_validate(headers=headers)

st run -H "Authorization: Bearer xyz" -H "data-partition-id: xyz" --skip-deprecated-operations --code-sample-style curl --experimental=openapi-3.1 URL

[Stranger6667]

I think that in CLI you need to specify --checks=all in order to run all the checks. But I missed that it is inconsistent between CLI & Python tests, thank you for reporting it!

[alan-cd]

Hi @Stranger6667 , we are using the Schemathesis for our API testing. But I met a problem with openapi-3.1 and concurrent testing.
Here is my command:

st run --workers auto --checks all *** --base-url *** --experimental=openapi-3.1 --hypothesis-suppress-health-check=filter_too_much

Initially I didn't add the parameter --workers auto, it works well. But when I add it, it throws a runtime error like below:
Here is the stacktrace:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/runner/impl/core.py", line 737, in network_test
    response = _network_test(case, *args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/runner/impl/core.py", line 784, in _network_test
    run_checks(
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/runner/impl/core.py", line 610, in run_checks
    skip_check = check(response, copied_case)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/specs/openapi/checks.py", line 129, in response_schema_conformance
    return case.operation.validate_response(response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/models.py", line 862, in validate_response
    return self.schema.validate_response(self, response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/specs/openapi/schemas.py", line 613, in validate_response
    if self.spec_version.startswith("3.1") and experimental.OPEN_API_3_1.is_enabled:
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/experimental/__init__.py", line 25, in is_enabled
    return self._storage.is_enabled(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/experimental/__init__.py", line 82, in is_enabled
    return feature in self.enabled
                      ^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/schemathesis/experimental/__init__.py", line 50, in enabled
    return self._local_data.enabled
           ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: '_thread._local' object has no attribute 'enabled'

I checked the trace log and I think it may relate to the experimental feature OpenAPI-3.1, so I comment here.

Do I make any mistakes?

[Stranger6667]

Hi, it looks like a bug, I’ll check it. Thanks for reporting

[Stranger6667]

Fixed, I am going to make a new release in the next few days

[Stranger6667]

Released in 3.25.2

[alan-cd]

Great! I've just run this again and now it is passed. Thank you very much for creating this great tool!

[Stranger6667]

Thank you @alan-cd ! Happy to hear it from you :) do not hesitate to reach out if you have any other questions :)