Prevent generating non-UTF8 content in body / deterministic generation?

[juhoinkinen]

Connexion has a problem with dealing with non-UTF8 content in a request body, and I would like to bypass that when testing with Schemathesis, but I cannot figure out how to do it. (This relates to upgrading to Schemathesis 4).

Previously I have successfully filtered out out some path parameters (to bypass another Connexion bug) with @schemathesis.hook("filter_path_parameters"), but I cannot make the same work for body, please see below the @schemathesis.hook("filter_body").

Neither does the setting schema.config.generation.codec = "utf-8" help.

This setup demonstrates the issue:

# myspec.yaml
openapi: 3.0.1
info:
  title: My Example REST
  version: v1
paths:
  /my-method:
    post:
      operationId: rest.my_method
      requestBody:
        content:
          application/json:
            schema:
              type: object
        required: true
      responses:
        "200":
          description: successful operation
          content:
            application/json:
              schema:
                type: object
        "400":
          description: Bad Request
          content:
            application/problem+json:
              schema:
                type: object

# rest.py
def my_method(body):
    return {}, 200, {"Content-Type": "application/json"}

# test_connexion.py
import schemathesis
import connexion

def create_app():
    app = connexion.FlaskApp(__name__, specification_dir=".")
    app.add_api("myspec.yaml")
    return app

app = create_app()
schema = schemathesis.openapi.from_asgi("openapi.json", app=app)
#schema.config.generation.codec = "utf-8"   # This does not work either
#schema.config.generation.deterministic = True

@schemathesis.hook("filter_body")
def filter_body(context, body):
    if body is None or isinstance(body, (dict, list, str, int, float, bool)):
        return True
    elif isinstance(body, (bytes, bytearray)):
        try:
            _ = body.decode("utf-8")
        except UnicodeDecodeError:
            return False
    return True

@schema.parametrize()
def test_openapi_fuzzy(case):
    case.call_and_validate()

Running pytest test_connexion.py produces

 Exception Group Traceback (most recent call last):
  |   File "/home/jmminkin/tmp/schemathesis-connexion/venv/lib/python3.12/site-packages/hypothesis/core.py", line 2246, in wrapped_test
  |     raise the_error_hypothesis_found
  | schemathesis.core.failures.FailureGroup: Schemathesis found 2 distinct failures
  | 
  | - Undocumented HTTP status code
  | 
  |     Received: 500
  |     Documented: 200, 400
  | 
  | - Server error
  | 
  | [500] Internal Server Error:
  | 
  |     `{"type": "about:blank", "title": "Internal Server Error", "detail": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.", "status": 500}`
  | 
  | Reproduce with:
  | 
  |     curl -X POST -H 'Content-Type: application/json' -d $'\x12�\x16��' http://localhost/my-method
  | 
  |  (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | schemathesis.openapi.checks.UndefinedStatusCode: Undocumented HTTP status code
    | 
    | Received: 500
    | Documented: 200, 400
    +---------------- 2 ----------------
    | schemathesis.core.failures.ServerError: Server error
    +------------------------------------

Also, this failure does not show every time, only about every other time. I have tried setting schema.config.generation.deterministic = True but it does not have an effect.

At the moment I have just set

schema.config.checks.not_a_server_error.enabled = False
schema.config.checks.status_code_conformance.enabled = False

but this reduces the benefit of using Schemathesis quite much.

Questions:

• How to make the schemathesis.hook("filter_body") work for my case or is there an other way to preveting bodes like -d $'\x12�\x16��'?
• What else than setting schema.config.generation.deterministic = True should be done to make make the schemathesis runs identical (if that's possible)?

[juhoinkinen]

• What else than setting schema.config.generation.deterministic = True should be done to make make the schemathesis runs identical (if that's possible)?

I noticed the answer in Discord channel:

if I use schemathesis from a Python script, is there a way to enforce a seed to reproduce the same tests?

You can use seed decorator from Hypothesis:

from hypothesis import seed

schema = ...

@seed(42)
@schema.parametrize()
def test_api(case):
    ...

[Stranger6667]

Thanks for opening it!

How to make the schemathesis.hook("filter_body") work for my case or is there an other way to preveting bodes like -d $'\x12�\x16��'?

I assume that the $'\x12�\x16��' payload happens because of the recently added syntax-level fuzzing where in some small number of cases Schemathesis generates random bytes instead of structured data. And I didn't connect it with any config options - will fix it. Right now it bypasses everything

What else than setting schema.config.generation.deterministic = True should be done to make make the schemathesis runs identical (if that's possible)?

Yes, seed should work. However, I think there are a few places where Schemathesis uses use_true_random=True (because it requires proper distribution for things to work), but looses reproducibility. I haven't explored this yet :(

[Stranger6667]

And filters were incorrectly applied to negative test cases :( Will be fixed in #3566

[Stranger6667]

In 4.11.0 it could be resolved with filters, but codec is still ignored because it has "utf8" as the default value, and I am reluctant to generate UTF-8 bytes by default as it will decrease bug discovery rate :( I need to think about this one a bit more

Also, thank you very much for all the details you've provided - it really helped me :)

[juhoinkinen]

Thanks! I'll try 4.11.0 right away.

[juhoinkinen]

Yes, with version 4.11.0 the filter_body hook like above works. 🙏

[Stranger6667]

Glad to hear it! I'll work on the config integration in the next minor release :)

Feel free to reach out if anything else pops up! :)