[Question] Remove testcases containing NULL or \000 when testing my API

[AryanGodara]

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker
  I have a common problem while testing my API (written in golang). Many fields test by putting string values as
  null or \u0000
  I want to turn off null values to be tested, but can't find openAPI specification docs for this. Is there an option to do so in schemathesis?

Below is a map[string]{interface} object in golang, described in openAPI.

UserMetadata:
      type: object
      nullable: false
      properties:
        metadata:
          type: object
          nullable: false
          additionalProperties: true
          description: Arbitrary, object-encoded user's data.
      required:
        - metadata

And the following is a failing command generated by st report :-

curl -X PUT -H 'Authorization: Bearer $MF_TOKEN' -d '{"metadata": {"\u0000": null}}' http://localhost/users

where example of MF_TOKEN is
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzQzNDA5MDgsImlhdCI6MTY3NDMwNDkwOCwiaXNzIjoibWFpbmZsdXguYXV0aCIsInN1YiI6ImV4YW1wbGVAZWcuY29tIiwiaXNzdWVyX2lkIjoiNzE0NTk5MmYtMzZkZi00NjE5LWE1YzQtOGJkMzg2YjI3YmE5IiwidHlwZSI6MH0.b5_D2lpOtqJ_CNtVO4AU9ieJULNM3w4rj2-DGt-otwE

[AryanGodara]

I know it's difficult to go through each issue, but can someone please direct me in the right direction on how to fix this issue

[Stranger6667]

Hi! Thank you for pinging and sorry for the delay :)

The underlying data generator, hypothesis-jsonschema, generates only data that matches the schema, hence the most effective way to avoid these values is to tune the schema. However it is not always practically convenient.

Leaving aside possible changes to hypothesis-jsonschema that will simplify some use cases, you can use Schemathesis hooks. I’ll share some examples shortly

[AryanGodara]

Hey @Stranger6667 , thanks for taking the time🙌.
Yes a few examples would be really helpful, especially on how to remove
null , u0000 or other such values from map[string]{inferface} types.

[AryanGodara]

Also, are the hooks available in schemathesis-github-actions, maybe as arguments?
I run the tests individually for multiple yaml files from github actions script:-

# Auth.yml
      - name: Run st for auth.yml
        if: always()
        uses: schemathesis/action@v1
        with:
          schema: './api/openapi/auth.yml'
          base-url: 'http://localhost'
          args: -H Authorization:${{ env.MF_TOKEN }}
          token: ${{ secrets.SCHEMATHESIS_TOKEN }}

      # Bootstrap.yml
      - name: Run st for bootstrap.yml
        if: always()
        uses: schemathesis/action@v1
        with:
          schema: './api/openapi/bootstrap.yml'
          base-url: 'http://localhost'
          args: -H Authorization:${{ env.MF_TOKEN }}
          token: ${{ secrets.SCHEMATHESIS_TOKEN }}

[Stranger6667]

There are two ways to change how Schemathesis generates data.

1. By construction. Here you need to modify the schema, either manually or via Schemathesis hooks during test execution. For example, you might want to add propertyNames to that metadata contains properties only matching a certain schema (e.g. via pattern):

UserMetadata:
      type: object
      nullable: false
      properties:
        metadata:
          type: object
          nullable: false
          propertyNames:
              pattern: "^[a-zA-Z]*$"
          additionalProperties: true
          description: Arbitrary, object-encoded user's data.
      required:
        - metadata

Here is a hook you may use:

import schemathesis


@schemathesis.hook
def before_load_schema(context, raw_schema):
    # Modify the schema at the place where UserMetadata lives
    raw_schema["components"]["schemas"]["UserMetadata"]["properties"]["metadata"]["propertyNames"] = {"pattern": "^[a-zA-Z]*$"}

2. Filtering / modification. Here generated data is either thrown away or modified so it matches your expectations.

import schemathesis


@schemathesis.hook
def before_generate_case(context, strategy):
    op = context.operation

    def no_nulls(case):
        # Run this filter only for "PATCH /users/{user_id}/" endpoint
        if op.method == "PATCH" and op.path == "/users/{user_id}/":
            # Check that generated body has no nulls
            return all("\x00" not in key for key in case.body["metadata"])
        return True

    return strategy.filter(no_nulls)

Filtering is a bit clunky as you need to check potentially nested structures of different kinds.

Here is how you connect hooks to your Schemathesis run.

P.S. I plan to overhaul the hooks system and if you'll find something that is not convenient or you have any comments on it, please feel free to share it in this issue

[Stranger6667]

Oh, that's an oversight! Currently, args are passed only to the run command, but hooks are passed to the st command itself. Here is an issue for this

[Stranger6667]

And thank you so much for sharing your context and asking questions. It is extremely valuable for Schemathesis :)

[AryanGodara]

Thank you for all the help @Stranger6667 , this'll really speed up the progress on my task!

[AryanGodara]

I just had one more question, is it possible to run tests on multiple yml files in a single command?
I have auth.yml, bootstrap.yml, http.yml
and more files in a single directory, for which I have to run tests individually with the same configuration.

[Stranger6667]

I just had one more question, is it possible to run tests on multiple yml files in a single command?

I'd suggest running tests via GH Action matrix instead as Schemathesis is designed to run against one schema at a time

[AryanGodara]

Thanks for the tip @Stranger6667 ! I'll look into this.

[Stranger6667]

Hey @AryanGodara

FYI - Hooks are now supported in Schemathesis GitHub Action starting from v1.0.3 (it requires at least Schemathesis 3.18.5 as well)

[AryanGodara]

Hey @AryanGodara

FYI - Hooks are now supported in Schemathesis GitHub Action starting from v1.0.3 (it requires at least Schemathesis 3.18.5 as well)

Thanks for the quick update on this @Stranger6667 !