[BUG] "FailedHealthCheck : 50 filtered examples but only 0 good ones." Schemathesis not producing tests for a valid open API yaml file

[AryanGodara]

Checklist

• I checked the FAQ section of the documentation
• I looked for similar issues in the issue tracker

Describe the bug
Error in all endpoints when testing a simple openAPI/swaggerUI specification file.

To Reproduce
Steps to reproduce the behavior:

1. Run this command

st run --base-url=http://localhost -H "Content-Type: application/json" -H "Authorization: Bearer $ACCESS_TOKEN" --validate-schema=true ./api/openapi/http.yml

2. See error :-

================================================================================ Schemathesis test session starts ================================================================================
Schema location: file:///home/aryan/mainflux/api/openapi/http.yml
Base URL: http://localhost
Specification version: Open API 3.0.1
Workers: 1
Collected API operations: 2

POST /channels/{id}/messages E                                                                                                                                                              [ 50%]
GET /health E                                                                                                                                                                               [100%]

============================================================================================= ERRORS =============================================================================================
__________________________________________________________________________________ POST /channels/{id}/messages __________________________________________________________________________________
hypothesis.errors.FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 0 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.

__________________________________________________________________________________________ GET /health ___________________________________________________________________________________________
hypothesis.errors.Unsatisfiable: Unable to satisfy schema parameters for this API operation

Add this option to your command line parameters to see full tracebacks: --show-errors-tracebacks

If you need assistance in solving this, feel free to join our Discord server and report it:

  https://discord.gg/R9ASRAmHnA

============================================================================================ SUMMARY =============================================================================================

No checks were performed.

Hint: You can visualize test results in Schemathesis.io by using `--report` in your CLI command.

If possible, please post a minimal version of your API schema that cause this behavior:

openapi: 3.0.1
info:
  title: Mainflux http adapter
  description: |
    HTTP API for sending messages through communication channels.
    Some useful links:
    - [The Mainflux repository](https://github.com/mainflux/mainflux)
  contact:
    email: info@mainflux.com
  license:
    name: Apache 2.0
    url: https://github.com/mainflux/mainflux/blob/master/LICENSE
  version: 0.14.0

servers:
  - url: http://localhost:8008
  - url: https://localhost:8008

tags:
  - name: messages
    description: Everything about your Messages
    externalDocs:
      description: Find out more about messages
      url: http://docs.mainflux.io/
      
paths:
  /channels/{id}/messages:
    post:
      summary: Sends message to the communication channel
      description: |
        Sends message to the communication channel. Messages can be sent as
        JSON formatted SenML or as blob.
      tags:
        - messages
      parameters:
        - $ref: "#/components/parameters/ID"
      requestBody:
        $ref: "#/components/requestBodies/MessageReq"
      responses:
        "202":
          description: Message is accepted for processing.
        "400":
          description: Message discarded due to its malformed content.
        "401":
          description: Missing or invalid access token provided.
        "404":
          description: Message discarded due to invalid channel id.
        "415":
          description: Message discarded due to invalid or missing content type.
        '500':
          $ref: "#/components/responses/ServiceError"
  /health:
    get:
      summary: Retrieves service health check info.
      tags:
        - health
      responses:
        '200':
          $ref: "#/components/responses/HealthRes"
        '500':
          $ref: "#/components/responses/ServiceError"

components:
  schemas:
    SenMLRecord:
      type: object
      properties:
        bn:
          type: string
          description: Base Name
        bt:
          type: number
          format: double
          description: Base Time
        bu:
          type: number
          format: double
          description: Base Unit
        bv:
          type: number
          format: double
          description: Base Value
        bs:
          type: number
          format: double
          description: Base Sum
        bver:
          type: number
          format: double
          description: Version
        n:
          type: string
          description: Name
        u:
          type: string
          description: Unit
        v:
          type: number
          format: double
          description: Value
        vs:
          type: string
          description: String Value
        vb:
          type: boolean
          description: Boolean Value
        vd:
          type: string
          description: Data Value
        s:
          type: number
          format: double
          description: Value Sum
        t:
          type: number
          format: double
          description: Time
        ut:
          type: number
          format: double
          description: Update Time
    SenMLArray:
      type: array
      items:
        $ref: "#/components/schemas/SenMLRecord"

  parameters:
    ID:
      name: id
      description: Unique channel identifier.
      in: path
      schema:
        type: string
        format: uuid
      required: true

  requestBodies:
    MessageReq:
      description: |
        Message to be distributed. Since the platform expects messages to be
        properly formatted SenML in order to be post-processed, clients are
        obliged to specify Content-Type header for each published message.
        Note that all messages that aren't SenML will be accepted and published,
        but no post-processing will be applied.
      required: true
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/SenMLArray"

  responses:
    ServiceError:
      description: Unexpected server-side error occurred.
      
    HealthRes:
      description: Service Health Check.
      content:
        application/json:
          schema:
            $ref: "./schemas/HealthInfo.yml"

  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: uuid
      description: |
        * Thing access: "Authorization: Thing <thing_key>"

    basicAuth:
      type: http
      scheme: basic
      description: |
        * Things access: "Authorization: Basic <base64-encoded_credentials>"

security:
  - bearerAuth: []
  - basicAuth: []

Expected behavior
I want to schemathesis to produce a result, where the endpoints either fail or pass, or atleasting show me example cases of why my endpoint is failing, not give up on generating usecases.

Environment (please complete the following information):

• OS: [Linux: Ubuntu 22.04 LTS ]
• Python version: [3.10.2]
• Schemathesis version: [3.19.6]
• Spec version: [Open API 3.0.1]

Additional Context

I'm using this inside a pull request for Mainflux IoT Platform
Link to the Pull Request: https://github.com/mainflux/mainflux/pull/1692

[Stranger6667]

Hi! Thanks for opening the issue! On t he first glance, the schema looks satisfiable and without logically incompatible blocks. I guess that, at least we can have a better error message that could point to logically incompatible parts if there are any. However, Schemathesis could be too strict in terms of data generation which leads to these results. You might want to disable health checks or adjust the schema a bit so it is easier to generate (eg adding additionalProperties:false leads to easier generation). At the moment I can’t give more details, but I’ll get back to it in the beginning of the next week

[AryanGodara]

Hi! Thanks for opening the issue! On t he first glance, the schema looks satisfiable and without logically incompatible blocks. I guess that, at least we can have a better error message that could point to logically incompatible parts if there are any. However, Schemathesis could be too strict in terms of data generation which leads to these results. You might want to disable health checks or adjust the schema a bit so it is easier to generate (eg adding additionalProperties:false leads to easier generation). At the moment I can’t give more details, but I’ll get back to it in the beginning of the next week

Thanks for the fast response @Stranger6667 !
For now, I'll add the additionalProperties: false tag to the parameter.

Also, please can you tell me how to disable health checks inside schemathesis github action (and the CLI).
My main goal is to run these tests inside a github action script.

[AryanGodara]

Hello @Stranger6667 , is there any progress on this one?
I need it to merge a PR 😅

[Stranger6667]

Thank you for the heads up!

Also, please can you tell me how to disable health checks inside schemathesis github action (and the CLI).

You can use the following option for CLI:

--hypothesis-suppress-health-check=filter_too_much,too_slow

Note, that you can supply multiple values via a comma. For GitHub Action it could be:

args: '--hypothesis-suppress-health-check=filter_too_much,too_slow'

Let me know if it works for you! :)

[AryanGodara]

Thank you for the heads up!

Also, please can you tell me how to disable health checks inside schemathesis github action (and the CLI).

You can use the following option for CLI:

--hypothesis-suppress-health-check=filter_too_much,too_slow

Note, that you can supply multiple values via a comma. For GitHub Action it could be:

args: '--hypothesis-suppress-health-check=filter_too_much,too_slow'

Let me know if it works for you! :)

I'm AFK at the moment, I'll update you on this, as soon as I get back.
And again, thanks for the prompt reply @Stranger6667 !! :)

[AryanGodara]

@Stranger6667 I'm still getting a similar error. (not able to create tests). It's the same endpoint, and everything else is running (I checked the endpoints manually)

st run --base-url=http://localhost -H "Content-Type: application/json" -H "Authorization: Bearer $ACCESS_TOKEN" --hypothesis-suppress-health-check=filter_too_much,too_slow --validate-schema=true ./api/openapi/http.yml
========================================================================= Schemathesis test session starts ========================================================================
Schema location: file:///home/aryan/mainflux/api/openapi/http.yml
Base URL: http://localhost
Specification version: Open API 3.0.1
Workers: 1
Collected API operations: 2

POST /channels/{id}/messages E                                                                                                                                               [ 50%]
GET /health E                                                                                                                                                                [100%]

====================================================================================== ERRORS =====================================================================================
___________________________________________________________________________ POST /channels/{id}/messages __________________________________________________________________________
hypothesis.errors.Unsatisfiable: Unable to satisfy schema parameters for this API operation

___________________________________________________________________________________ GET /health ___________________________________________________________________________________
hypothesis.errors.Unsatisfiable: Unable to satisfy schema parameters for this API operation

Add this option to your command line parameters to see full tracebacks: --show-errors-tracebacks

If you need assistance in solving this, feel free to join our Discord server and report it:

  https://discord.gg/R9ASRAmHnA

===================================================================================== SUMMARY =====================================================================================

No checks were performed.

Hint: You can visualize test results in Schemathesis.io by using `--report` in your CLI command.

[AryanGodara]

Also, is this the correct way to pass multiple arguments inside the GitHub Workflow :-
(YAML becomes invalid this way, due to duplicate key value)

- name: Run st for bootstrap.yml
        if: always()
        uses: schemathesis/action@v1
        with:
          schema: './api/openapi/bootstrap.yml'
          base-url: 'http://localhost'
          args: '-H "Authorization: Bearer {{ env.ACCESS_TOKEN }}"'
          args: '--hypothesis-suppress-health-check=filter_too_much,too_slow'
          token: ${{ secrets.SCHEMATHESIS_TOKEN }}

Or do I pass them inside a single args: '-H ...' , '--hypothesis...' like this?

[Stranger6667]

Hi @AryanGodara,

That looks like a bug. Interestingly, removing the bearer token makes Schemathesis generate data. Let me check it in detail today

[Stranger6667]

It could be this way:

args: '-H "Authorization: Bearer {{ env.ACCESS_TOKEN }}" --hypothesis-suppress-health-check=filter_too_much,too_slow'

[AryanGodara]

Hi @AryanGodara,

That looks like a bug. Interestingly, removing the bearer token makes Schemathesis generate data. Let me check it in detail today

I also noticed that, even when passing empty/ or no bearer token
Tests are still passing (even in the range of 70-90%), even though the endpoints would all be returning StatusErrAuth.
It's confusing to figure this one out 😅

[Stranger6667]

I guess that StatusErrAuth is the expected outcome from the API schema point of view, hence it is not an error. However, I believe Schemathesis should output a warning regarding the number of 401. Do you observe such warnings?

[AryanGodara]

I guess that StatusErrAuth is the expected outcome from the API schema point of view, hence it is not an error. However, I believe Schemathesis should output a warning regarding the number of 401. Do you observe such warnings?

Yes, there are warnings similar to, "Most of your responses are 4xx"

[Stranger6667]

Hi @AryanGodara

The issue has been fixed in 3.19.7 let me know if the fix works for you

[AryanGodara]

@Stranger6667 How do I update the workflow yaml file to use version 3.19.7?
Or does it automatically download the latest version?

[Stranger6667]

It defaults to latest

[AryanGodara]

Ahh, thanks @Stranger6667 ! Let me test it again and report back to you

[AryanGodara]

For some reason, after updating the args, the schemathesis workflow just isn't starting.
Can you please take a look, if there's a common issue here @Stranger6667

name: Test OpenAPI using Schemathesis

on: [push, pull_request] 

env:
  MF_TOKEN: abcd
  EMAIL: admin@example.com
  PASSWORD: 12345678
  DEVICE: mf-device

jobs:
  api-tests:
    runs-on: ubuntu-20.04
    steps:
      - name: checkout repo
        uses: actions/checkout@v2

      - name: find number of cores
        run: lscpu|egrep 'Model name|Socket|Thread|NUMA|CPU\(s\)'

      - name: create images
        if: always()
        run: make all -j 2 && make -j 2 dockers_dev

      - name: run containers
        if: always()
        run: make run

      - name: REST API with curl to provision user
        if: always()
        run: |
          curl -sSiX POST http://localhost:9002/users -H "Content-Type: application/json" -d '{"name": "John Doe","credentials": {"identity": "admin@example.com","secret": "12345678"}}'
      
      - name: Set ACCESS_TOKEN
        id: set-access-token
        run: |
          response='{"access_token":"..." ... }' # Replace with your JSON response
          access_token=$(echo "$response" | grep -o '"access_token":"[^"]*' | cut -d'"' -f4)
          echo "ACCESS_TOKEN=$access_token" >> $GITHUB_ENV

      - name: Set BEARER_TOKEN
        run: echo "BEARER_TOKEN=rer $MF_TOKEN" >> $GITHUB_ENV

      - name: print bearer token
        if: always()
        run: echo $BEARER_TOKEN
      
      - name: Set token env variable
        run: echo "MF_TOKEN=$(echo $MF_TOKEN)" >> $GITHUB_ENV

      - name: Test
        run: echo ${{ env.ACCESS_TOKEN }}
      
      # Bootstrap.yml
      - name: Run st for bootstrap.yml
        if: always()
        uses: schemathesis/action@v1
        with:
          schema: './api/openapi/bootstrap.yml'
          base-url: 'http://localhost'
          args: '-H "Authorization: Bearer {{ env.ACCESS_TOKEN }}" --hypothesis-suppress-health-check=filter_too_much,too_slow'
          token: ${{ secrets.SCHEMATHESIS_TOKEN }}
        
      # Certs.yml
      - name: Run st for certs.yml
        if: always()
        uses: schemathesis/action@v1
        with:
          schema: './api/openapi/certs.yml'
          base-url: 'http://localhost'
          args: '-H "Authorization: Bearer {{ env.ACCESS_TOKEN }}" --hypothesis-suppress-health-check=filter_too_much,too_slow'
          token: ${{ secrets.SCHEMATHESIS_TOKEN }}

[AryanGodara]

@Stranger6667 Can you please check this above comment. The health check is working locally in the CLI now, but not working in the workflow

[Stranger6667]

@AryanGodara

What does it output? All looks fine at the first glance