Getting the error as Invalid Schema, but extremely difficult to understand what the issue is

[tl-madhulika-mitra]

I am getting the below error when trying to run Schemathesis on my Open API spec version 3 definition

Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/schemathesis/specs/openapi/schemas.py", line 206, in get_all_operations
yield Ok(self.make_operation(path, method, parameters, raw_definition))
File "/opt/homebrew/lib/python3.9/site-packages/schemathesis/specs/openapi/schemas.py", line 256, in make_operation
self.security.process_definitions(self.raw_schema, operation, self.resolver)
File "/opt/homebrew/lib/python3.9/site-packages/schemathesis/specs/openapi/security.py", line 20, in process_definitions
if definition["type"] == "apiKey":
KeyError: 'type'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/schemathesis/specs/openapi/schemas.py", line 215, in _into_err
raise InvalidSchema(SCHEMA_ERROR_MESSAGE, path=path, method=method, full_path=full_path) from error
InvalidSchema: Schema parsing failed. Please check your schema.

Steps to reproduce the behavior:

1. Run this command st run --checks all <> --base-url <> --show-errors-tracebacks

I am sorry, unable to provide to api spec, but it could be nice to understand what might be the reason for such an error. Note, we have oneOf, allOf etc in our spec def.

Environment (please complete the following information):

• OS: Mac M1
• Python version:3.9
• Schemathesis version: [3.15.2]
• Spec version: [Open API 3.0.1]

[Stranger6667]

Hi @tl-madhulika-mitra

Thanks for opening the issue, that is indeed a bad error report from Schemathesis. As far as I see, the underlying issue is that some of the security schemes miss the "type" key. You might want to run Schemathesis with --validate-schema=true first, which will output errors from the JSON Schema validator, which may not be much helpful as well :(

At the moment I plan to rework schema validation on the SaaS side, so it outputs precise locations and what is missing in the schema. Though, there is no ETA for it

[tl-madhulika-mitra]

Just a note, I realised I was using a spec file here, which was in turn trying to build from more $ref files in my projects, which Schemathesis mostly likely might not have been able to access, not super sure, but strangely, some apis still seemed to have work, while some did not :) . When I tried to test with the detailed schema which did not need anything externally but was the final Yaml, Schemathesis worked its way through it. Yet having a nicer error will always help.

Thank you

[Stranger6667]

Thanks for more context! Can it be that you have $ref inside one of your security scheme definitions? Not inside the securitySchemes key, but inside individual items there?

[tl-madhulika-mitra]

1. Thanks for the revert. Yes you are right, a few $ref are under securitySchemes but most of them are not. So you think this caused the error?

2. Also besides, when I do run for the fully declared file, there are 1 error I encounter: What does this essentially mean?
   Note that these apis have OneOf, AllOfs, so is it like the schema is complex and hence example generation is difficult.

Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/schemathesis/runner/impl/core.py", line 294, in run_test
test(checks, targets, result, errors=errors, headers=headers, **kwargs)
File "/opt/homebrew/lib/python3.9/site-packages/schemathesis/runner/impl/core.py", line 507, in network_test
case: Case,
File "/opt/homebrew/lib/python3.9/site-packages/hypothesis/core.py", line 1235, in wrapped_test
raise the_error_hypothesis_found
File "/opt/homebrew/lib/python3.9/site-packages/hypothesis/internal/healthcheck.py", line 27, in fail_health_check
raise FailedHealthCheck(message, label)
hypothesis.errors.FailedHealthCheck: Data generation is extremely slow: Only produced 5 valid examples in 1.02 seconds (0 invalid ones and 1 exceeded maximum size). Try decreasing size of the data you're generating (with e.g. max_size or max_leaves parameters).
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.too_slow to the suppress_health_check settings for this test.

3. At the end of the run we see a report like this - Is there a way to run what kind of tests actually ran?
   Performed checks:
   not_a_server_error 358 / 358 passed PASSED
   status_code_conformance 315 / 358 passed FAILED
   content_type_conformance 241 / 358 passed FAILED
   response_headers_conformance 358 / 358 passed PASSED
   response_schema_conformance 263 / 358 passed FAILED

[Stranger6667]

a few $ref are under securitySchemes but most of them are not. So you think this caused the error?

Most likely so, because the type key is required there. Schemathesis resolves only the top-level $ref in securitySchemes because the spec defines it there, but it doesn't do this on each individual scheme because the spec doesn't expect references there. I think Schemathesis can just resolve it recursively there to avoid such cases altogether.

What does this essentially mean? Note that these apis have OneOf, AllOfs, so is it like the schema is complex and hence example generation is difficult.

Exactly as you describe it - the example generation is too slow and Hypothesis raises an error because of it. There could be different causes on the schema level. Not necessarily oneOf / allOf are causing this - in many cases they are canonicalised to simpler forms and don't have any performance implications. Hard to say without seeing the schema, but I often observe it when the schema is too loose / too nested. For example, having additionalProperties: false helps, because Schemathesis stops generating arbitrary properties for object nodes in the schema where this keyword is defined.

At the end of the run we see a report like this - Is there a way to run what kind of tests actually ran?

Schemathesis executes checks on each received response and then calculates these total values. Some checks are implemented here, and here are the docs on how they work.

Let me know if you need more info :)

P.S. You mentioned that you can't provide the spec, but if it would be suitable, I can take a look it privately (e.g. in Discord DM). Then I can say more details :)

[Stranger6667]

I think that the steps to improve it are:

• Run JSON Schema validation if we encounter any sort of schema errors and report the problem. In such cases it is caused by missing required fields, so JSON Schema validation should find it
• In case JSON Schema does not find it, it is likely a bug in Schemathesis and there should be an appropriate message about it.