[BUG] Generating invalid HTTP headers #1142
Comments
Lakitna
added
Status: Needs Triage
Stranger6667
added
Difficulty: Beginner
|
Cool! Thank you for the detailed report! :) I will take a look today |
|
I've skimmed through RFC 7230, and it seems like Having
Which gives the total interval For such headers, schemathesis uses this Hypothesis strategy - Maybe I missed something, let me know what do you think :) |
|
Hmm reading a bit in 3.2: 3.2.4:
Where ISO-8859-1 = I think I'm feeling that one. It looks like I'm seeing this behaviour with everything > Maybe some servers implement the full I ran some more tests. It looks like my server returns a generic 400 with an empty body and no content type for the following situations: API key header == And all the following values in the API key headerI ran 1000 tests, so I'm pretty confident that I've got all cases where my server fails. The server I'm using by the way is a local Azure Functions instance with an HTTP trigger. Edit: Conclusion after looking at this for too long... The accepted character range of headers for my server is |
|
Hmm, interesting. If Schemathesis will have a way to customize this behavior - will it help your use case? To adjust the range for possible characters in HTTP headers. Btw, I noticed that at the moment, the strategy for the |
|
Yeah that would help for me. One extra thing I noticed though, Postman also does not support all these characters in headers. For example:
I would be happy with an option to reduce the char range. At the same time, I think it's worth thinking about future users too. Not having them go through this would be preferable. That being said, I do see the value of generating with the full range for more security-focused tests. So, how about an option to change the char range with a default of |
I think it is a reasonable thing to do if on average it will increase the Schemathesis effectiveness with default settings. I don't have such numbers at the moment, but I am working on an evaluation suite for the "Property-Based Testing of Web APIs" paper. As soon as it is ready I'll look at this aspect - so far I didn't see popular stacks from Python / Rust / Go / JavaScript worlds complaining about that matter, but certainly will re-check that more thoroughly. |
It's a first for me too, but now that I've seen it once... I would have never thought to check Postman for this stuff before!
That's the goal! This got me thinking though. You'll know better how possible this is. When running into an issue like this, can you identify it as an 'Only accepts Latin-1 in headers' situation automatically? If that's possible, you can easily keep the full char range and instruct users to set a narrower range when they're using a server like mine. It would make it a conscious choice to change the range, without the headaches I encountered while troubleshooting.
Sounds interesting! workaroundFor now, I've wrapped def validate_response(case: schemathesis.models.Case, response: schemathesis.models.Response):
"""
Validate the response. Includes workaround for Azures default 400.
Normally we return a 400 with a Problem, this is a workaround because we can't always control
this.
This is needed because of how schemathesis works. It will try to add all kinds of characters
to the HTTP headers. However, Azure Functions can only handle `0x01` through `0x7f`.
Schemathesis will try to insert other characters as well, causing the OAS validation to fail.
This function lives here pending the open issue on this:
https://github.com/schemathesis/schemathesis/issues/1142
"""
if response.headers.get("content-type") is not None:
# No need for the workaround
return case.validate_response(response)
try:
case.validate_response(response)
raise AssertionError("Expected an exception to be raised")
except schemathesis.exceptions.CheckFailed:
assert response.status_code == 400
headers = response.headers
assert headers.get("content-type") is None
assert response.content == b"" |
I think it depends on the server implementation and how it responds, probably there is no direct way to detect it for Schemathesis automatically. However, I have a prototype that uses Targeted Property-based testing (via Hypothesis's Alternatively, if there is a way to detect some known cases, then Schemathesis can run some requests upfront and then adjust data generation strategies. Could you share a bit more details about your server and how I can reproduce it locally? I'll try to play around with it |
That'd be good, indeed. As long as it still does generate some 4xx codes. But I don't think that should be an issue in most cases. Having a wide variety of situations is the great power of property-based testing. IMO that should translate to a variety of status codes as well. Along the same lines, I have been thinking of enforcing at least one 2xx response per endpoint. Right now, I have a passing test of an endpoint that has not been implemented. All calls will return 404. It's a bit out of scope of my current story though, and I couldn't quickly squeeze it in with hooks (missing afterAll hook).
I guess OASs
I'm running Python Azure Functions locally. I'm using a standard HTTP trigger to make my calls. If you follow the guide below you'll quickly end up with a local Azure Functions app with HTTP triggered function :) |
|
Thanks for providing more context! I'll check this a bit later. Meanwhile, the headers can be filtered with the import schemathesis
@schemathesis.hook
def filter_headers(context, headers):
for name, value in headers.items():
if not in_range(name) or not in_range(value):
return False
return True
def in_range(string):
return all(0x01 <= ord(char) <= 0x7f for char in string)More about hooks - https://schemathesis.readthedocs.io/en/stable/extending.html#hooks |
|
I actually ran into this today with a coworker with Did something change with a recent release? Stacktrace of shrunk error: |
|
Thank you for leaving a comment. It is strange, I will look at it over the weekend. |
|
Sounds good. For now, we've fixed it with the hook ;) |
|
I think I made a mistake in 2418188 when tried to optimize header generation. It removes filtering more cases than it should - it should fallback to filtering. Could you, please share the API schema of your header parameters (just the top-level of the |
|
Hello @Stranger6667 I'm the coworker mentioned by @Lakitna, here are the headers used within our schema. We also insert another header through the commandline which is a string of up to 40 characters long.
|
|
@hoog1511 thanks! |
|
@Stranger6667 the bug seems to have been introduced in version 3.10.0, I reverted to version 3.9.7 and the encoding error seems to no longer appear during testing. Hope this helps. |
|
Thank you, it confirms, that the root cause is this commit. It should be relatively easy to fix, I'll work on this today/tomorrow |
|
FYI, the fix for the recent issue is released in 3.11 |
|
Hello people, I currently have encountered a similar bug in my application. The bug occures when using "Bearer {token}" for authentication. The token is automatically generated by OpenIdConnect while using FastAPI. Is this a known issue already? Is there any fix available? Thanks in advance :) |
|
Now there is a way to control what characters are used for strings (globally, not for headers specifically). I also will consider adjusting the defaults for header generation in Schemathesis 4.0. Meanwhile, if anybody has any issue with header generation, feel free to open a new issue. |
Describe the bug
Schemathesis generates headers like:
The
\x80character is not a part of ISO-8859-1 (AKA Latin-1). This is an issue because according to the HTTP 1.1 standard, all HTTP headers are encoded with ISO-8859-1. This means that my tests will always fail because of something I can't actually control.To Reproduce
Expected behavior
I expect Schemathesis to not generate things that are unsupported due to the HTTP spec. By definition, these things will not have stable behaviour, so it'll be impossible to test with Schemathesis.
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: