[FEATURE] Custom request data in CLI

[Stranger6667]

Problem

Currently, Schemathesis CLI doesn't allow the user to directly set specific values for different elements in generated API requests such as query parameters, cookies, and body. Only headers are possible directly (via the -H CLI option) and the rest are only possible with Python hooks.

Solution

Add a set of new CLI options, --set-<location> to allow users to set fixed values in API requests. The location could be header, query, cookie, or body.

Examples for header, cookie, and query:

st run --set-header 'Authorization:Bearer 12345' ...
st run --set-cookie 'session_id:bar' ...
st run --set-query 'foo:bar' ...

The syntax above is what -H uses now for headers, so I believe it could be a simpler transition that provided a unified way to set values there.

{name}:{value}

Multiple options could be specified too!

NOTE: Implementing this feature effectively deprecates the -H option which is inherited from cURL and users are familiar with it. But I think it is more important to have a separate namespace for customizing the generated data via CLI which will unify all possible request elements.

For body it is a bit different - no need to specify the name part:

schemathesis run --set-body '{"first_name": "John", "last_name": "Doe"}' ...

Future extensions

I think it could be a minimal version for now, just set the whole body. In the future we might have something like this to set specific values within e.g. a JSON value:

schemathesis run --set-body-at 'foo[3].bar:42' ...

and it will take the foo field, then its 4th element, and set the bar field of this element to 42. However, such a feature requires filling some gaps within the value when e.g. there are not enough elements or they don't have such fields, or maybe they even have different types! These cases are important to resolve upfront before implementing this extension. Another aspect of it is that the "light" version of this feature could work by modifying the schema with some "const" keywords, but this extended version can not in the general case (as the generation of complex structures could be affected by many different keywords).

To discuss

• Should a space be allowed after :? What if the user wants to pass a space as a part of the value?
• Setting values for specific endpoints. Probably it could be nice to have, but I think that having a config file is better for such a scenario, otherwise CLI could be quite overloaded and hard to manage.
• Add --set-form-data for Open API 2 with the same syntax as for query / header / cookie?
• What about path parameters?
• Usage in the pytest integration? It feels less relevant as usually users just set desired values directly on the generated Case instance. This is basically what Ability to pass parameters directly #8 suggests.

Implementation details

• Extend CLI with new parameters and collect them into a separate entity
• Pass this entity through the runner and when it comes to the Case generation via the get_case_strategy modify the schemas, relevant to these components.
• Open API: It could be adding a new property with a const schema inside properties and extending the required list. For negative cases, it should come after mutation.
• GraphQL: Just modify the generated case (or skip the generation step if body is provided)
• Add deprecation notice for -H

Related issues

• [FEATURE] CLI option for queries #1639
• Ability to pass parameters directly #8

[hoog1511]

Being able to provide certain values to always be used in your tests seems like an excellent idea to me. 
I personally use schemathesis for Bi-Directional contract testing where I need to verify that an APIs request/response pairs function as is documented in an OpenAPI specification. Due to the complexity of most of the APIs, I work with it is quite difficult to get test cases generated for some of these responses using schemathesis.
Having the option to hardcode some of the values probably will assist in getting some better testcases for my situation.

[Stranger6667]

@hoog1511 thank you for sharing your use case and context! I have a few questions:

• Which specific types of values do you need to be able to hardcode? E.g. headers / query parameters / etc
• How do you currently handle such use cases, if at all?
• Do you anticipate any potential downsides, such as increased maintenance or difficulty in updating hardcoded values?
• Do you think you'd need to dynamically change these hardcoded values during a test run? or e.g. provide N possible options to choose randomly

[hoog1511]

Hey @Stranger6667 here are my answers to your questions.

Which specific types of values do you need to be able to hardcode? E.g. headers / query parameters / etc
Currently Schemathesis already allows the "hardcoding" of specific headers in the CLI. I would like a similar function for queries (I made a feature request for that #1639). I have a use case where a query value decides from which environment a request is performed and I would like to provide the value for this in the CLI command instead of providing it with a enum value.
Perhaps having the option for providing path parameters in the CLI might also be useful for testing. I have had some use cases where there are specific items in a database which are for testing purposes (other items are added/removed depending on the time of day) and to interact with these item I had to provide specific path parameters using enum.

How do you currently handle such use cases, if at all?
By providing these values in an enum in my OAS and in some cases this enum is dynamically updated using a SED command prior to the Schemathesis test.

Do you anticipate any potential downsides, such as increased maintenance or difficulty in updating hardcoded values?
For my own use cases I expect no downsides or extra difficulty.

Do you think you'd need to dynamically change these hardcoded values during a test run? or e.g. provide N possible options to choose randomly
Yes, there are some use cases where I will need to dynamically change the values prior to the test run.

Yes, I think there could eventually be some use cases where I have to provide multiple hardcoded values to test various example situations (where I do not want to provide them in the OAS for security reasons). Perhaps the option for providing a file containing multiple values for a header, query or path parameter in the CLI might be a viable solution for this.