Redesign CLI

[Stranger6667]

It turned out that the current CLI design is pretty stiffed and doesn't easily allow adding new features (e.g., adding the "Stateful testing" feature is painful). Other things:

• A lot of not useful info. Users don't care where Hypothesis stores its database; it is an implementation detail that eats a lot of space. The same goes for internal dependency versions, rootdir, etc.
• GraphQL integration. Everything in the current design implies Open API - many options are specific to it (--method, --operation-id, etc.). Adding GraphQL may be done with adding another option with choices & ignoring other options, but it just another addition to the pile of technical debt :(
• Statistic is confusing. Especially numbers - sometimes they hardly can be connected with the number of operations or the value of the --hypothesis-max-examples option. This is connected with how Hypothesis works but still confusing for the end-user.
• No machine-readable output. It can be done with the current design, though.
• Performance. It was tricky to make the VCR cassettes work fast but still suboptimal

It was mostly copied from pytest, but probably it is not the best fit for the Schemathesis workflows :(

I want to redesign the Runner / CLI pair so it is more consistent & task-oriented.

[Hultner]

Would it make sense to have sub commands for openapi and graphql?

E.g.

$ schemathesis graphql [... [url]]
$ schemathesis openapi [... [url]]

[Stranger6667]

Would it make sense to have sub commands for openapi and graphql?

It is definitely an option to consider. I am not entirely sure what will be the most intuitive way to use Schemathesis, but this approach adds an additional step for the user - explicitly specify the spec. It is not much, though, but on the other hand having a single command that tests an API is cool, no need to worry about the spec version, etc. But subcommands are more explicit.

Yesterday I was also thinking about auto-detecting GraphQL - which may also be an option, which in the worst case will cost multiple requests to the schema (there will be some fallback options)

[Hultner]

Ohh I like the auto detect idea, but it would be good to have the possibility to explicitly override the auto detect as well. The override could be something like --spec=openapi / -s openapi with an optional version.

Auto detection is far superior UX imho.

Something else that would be really nice is man-pages. I love well written man-pages, frequently saves me from a context switch by round tripping google via the browser.

[Hultner]

@Stranger6667 Have you seen this site?
https://clig.dev/

I don’t agree with everything 100% but it does make some good points worth considering.

[Stranger6667]

Ohh I like the auto detect idea, but it would be good to have the possibility to explicitly override the auto detect as well. The override could be something like --spec=openapi / -s openapi with an optional version.

Yep, overriding should be available, I recently added force_schema_version, which allows overriding some autodetection for Open API schemas, but it may be extended, surely.

Auto detection is far superior UX imho.

Agree. Having to pay an extra HTTP call (in the worst case) is basically nothing :)

Something else that would be really nice is man-pages. I love well written man-pages, frequently saves me from a context switch by round tripping google via the browser.

Great idea! I am not using them much, but it definitely will improve the UX (probably something like https://github.com/click-contrib/click-man can be used)

Have you seen this site? https://clig.dev/

No, thank you for suggesting :) Yesterday I found some links relevant to the CLI design (all of them are mentioned at the end of this one). Now, after reading it, I have a long list of things to investigate deeper :)

[Stranger6667]

This issue is quite outdated, going to close it and open more concise ones if needed.