gRPC support for Schemathesis ?

[jenstroeger]

It looks to me like Schemathesis has so far focused on testing HTTP/REST APIs specified using OpenAPI schemas.

Are there any attempt to use it for gRPC APIs specified e.g. using a set of protobuf files? I saw hypothesis-pb referenced here but that project seems abandoned.

[Stranger6667]

Hi @jenstroeger

I want to add it eventually, and the first step is scheduled for the next minor release (3.22) - the transport layer will be refactored, so it is easier to use gRPC-specific transports. Then it would need to integrate Hypothesis-specific strategies under another "Schema" kind and expose it everywhere (e.g. in CLI, etc).

P.S. It would be similar to how GraphQL is integrated at the moment

[Stranger6667]

I did some drafts, and it looks like refactoring transports to achieve this kind of flexibility can't be done without breaking changes, so I released 3.22 without them instead. There are a few more things I wanted to do for the 4.0.0 release, but probably they won't block the transport-related work anyway.

With that, do you have an estimate of how long such a feature would take, how much effort would be involved in implementing it?

I'd say it is realistic to ship all the building blocks needed for gRPC support in 3-4 weeks. Here are some quick notes:

• creating a new schema type that supports gRPC-related logic for iterating over different operations, counting them, etc.
• make a loader for it, e.g. from_file so such a schema could be created from a protobuf file.
• implementing get_case_strategy with hypothesis-pb or similar to generate Case instances
• extend the Case class to provide the right transport there.

So, strictly speaking, refactoring transports is not 100% necessary to implement gRPC support, but I wanted to do it in order to avoid making the codebase more complex.

Would you be able to help or guide possible contributions to that feature?

Yes! I'd be keen to do so :)

And: that would include stateful endpoint tests as well, yes?

Most of the things will work out of the box, but we need a way to build that state machine from some input source. I assume there are some definitions for services in the gRPC world, right? if so, they could be used to build a state machine

[jenstroeger]

Thank you for the details, @Stranger6667!

Yes! I'd be keen to do so :)

Alright, let me discuss this with a few folks and I’ll get back to you within the next few days.

I assume there are some definitions for services in the gRPC world, right?

Yup, services and endpoints are declared in .proto files as well, see the Defining Services section in the language guide.

What are your plans wrt. proto 2? Selfishly, I prefer to get started with proto 3 and then — if need be — consider proto 2.

[Stranger6667]

Great!

What are your plans wrt. proto 2? Selfishly, I prefer to get started with proto 3 and then — if need be — consider proto 2.

Indeed, I'd start with v3 - likely it will be a separate library anyway (like hypothesis-graphql), so the support could be added later on. Also, to be clear, I'd prefer to keep this potential hypothesis-* library under the schemathesis org umbrella.

[Stranger6667]

Probably, parsing the service definition still should happen here too - i.e. hypothesis-protobuf will generate entities based on the .proto file input (parsed / raw), and then schemathesis will use it to generate test cases for rpc calls. Those rpc calls definitions will be taken from the same .proto file. AFAIK usually the compiler generates some Python code that is supposed to be used later on, but I am not yet sure what would be the most flexible approach here. Hopefully you can guide me on that :)

P.S. Here you can see how little experience I have with gRPC :)

[jenstroeger]

P.S. Here you can see how little experience I have with Schemathesis internals :)