Connect Conformance

A test suite for Connect that verifies cross-platform conformance for both clients and servers.

Summary

The Connect conformance test suite is a series of tests that are run using a client and server to validate interoperability, compatibility, and conformance across the Connect, gRPC, and gRPC-Web protocols. The test suite is meant to exercise various scenarios with a client-server interaction to ensure the results are as expected across platforms.

Tests are divided into two types: client tests and server tests. Those which verify clients are run against a reference server implementation of the Conformance Service written with connect-go.

Likewise, servers under test will be verified by a reference client implementation of the Conformance Service also written with connect-go.

To verify compatibility with other protocol implementations, the conformance tests also use reference client and server implementations that use the gRPC-Go module and a reference server implementation that uses the gRPC-Web Go server.

Documentation

Detailed guides can be found in the "docs" sub-directory of this repo. If you first want to read a shorter overview, see the next section below. But when you really get started with any of these tasks, you'll want to read one or more of these guides.

• Configuring and Running Tests
• Testing Server Implementations
• Testing Client Implementations
• Authoring New Test Cases

Testing your implementation

Setup

The conformance runner has the ability to test a client, a server, or both simultaneously. This means that if you are validating both a server and a client, you can use the conformance suite to run them against each other. Testing either a client or server in isolation will use the corresponding reference implementation to verify conformance.

Below are the basic steps needed for setting up the suite to run against your implementation:

1. The first step is to access an SDK for the Conformance protos. These can be found on the Buf Schema Registry: https://buf.build/connectrpc/conformance. You can download SDKs for some languages here.

   If you are using a language that is not supported by the BSR's selection of SDKs, you can generate one yourself using the buf command-line tool. After creating a buf.gen.yaml file, to configure the code generation, you'll then run buf generate buf.build/connectrpc/conformance.

2. Once you have an SDK, with generated code for the Conformance Service and related messages, you will need to implement either the service, the client, or both (depending on which you are testing). To do so, follow the instructions specified in the ConformanceService proto.

   For working examples, refer to the Go reference client and reference server.

3. Your service-under-test or client-under-test needs to be a program that can easily be invoked from the command-line. This is how the conformance test runner will invoke it, too.

4. Next, visit this repository's Releases page and download the conformance runner binary: connectconformance. You may want to add it to your $PATH to make it easier to run interactively from the command-line.

5. Finally, integrate the conformance tests into the continuous integration and testing process for your code, so every change you make can validate that the implementation remains conformant. You can see an example of how this can be done using make in the connect-kotlin repo, here and here.

Running the tests

The commands for testing will depend on whether you are testing a client, a server, or both. Specifying which implementation is done via the mode command line argument.

Once you have completed setup, the following commands will get you started:

Testing a client

connectconformance --mode client -- <path/to/your/executable/client>

Testing a server

connectconformance --mode server -- <path/to/your/executable/server>

Testing both a client and server

To test your client against your server, specify a mode of both, with the client path first, followed by ----, then the path to your server:

connectconformance --mode both -- <path/to/your/executable/client> ---- <path/to/your/executable/server>

Running the reference tests

To test this repo and the reference clients and servers, we can use the conformance suite itself. To run the suite, using the reference clients against the reference servers, and see the process in action, use the following command:

make runconformance

This will build the necessary binaries and run tests of the following implementations.

• Connect reference client and reference server
  • These implementations are written using connect-go, but with numerous extensions that allow them to more closely examine the on-the-wire format of the RPC protocol to make stronger assertions about conformance.
  • They support all features that can be tested by the conformance tests, which includes all three protocols (Connect, gRPC, gRPC-Web), all HTTP versions (HTTP 1.1, HTTP/2, and even HTTP/3), and a variety of compression encodings ("gzip", "br", "zstd", "deflate" and "snappy").
• gRPC client and server
  • These implementations are written using grpc-go.
  • They support the gRPC protocol and HTTP/2. They only support the "proto" codec for message encoding and the "gzip" compression encoding.
  • The server also supports the gRPC-Web protocol and HTTP 1.1 using the improbable-eng/grpc-web Go implementation. This implementation is listed in the official gRPC-Web documentation as a server/proxy option here. (Note that the gRPC client does not support gRPC-Web.)
  • These implementations are also used against clients-under-test and servers-under-test, to confirm interoperability with official gRPC implementations.

Both of the above clients are tested against both the Connect reference server and the gRPC server. The servers are tested against the Connect reference client and the gRPC client. And since the gRPC client does not support gRPC-Web, the servers are also tested against the official gRPC-Web JS client.

Status: Stable

This repo is stable and follows semantic versioning.

We will not make breaking changes to the command-line interface, the YAML config formats, or the Protobuf messages used by clients and servers under test in the 1.x series of releases.

Note, however, that we reserve the right to rename, remove, or re-organize individual test cases, which may impact the "known failing" and "known flaky" configurations for an implementation under test. We will document these changes in the release notes.

Ecosystem

Implementations

• connect-go: The Go implementation of Connect
• connect-es: The TypeScript implementation of Connect
• connect-kotlin: The Kotlin implementation of Connect
• connect-swift: The Swift implementation of Connect

Examples

• examples-go: Example RPC service powering https://demo.connectrpc.com and built with connect-go
• examples-es: Examples for using Connect with various TypeScript web frameworks and tooling

Ancillary

• connect-playwright-es: Playwright tests for your Connect application
• connect-query-es: TypeScript-first expansion pack for TanStack Query that gives you Protobuf superpowers

For more on Connect, see the announcement blog post, the documentation on connectrpc.com (especially the Getting Started guide for Go), or the demo service.

Legal

Offered under the Apache 2 license.