ndc-test

NDC Test Runner

The test runner can be used to validate and test a connector during development. It provides two functions:

• Automated testing, based on requirements from the specification
• Manual testing, in the form of a generic snapshot-based testing tool, for connector-specific tests which would not be covered by the specification.

The automated tests are not comprehensive, and should be treated as a quick validation of the basic functionality of a connector. As an example, the test runner will generate some simple equality predicates in order to test the predicates functionality, but cannot generate an exhautive set of test cases covering all possible predicates.

In order to properly validate a connector for release, authors should augment the automatic test suite with some custom tests.

Getting Started

Test a connector

cargo run --bin ndc-test -- test --endpoint http://localhost:8100

To modify the random seed used to generate test data, use the --seed argument with a 32-character string:

cargo run --bin ndc-test -- test --endpoint http://localhost:8100 --seed 'ABD1FFEA148FE165FAC69B66B58972A8'

Generate/replay snapshot tests

ndc-test generates sample query requests which are issued against a connector. These requests can be saved to disk using the --snapshot-dir argument:

cargo run --bin ndc-test -- test --endpoint http://localhost:8100 --snapshots-dir snapshots

If the files already exist on disk, they will be validated against the actual responses received. If not, new snapshot files will be written.

Note: different values for --seed can generate different snapshot tests.

To replay existing snapshot tests from disk without regenerating the requests, use the replay command:

cargo run --bin ndc-test -- replay --endpoint http://localhost:8100 --snapshots-dir snapshots

Customizing the test generation strategy

Several options are available to customize the test generation:

Option	Description	Default
-x, -xx, ...	Increases the complexity level of generated queris in general, e.g. generates more deeply-nested boolean expressions	None
--test-cases	The number of test cases to generate per scenario	10
--sample-size	The number of example rows to fetch from each collection	10
--max-limit	The maximum number of rows to fetch per test query	10

Custom tests

Connector developers can add custom query and mutation snapshot tests to the snapshot directory. The test command does not generate mutation tests by default, because it is up to the connector developer to ensure a reproducible test environment.

For example, to run the existing suite of snapshot tests for ndc-reference, we can use the replay command (assuming the reference connector is running on port 8100):

cargo run --bin ndc-test -- replay --endpoint http://localhost:8100 --snapshots-dir ndc-reference/tests

Benchmarks

ndc-test also implements a basic benchmarking tool, to test and monitor the performance of your connectors. The bench command accepts a snapshot directory (generated by the test command, or by hand), and issues each query several times to collect some basic runtime statistics (mean, min, max and standard deviation).

To benchmark the reference connector:

cargo run --bin ndc-test -- bench --endpoint http://localhost:8100 --snapshots-dir ndc-reference/tests

Benchmark reports will be stored alongside the query request JSON files, as report.json.

To test for performance regressions, use the --tolerance argument. For example, to fail if any test averages 10% longer than the previous recorded mean time, use --tolerance 0.1.