Compatibility Acceptance Tests

[OlegIlyenko]

After creating scala GraphQL implementation I must admit, that the test quality and coverage of nodejs reference implemention is very good. In fact it helped me a lot during the development (as well as the reference implementation itself, of course). After re-implementing most of the tests in scala I now pretty confident, that most of the functionality is tested.

After seeing all these tests it become apparent to me, that there are a lot of edge cases and things that GraphQL library can get wrong, especially in terms of semantics, validation, execution, error handling, etc.. People are already created implementations in many different languages. I feel that it is important to create a TCK (Technology Compatibility Kit) for GraphQL specification in order to ensure success of GraphQL itself, higher adoption and growth of the ecosystem. Having implemented other specifications in the past, I must admit that a TCK makes specification implementation much easier and faster process. It also helps to adopt an implementation as specification evolves.

With TCK I mean a set of acceptance tests, that are distributed in some form (as a library, for instance) and can be used by library authors to verify, that their implementation confirms to the spec. Of course, TCK can't cover all of the cases, but at least it gives a good baseline and as ecosystem grows, it ensures that all parts of it remain compatible with each other or at least minimizes risk of incompatibility.

At the react-europe conference I had a short conversation with @schrockn about this topic. He said, that there are plans for something like this, so I wanted to follow-up here and ask for more information. Do you still have plans to create something like a TCK for GraphQL spec? If yes, do you already have an idea which form it will take? I guess implementing something like this can be a challenging task, considering that it suppose to test implementations written in different languages. As far as I saw, tests in the reference implementation are pretty generic - many of them take some query, execute it and then verify resulting JS object. At the moment they all implemented in terms of the nodejs implementation, but maybe they can be reused or extracted in the TCK somehow?

[leebyron]

Thanks, for opening this. I'm renaming the task to use the terminology we have been using to talk about this.

We agree that this is really important, but are not yet actively working on it.

[leebyron]

do you already have an idea which form it will take?

Sort of.

I think it will specify a schema to implement, along with a data set to use. Ideally this schema is as small as possible while still illustrating every feature of GraphiQL and supporting all of the acceptance tests. Ideally this schema is written in the GraphQL Type DSL so it is easy to understand.

Then there should be a list of "tests" where each test includes the following:

• Description: what is this test testing and why
• Query: a document to submit to the GraphQL service
• Response: The response expected from the GraphQL service

Then optionally also include Variables and OperationName when those are relevant to a test.

This form should allow us to test everything from parser corner cases and proper detection and reporting of syntax errors, to validation errors for all the rules we specify, to execution behavior.

[leebyron]

I guess implementing something like this can be a challenging task, considering that it suppose to test implementations written in different languages.

Ideally, the form I wrote above only needs to be written once and can leverage the fact that the GraphQL language is represented as a string independent of any one backend language and the output can be serialized to JSON.

However the provided schema still needs to be implemented by that backend. Either by interpreting the GraphQL Type DSL or by manually implementing it. Hopefully that is not too much of a burden.

However overall I would certainly rate this task as "challenging" :D

[OlegIlyenko]

Thanks for the reply! Glad to hear that you already thought about it. The approach you described sounds good to me. I was also thinking about something like this.

About the schema, reference implementation now has a schema materialiser, which also can be implemented in other languages. Resulting schema does not contain the resolve method, but even in this form it can be used for many tests that do not require execution, like query validation and parsing or introspection. As far as I saw, even execution tests have very similar resolve implementations, which are in most cases fall in one of these categories:

• successful resolve with null
• successful resolve with not-null value
• successful resolve of promise with null
• successful resolve of promise with not-null value
• exception is thrown
• promise was resolved with failure

Maybe we can create an enum of possible implementation behaviours and then enhance textual schema representation to include this information in some form. This will allow schema materialiser to create simple implementations of resolve method. This means that a test schema can be completely generated from textual description, just like test cases. What do you think about this idea?

[OlegIlyenko]

We started to work on compatibility acceptance tests here:

https://github.com/graphql-cats/graphql-cats

If you are working on a GraphQL implementation, it would be great if you could share your opinion and join us.

[leebyron]

I'm going to close this issue since graphql-cats is well underway!