Tech stack and operational aspects for the test suite

[spawnia]

As discussed in the last working group meeting, there is strong interest in providing a conformance test suite to go along with the spec.

Fundamentals

In order to not limit what kind of tests we can run, i narrowed down the choice to testing frameworks that allow running actual code. Some classes of tests are really hard/impossible to implement using only static request/response pairs.

JavaScript TypeScript seems like the obvious choice for the language. It is basically the lingua franca of the web and the most popular GraphQL implementations, including the reference implementation, are written in it.

Organization

To organize the tests, we might just use a test runner such as Jest or Mocha, if we want to stick with pure JavaScript. I think that would get us up and running very quickly.

Another route we could go would be to define the tests more abstractly, using a BDD tool such as https://gauge.org/ or http://cucumber.io/. That would allow us to define the test scenarios in more readable plain text and hook up the executable tests under the hood.

Assertions

For making the actual assertions, https://github.com/visionmedia/supertest seems like a popular and practical choice. Haven't really found much more in that regard, open to suggestions.

[IvanGoncharov]

@spawnia Thanks for volunteering 👍
One more factor: GraphiQL team moved their codebase to TS + Jest and I plan to do the same for graphql-js/express-graphql and since some features (e.g. batching) involve modifying graphql-js, express-graphql, GraphiQL and our future test suite it would be great if we keep it unified.
That said I'm totally for using some other stack if it will provide significant benefits.

[spawnia]

Oh yeah, when i say JavaScript i actually mean TypeScript.

I am going to try out Gauge and Cucumber for a bit and see if they actually simplify writing tests. If not, the combination of Jest + TS looks like a solid pick.

[sjparsons]

I'm all for Jest + TS based on my personal experience. Jest is very ergonomic.

[sjparsons]

We talked about this a bit at the working group meeting today. One of things we explored was the ability to run the test suite in the browser. This would allow us to provide a service where people could enter a URL and then receive results in their browser. They wouldn't have to download or set up anything and they could point it to private URLs that only they had access to.

We also talked about possibly finding a way to do this within GraphiQL perhaps as some plugin/module.

Although we may want to run the test in the command line, there's some benefits to also being able to run them in the browser.

@jaydenseric agreed to take up an action and help draft a PoC over the next month to explore that. @spawnia please feel free to be as involved as you want to be, but we didn't want to give you a action when you weren't present!

The way I see it, is that a PoC allows us to explore this and see if there are any issues. There's nothing stopping us looking at a couple of PoCs to explore different possibilities if others have some other ideas in mind.

[balshor]

Although we may want to run the test in the command line, there's some benefits to also being able to run them in the browser.

Browser based tests are significantly harder to integrate into build pipelines, both locally for developers and in the CI pipeline. If there is a way to easily run the tests from the CLI and without a browser, we will be much more likely to be able to continuously validate that our GraphQL services follow the spec.

[spawnia]

I don't really see the use case for running them in the browser, i guess it could be a quick way for a one-of check of an API? I would always go for an automatic process that continually asserts spec compliancy of a GraphQL server i develop/depend on.

I think that automatic test execution should be our first priority, so i would focus on a CLI tool first before attempting browser integration. That said, if something like a GraphiQL integration can be tacked on without much additional effort, i am not opposed to it.

[ghmcadams]

@spawnia Is this issue still relevant?

[spawnia]

Is this issue still relevant?

Somewhat. Perhaps it does not need to stand on its own, as by itself it is not really actionable, but it would be relevant when beginning work on an implementation of a test suite.

How about this: we leave this and other issues labeled with "audits" open until the next working group and decide on a plan for the test suite then. In terms of organisation, perhaps a working test suite could be a milestone?

[spawnia]

Feedback from the July 2022 working group meeting:

• Browser Testing vs. CLI
  • Browsers can not make any kind of request, which would limit capabilities
• Technology
  • JavaScript
  • Get inspired by https://github.com/apollographql/apollo-federation-subgraph-compatibility
  • As few dependencies and as simple as possible
• Errors for MUST, warnings for SHOULD
• Host in a separate repository under the GitHub graphql organization

Thanks everyone for the input, I intend to try implementing something with webonyx/graphql-php#1206