Consider running graphql-over-http spec tests automatically #7158
Comments
glasser
added a commit
that referenced
this issue
Nov 18, 2022
This PR adds the spec audit suite from `graphql-http` (designed to validate compatibility with the "GraphQL over HTTP" specification) to the integration test suite. It expects all required (MUST) audits to pass, and for all optional (MAY/SHOULD) audits to pass that we haven't explicitly excepted. All required audits already passed. Failing optional audits fell into three categories: - Ignoring `operationName`, `variables`, and `extensions` when provided with incorrect types instead of returning a 400 error. The SHOULD here seemed reasonable and so this PR changes our POST handler to return 400s if incorrect types are provided here. As described in the changeset, this is backwards incompatible but we are comfortable making this choice today. - Returning 400 errors for various error conditions like GraphQL parse errors, for the original `application/json` response type. The one aspect of the GoH spec that is prescriptive rather than descriptive is the invention of the `application/graphql-response+json` response MIME type; the theory is that you can't really "trust" that a non-2xx response with MIME type `application/json` is actually generated by the GraphQL server rather than by some other proxy, so it has you use this new type along with 4xx for these errors, and then SHOULDs that these errors come with 200 for the original MIME type. The main reason that this is a SHOULD is because many servers like Apollo Server already return 400s in this case, and we're not interested in changing that. So we ignore these particular errors. - There's a sorta buggy audit that has opinions about how bad JSON errors should look. See graphql/graphql-http#25 Note that we deliberately make a strict dependency on a single version of graphql-http, so that the tests run by a given version of `@apollo/server-integration-test-suite` are well defined. Fixes #7158.
glasser
added a commit
that referenced
this issue
Nov 21, 2022
This PR adds the spec audit suite from `graphql-http` (designed to validate compatibility with the "GraphQL over HTTP" specification) to the integration test suite. It expects all required (MUST) audits to pass, and for all optional (MAY/SHOULD) audits to pass that we haven't explicitly excepted. All required audits already passed. Failing optional audits fell into three categories: - Ignoring `operationName`, `variables`, and `extensions` when provided with incorrect types instead of returning a 400 error. The SHOULD here seemed reasonable and so this PR changes our POST handler to return 400s if incorrect types are provided here. As described in the changeset, this is backwards incompatible but we are comfortable making this choice today. - Returning 400 errors for various error conditions like GraphQL parse errors, for the original `application/json` response type. The one aspect of the GoH spec that is prescriptive rather than descriptive is the invention of the `application/graphql-response+json` response MIME type; the theory is that you can't really "trust" that a non-2xx response with MIME type `application/json` is actually generated by the GraphQL server rather than by some other proxy, so it has you use this new type along with 4xx for these errors, and then SHOULDs that these errors come with 200 for the original MIME type. The main reason that this is a SHOULD is because many servers like Apollo Server already return 400s in this case, and we're not interested in changing that. So we ignore these particular errors. - There's a sorta buggy audit that has opinions about how bad JSON errors should look. See graphql/graphql-http#25 Note that we deliberately make a strict dependency on a single version of graphql-http, so that the tests run by a given version of `@apollo/server-integration-test-suite` are well defined. Fixes #7158.
glasser
added a commit
that referenced
this issue
Nov 21, 2022
This PR adds the spec audit suite from `graphql-http` (designed to validate compatibility with the "GraphQL over HTTP" specification) to the integration test suite. It expects all required (MUST) audits to pass, and for all optional (MAY/SHOULD) audits to pass that we haven't explicitly excepted. All required audits already passed. Failing optional audits fell into three categories: - Ignoring `operationName`, `variables`, and `extensions` when provided with incorrect types instead of returning a 400 error. The SHOULD here seemed reasonable and so this PR changes our POST handler to return 400s if incorrect types are provided here. As described in the changeset, this is backwards incompatible but we are comfortable making this choice today. - Returning 400 errors for various error conditions like GraphQL parse errors, for the original `application/json` response type. The one aspect of the GoH spec that is prescriptive rather than descriptive is the invention of the `application/graphql-response+json` response MIME type; the theory is that you can't really "trust" that a non-2xx response with MIME type `application/json` is actually generated by the GraphQL server rather than by some other proxy, so it has you use this new type along with 4xx for these errors, and then SHOULDs that these errors come with 200 for the original MIME type. The main reason that this is a SHOULD is because many servers like Apollo Server already return 400s in this case, and we're not interested in changing that. So we ignore these particular errors. - There's a sorta buggy audit that has opinions about how bad JSON errors should look. See graphql/graphql-http#25 Note that we deliberately make a strict dependency on a single version of graphql-http, so that the tests run by a given version of `@apollo/server-integration-test-suite` are well defined. Fixes #7158. Co-authored-by: Denis Badurina <badurinadenis@gmail.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The GraphQL-over-HTTP working group made a test suite for the protocol. They run audits on various servers in their system, but it might be nice for us to use their tests directly in our test suite so we can find out if we break anything.
https://github.com/graphql/graphql-http/tree/master/implementations/apollo-server
As of right now we pass all the required tests but not all the SHOULD tests. Some of this is intentional (ie, the reason that they are SHOULDs is because the spec folks know that existing servers don't do what is SHOULDed and won't change for back-compat reasons) and others might now be. We should look into improving any unintentional spec warnings while we're at it.
The text was updated successfully, but these errors were encountered: