New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compliance test suite #1209

Open
beauby opened this Issue Aug 4, 2017 · 12 comments

Comments

Projects
None yet
7 participants
@beauby
Contributor

beauby commented Aug 4, 2017

The issue

This topic has been brushed a few times already (#1098, #892, #733). The idea is to provide a test suite so that library implementers can 1. prove compliance to the spec and 2. uniformly showcase their abilities.

The solution

I went ahead and created some basic infrastructure: jsonapi-grader, which is mainly a bunch of request specs.

The idea with the validation of server libraries is that each maintainer would set up a service (on a free heroku instance or similar) against which users could run the test suite (and possibly the jsonapi.org maintainers would periodically run it as well). The test scenarii would possibly be documented on the jsonapi.org website.

Making it happen

Sounds interesting? Get involved and help build the actual test suite (suggesting scenarii and/or implementing them).

@beauby

This comment has been minimized.

Show comment
Hide comment
@beauby

beauby Aug 4, 2017

Contributor

This could be a helpful reference: https://json-ld.org/test-suite/

Contributor

beauby commented Aug 4, 2017

This could be a helpful reference: https://json-ld.org/test-suite/

@evolve2k

This comment has been minimized.

Show comment
Hide comment
@evolve2k

evolve2k Aug 5, 2017

Since this is the new home for this discussion I'll bring my recommendation here from #1098

I'm proposing we use 'Author > Books > Reviews' as the basis of an example app.

Lets make it so that all resources in the domain have names that are very easy to grok and can not be in any way misunderstood for code related terms.

evolve2k commented Aug 5, 2017

Since this is the new home for this discussion I'll bring my recommendation here from #1098

I'm proposing we use 'Author > Books > Reviews' as the basis of an example app.

Lets make it so that all resources in the domain have names that are very easy to grok and can not be in any way misunderstood for code related terms.

@beauby

This comment has been minimized.

Show comment
Hide comment
@beauby

beauby Aug 5, 2017

Contributor

@evolve2k Thanks for the input. I agree that we should keep the test scenarii and the tested resources/endpoints simple to reason about.
However, I believe it would be easier for implementers that each scenario tests a specific thing (which is not opposed to having resources/endpoints named as you suggest, but I'd like to avoid having 3 endpoints that have to comply to many different scenarii at the same time).
One possible solution would be to have each test scenario hit namespaced endpoints (e.g. for the first test http://compliance.example.com/test1/books).

Contributor

beauby commented Aug 5, 2017

@evolve2k Thanks for the input. I agree that we should keep the test scenarii and the tested resources/endpoints simple to reason about.
However, I believe it would be easier for implementers that each scenario tests a specific thing (which is not opposed to having resources/endpoints named as you suggest, but I'd like to avoid having 3 endpoints that have to comply to many different scenarii at the same time).
One possible solution would be to have each test scenario hit namespaced endpoints (e.g. for the first test http://compliance.example.com/test1/books).

@fotinakis

This comment has been minimized.

Show comment
Hide comment
@fotinakis

fotinakis Aug 5, 2017

Contributor

One thing I think should be in there: a model with a two-word name, like CustomerReviews, so that we can test serializing the type to customer-reviews.

Contributor

fotinakis commented Aug 5, 2017

One thing I think should be in there: a model with a two-word name, like CustomerReviews, so that we can test serializing the type to customer-reviews.

@jaredcnance

This comment has been minimized.

Show comment
Hide comment
@jaredcnance

jaredcnance Aug 5, 2017

Contributor

@fotinakis I'm not certain we should be concerned with the casing choice of the Server since, although it is a recommendation, is not required. For example, an implementation may only support camelCased or snake_cased names. Compound resource names introduce, what I would consider to be, an unnecessary complexity. It makes result validation a bit more complicated since now we need to check the keys of the returned document against several different casing options. But, I also see the value in being able to state what casing mode the Server uses by default. If we did use compound names, I believe it should be isolated to a single test, validating ONLY that.

Contributor

jaredcnance commented Aug 5, 2017

@fotinakis I'm not certain we should be concerned with the casing choice of the Server since, although it is a recommendation, is not required. For example, an implementation may only support camelCased or snake_cased names. Compound resource names introduce, what I would consider to be, an unnecessary complexity. It makes result validation a bit more complicated since now we need to check the keys of the returned document against several different casing options. But, I also see the value in being able to state what casing mode the Server uses by default. If we did use compound names, I believe it should be isolated to a single test, validating ONLY that.

@ethanresnick

This comment has been minimized.

Show comment
Hide comment
@ethanresnick

ethanresnick Aug 7, 2017

Member

I love this discussion! Don't have the bandwidth to add much, but I would like to point to one of @tkellen's old projects — https://github.com/endpoints/fantasy-database — which could offer a schema/test data to work from easily.

Member

ethanresnick commented Aug 7, 2017

I love this discussion! Don't have the bandwidth to add much, but I would like to point to one of @tkellen's old projects — https://github.com/endpoints/fantasy-database — which could offer a schema/test data to work from easily.

@remmeier

This comment has been minimized.

Show comment
Hide comment
@remmeier

remmeier Aug 11, 2017

Contributor

IMDB is also a popular example dataset.

on the application side I could offer crnk.io as guinea pig to setup and pass those tests. It has both a full server and implementation.

Contributor

remmeier commented Aug 11, 2017

IMDB is also a popular example dataset.

on the application side I could offer crnk.io as guinea pig to setup and pass those tests. It has both a full server and implementation.

@beauby

This comment has been minimized.

Show comment
Hide comment
@beauby

beauby Aug 29, 2017

Contributor

Thanks for the inputs – a dataset will definitely come in handy.
I think the next step should be establishing together a list of specs (sorted by category) that we want for the first version. I'll open a PR so that people can suggest new tests/categories.

Contributor

beauby commented Aug 29, 2017

Thanks for the inputs – a dataset will definitely come in handy.
I think the next step should be establishing together a list of specs (sorted by category) that we want for the first version. I'll open a PR so that people can suggest new tests/categories.

@beauby

This comment has been minimized.

Show comment
Hide comment
@beauby

beauby Sep 14, 2017

Contributor

Update: I built part of a test suite. Anybody willing to give a hand?

Contributor

beauby commented Sep 14, 2017

Update: I built part of a test suite. Anybody willing to give a hand?

@jaredcnance

This comment has been minimized.

Show comment
Hide comment
@jaredcnance

jaredcnance Sep 14, 2017

Contributor

@beauby thanks for leading this. i'll be taking a deeper dive into it this weekend.

Contributor

jaredcnance commented Sep 14, 2017

@beauby thanks for leading this. i'll be taking a deeper dive into it this weekend.

@beauby

This comment has been minimized.

Show comment
Hide comment
@beauby

beauby Sep 15, 2017

Contributor

@jaredcnance (and everyone, really): I updated the README with a few explanations, and added a (partial) example compliance implementation for jsonapi-rails.

Contributor

beauby commented Sep 15, 2017

@jaredcnance (and everyone, really): I updated the README with a few explanations, and added a (partial) example compliance implementation for jsonapi-rails.

@tkellen

This comment has been minimized.

Show comment
Hide comment
@tkellen
Member

tkellen commented Sep 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment