Discuss: Standard spec for JSON REST API?

[manuelmitasch]

As @sebgie seems to be cleaning up the JSON REST API I would like to start a discussion why or why not it could make sense to use a standard spec/format for the API.

The format suggested by @sebgie in #2347, #2348, #2349, #2350 looks beautiful and straight forward to use.

In the last 1-2 years several standard formats/specs/media-types for HYPERMEDIA APIS have been developed.

PRO

• Profit from clear set of rules + expertise of creators
• There are libraries/tools that can provide/consume these standards out of the box. This reduces the dependency between client and server - loose coupling!
• Clients could consume it based on a API description.

CON

• standards are young and still evolving

Some standards:

• JSON API - The standard format used by Ember Data! The current ghost API seems already close to this format. By @steveklabnik
• HAL
• Collection+JSON
• Siren
• Restful Objects
• OData

Edit: Added OData as suggested by @ErisDS

[ErisDS]

This misses OData which is the standard we are currently looking at.

[sebgie]

The proposed changes in the issues mentioned are very close to what is suggested in JSON API. In terms of formatting JSON responses the thing I left out is wrapping relations in a link property. Adding the link property would mean adding pre- and post-processing steps to every response for the sake of complying to the specification.

Another source of inspiration for the API is Stripe who does an amazing job providing a consistent, clean and well documented API to developers. As a rule of thumb I would say that our API should follow either Stripes example or the JSON API spec.

The reason why I wouldn't like to commit to a specification right now is that, as you said, they are still young and leave more or less room for interpretation. Another thing we have to keep in mind is that we not only have to serve a restful API, but we use the same interface for front end rendering and will soon open it up to Apps.

[sebgie]

Sorry for multi posting, my Browser didn't refresh the view :-/.

[steveklabnik]

We are using JSON API in production at @balanced, pushing $1MM/day in card transactions through it, so it's at least stable enough to do that. ;)

I'd love to have Ghost also using JSON API, anything I can do to help you understand the details, please let me know.

[steveklabnik]

Another thing we have to keep in mind is that we not only have to serve a restful API, but we use the same interface for front end rendering

JavaScript SPAs are directly in JSON API's heritage, so we should be quite good for that.

[ErisDS]

Our API is used over HTTP, internally for rendering blogs, is passed to apps via a proxy, and will likely one day be used by a query helper in themes.

It has to serve multiple use-cases, not just being JSON served over HTTP.

Expanding fields using an expand property makes sense to me as it keeps a simple JSON structure, but splitting expanded fields into a 'links' object doesn't seems like a useful thing to do. As @sebgie said, it will require us to both pre and post-process data just to have it send down the tubes in a particular format.

I'm not sure what benefit having a separated links object has over inline expanded object?

I'm also not sure we would get much benefit from adhering to a standard as they are not commonplace.

Sent from my iPhone

On 8 Mar 2014, at 20:19, Steve Klabnik notifications@github.com wrote:

Another thing we have to keep in mind is that we not only have to serve a restful API, but we use the same interface for front end rendering

JavaScript SPAs are directly in JSON API's heritage, so we should be quite good for that.

—
Reply to this email directly or view it on GitHub.

[steveklabnik]

To me, the primary benefit of choosing a format is that you have to do way less design work (aka bike shedding) and you get to re-use existing tooling.

I'd highly recommend picking SOME format and following it, rather than building yet another bespoke, artisanal, hand-crafted API.

@wycats should also be involved here.

[manuelmitasch]

As @sebgie pointed out the big difference between the current ghost API and eg. JSON API format is the handling of associations.

With the use of a HYPERMEDIA format like JSON API the client can determine (based on the conventions) how to load AND save models that have an association from the initial model loaded.

With the current ghost API the related models simply are embedded (nested or expanded properties) where they are associated. For the simple read-only use case this is very convenient. The drawbacks are:

• The client needs to have hard-coded information how to save these models later.
• The payload needs to includes the whole payload for an associated model every time. This can increase the payload significantly.

I have created a small gist to demonstrate the payload difference between ghost API + JSON API by just looking at the author model. It is 468 LOC vs. 178 LOC or 12.5kB vs. 2.8kB. Once this would be expanded to tags, created_by, updated_by and published_by the difference would become even more significant.

With JSON API and it's compound documents we could still serve all needed models in one json payload, but in a much more concise format. The pre and post processing mentioned by @sebgie is minimal. To find an author with id 1 it would be something like payload.linked.authors.findBy("id, "1") (using a generic findBy method).

In the case of the ember rewrite we will most likely need pre and post processing anyway. For just displaying models the "expanded" properties are okay. Once we need to change one of the expanded properties we would of course want to update the value everywhere in the UI. When just pulling in the expanded properties into the templates each author property with the id 1 would be seen as a separate object. Thus, we need a thin model layer or a proper persistence library (eg. Ember Data) to handle these cases.

I think there is a very good reason why computer science has invented relations/associations between objects and a clean API should also try to embrace this. I did not have a look at the way the API is currently designed, but it should not be too much work to serve different formats through content negotiation. If a API consumer requests "application/json" it gets the ghost api format served. If a API consumer requests "application/vnd.api+json" it returns JSON API. I'm certain that using a HYPERMEDIA format will have many long-term benefits, that we can not even foresee at the moment.

If there is an interest in (additionally) serving JSON API I would be happy to help with implementing this!

[steveklabnik]

While I appreciate Manuel's hypermedia enthusiasm, I also want to make sure that you know that JSON API can also do the usual id based associations. We think that eventually, you'll see the advantage of links, but it's not required to get started.

[sebgie]

In the case of the ember rewrite we will most likely need pre and post processing anyway.

I get the feeling that emberJS is not only opinionated about how to do things on the front end, there is also a very strong opinion on how to do things on the back end? Can you please describe what the requirements for ember are and what we have to implement to meet embers needs? If linked objects make the implementation of the admin better or cleaner it is the right thing to do. Until now linked objects seemed more like a nice to have than an obstacle for implementing functionality?

I have created a small gist to demonstrate the payload difference between ghost API + JSON API by just looking at the author model. It is 468 LOC vs. 178 LOC or 12.5kB vs. 2.8kB.

This is the first technical reason I see in this discussion that is worth considering a change from embedded to linked objects.

If a API consumer requests "application/json" it gets the ghost api format served. If a API consumer requests "application/vnd.api+json" it returns JSON API.

Ghost is way too young to introduce different formats for JSON responses and this would be in stark contrast to GDK principles (consistency, one way to do things, ...).

Although I think that it is a good thing to introduce linked objects if needed, I would not like to label our API as following the JSON API specification at the moment for the following reasons:

• We don't support PATCH methods now or in the foreseeable future.
• JSON API doesn’t have a version number so that I can say we are following v1.2 of the specification and don’t have to implement something new every time there is an addition to the specification.
• Missing error format which leaves a lot of room for interpretation. At the moment we use
  • {code: <http error code>, message:'Your human readable error message'}

[manuelmitasch]

I get the feeling that emberJS is not only opinionated about how to do things on the front end, there is also a very strong opinion on how to do things on the back end?

Ember in no way enforces a certain backend API style. Even Ember Data is very flexible concerning the API style. It of course needs a certain format internally. With custom adapters + serializers you can use any API format and transform it into the internal format. It should be quite easy to use the current ghost API format with Ember Data.

If you are implying that most ember devs tend to clean, well-designed solutions. YES!

We don't support PATCH methods now or in the foreseeable future.

Personally, I had reservations to PATCH my self and I'm still not super happy with it. I can understand why you would not want to implement PATCH for simple cases.

JSON API doesn’t have a version number

True and it's a bit sad. It makes communication a lot more difficult.

Missing error format which leaves a lot of room for interpretation.

I think JSON API has not settled on an error format. I think just returning a message as you suggested will lead to poor UX. Providing some sort of context where the error appeared is very useful:

{
  "errors": {
    "firstName": ["cannot be blank", "must be 10+ characters"],
    "lastName": ["cannot contain symbols"]
  }
}

[steveklabnik]

True and it's a bit sad. It makes communication a lot more difficult.

It actually makes our spec work more difficult, not your implementation work. JSON API will be backwards compatible forever, just like HTML, and for that matter, JSON.

We haven't made that guarantee yet, but will quite soon. The spec is pretty stable, and people are using it. I don't expect there to be any incompatible changes at this point, though there may be one or two small ones.

I think JSON API has not settled on an error format.

This is true: json-api/json-api#7

[sebgie]

I think just returning a message as you suggested will lead to poor UX.

I'm not suggesting anything, that is what Ghosts API returns atm.

If you are implying that most ember devs tend to clean, well-designed solutions. YES!

I'm implying that there is not only one clean, well-designed solution.

---

As it stands for me there is no sense in officially committing to the JSON API as we can not comply with all aspects of the specification. As far as I can see the most important thing for @manuelmitasch is to adopt linked objects and compound documents. Given the reasoning it outweighs the need of adding special wrapping in the long run and makes sense (cc @ErisDS). For future API discussions we can use JSON API as the go to reference point.

[ErisDS]

I am disappointed by the lack of discussion of the various standards and what their pros and cons are, why they are good and why they would be the right thing for Ghost. I feel that this discussion reads a bit too much like a sales pitch for JSONAPI.org.

Ghost is not going to support PATCH methods in the near future. That and the lack of error format and versioning makes it clear that JSON API is not the right solution for Ghost at present. If we're going to officially commit to a standard, it needs to be declared stable and I believe that a version number is important in this. Personally, I think committing to 'only add, never remove' seems like it would hamstring the development of a very young format - at a certain point it is going to become impossible to evolve the format to meet the demands of the ever changing internet landscape.

All that being said, @manuelmitasch's argument for reducing payload via links is incredibly well made. I think it is agreed by all that our API needs to move towards this sort of structure as it is technically better than our inline format, and the recursive formats of other standards like HAL.

I do have a reservation however, about using the term links for talking about relations, and meta for other URLs when several other standards use the term links to mean specifically navigational links (URLs) and would include both relational links and meta data links like pagination & rss (ref Siren, HAL / Collection+JSON etc).

I think it's great to move away from using URLs to recursively embed documents, and meta makes perfect sense for pagination & rss URLs, but I think it would also ease confusion if at the same time, the relational links were referred to directly as relations to remove any conflict in the term links.

Finally, even using JSON API as a reference point, we are left without any clear direction for error formats. However, I think there's room for a whole other discussion here as there are so many different types of error to consider.

All-in-all, I'm tending to agree with @sebgie that there isn't a format that fits our needs.

[steveklabnik]

I feel that this discussion reads a bit too much like a sales pitch for JSONAPI.org.

Well none of the other spec authors have chimed in. I can give you a rundown of my opinions of other formats, but I'm biased, of course.

[ErisDS]

Please do share ;)

[wycats]

@ErisDS I wanted to respond to a few of your comments 😄

First of all, I think it's important to note that JSON API is a full protocol, while things like HAL and Collection+JSON are response formats. Collection+JSON doesn't use the PATCH verb because it doesn't tell you what to do at all about updates.

Ghost is not going to support PATCH methods in the near future

The reason I originally went with the PATCH verb has everything to do with relationships. It's easy enough to update a set of attributes to a resource via PUT, but how do you add a comment to a post using PUT? Applications use ad-hoc approaches to this problem (just PUT the entire array, update a foreign key), but none of the approaches actually work reliably across the board.

For example, if two people PUT the entire array at the same time, the last one will win, and the first person's comment will be lost. Going with a foreign key works ok for databases, but not for document stores.

Because of these problems, I originally went with PATCH, which can express "add Comment X to Post Y" simply, support multiple simultaneous clients, and works with document stores.

While I stand by that decision for updating relationships, I see no reason not to add support for PUT for attribute updates. There's an ongoing discussion on json-api/json-api#201, and I think it'll make it in.

Which leads me to a question: what specifically concerns you about PATCH? Do you use a web server that doesn't support the verb? Does it just seem hard to implement? Unclear benefits? If it's the last one, I hope the above has shown the benefits of PATCH for relationships.

That and the lack of error format

This one is a no-brainer. We should define an error format (straw man: "errors": { "field": ["reason", "reason"] }) ASAP instead of leaving the format unspecified.

and versioning

Again, a no brainer. I spoke to @steveklabnik today while at Fluent and we agreed to mark the current version 1.0 as soon as possible. People (including Balanced) are using JSON API in production, and having a stable version seems like an obvious win.

Personally, I think committing to 'only add, never remove' seems like it would hamstring the development of a very young format

I've personally been very surprised by how far you can get with backwards-compatible changes only, but I agree that having the ability to eventually release a JSON API 2.0 if the need arises is a good option to keep open.

I do have a reservation however, about using the term links for talking about relations, and meta for other URLs when several other standards use the term links to mean specifically navigational links (URLs) and would include both relational links and meta data links like pagination & rss (ref Siren, HAL / Collection+JSON etc).

This objection seems to be about disagreement over some names chosen in the current spec. I totally understand where you're coming from, but I hope that naming questions aren't enough reason to go with a custom format instead of a community effort.

You should consider opening issues on json-api/json-api.

All-in-all, I'm tending to agree with @sebgie that there isn't a format that fits our needs

Are there any other issues that make you feel that way that I didn't address above?

One thing that's really nice about community efforts is that they tend to result in reusable tooling that everyone can share. Things are still evolving on that front, but I definitely expect the community to build a JSON-API browser, and to see more client and server libraries in more languages.

Finally, I want to reiterate the point I made earlier. JSON-API isn't just a response format; it's an attempt to come up with a shared protocol for communicating with RESTful endpoints. That forces out into the open questions like the relationship updating discussion above. It also gives the community a place to talk about questions like pagination, sparse fields, saving multiple records at once, and conventional ways to map resources onto client-side object types.

I think a protocol that explicitly aims to continue evolving to tackle more interesting problems over time should be very interesting to Ghost, which will no doubt discover some of those very problems as it grows.

Thanks for listening to this (somewhat long) response.

[ErisDS]

@wycats Thank you for weighing in 😄

To respond to the main points:

PATCH vs PUT

Which leads me to a question: what specifically concerns you about PATCH? Do you use a web server that doesn't support the verb? Does it just seem hard to implement? Unclear benefits? If it's the last one, I hope the above has shown the benefits of PATCH for relationships.

We currently use PUT, we intend to make a lot of changes to the API but this one seems to be low on the value-to-time ratio. We can and will do it, but it might not land in time for the first version of Ghost that would use JSON-API.

Errors

This one is a no-brainer. We should define an error format (straw man: "errors": { "field": ["reason", "reason"] }) ASAP instead of leaving the format unspecified.

The discussion over at json-api/json-api#7 (comment) seems to have moved towards using http://tools.ietf.org/html/draft-nottingham-http-problem-06 which seems like a sensible idea to me. I think we will work towards implementing that and see how it goes hopefully JSON-API will declare it as part of the format or alternatively perhaps we'll find some issues to further the discussion.

Versioning

Again, a no brainer. I spoke to @steveklabnik today while at Fluent and we agreed to mark the current version 1.0 as soon as possible.

Awesome, we'll keep an eye out. The point at which 1.0 is declared will be the point at which we know whether we are 'officially' commiting to JSON-API (because everything is compatible) or using it as our reference point but diverging in some areas. I sincerely hope that our two projects will naturally align.

links vs relations

This objection seems to be about disagreement over some names chosen in the current spec. I totally understand where you're coming from, but I hope that naming questions aren't enough reason to go with a custom format instead of a community effort.

Naming the bikeshed isn't that different to painting it - this I know, it was just a concern. Reading some of the the issues on json-api it seems that it will may allow both navigational and relational links in the links object, which would render my point mute.

Moving forward

Are there any other issues that make you feel that way that I didn't address above?

A vague concern that some of the things we need to do with our API are inherently non-RESTful and may not fit. We won't know for sure until we get there.

All of your points about community and tackling interesting problems resonated with me. Ghost certainly doesn't want to go out and do this on its own, we were just feeling like it was our only option. I think you've addressed that concern very well.

At this point I think we can safely say that we intend to work towards using JSON-API, raising issues where we find problems and hopefully solving them at the JSON-API level so that we can declare official support when 1.0 gets tagged. @sebgie is going to update all of our issues about JSON responses.

[steveklabnik]

I've set the 1.0 milestone for 9 days from today. I don't believe anything backwards incompatible will change at this time, but there are a few questions that should be answered first.

[tomdale]

@ErisDS:

We currently use PUT, we intend to make a lot of changes to the API but this one seems to be low on the value-to-time ratio. We can and will do it, but it might not land in time for the first version of Ghost that would use JSON-API.

Isn't this just changing the string PUT to PATCH in HTTP code? Can you help me understand where other changes to the codebase would need to happen?

[sebgie]

@tomdale

Isn't this just changing the string PUT to PATCH in HTTP code? Can you help me understand where other changes to the codebase would need to happen?

I'm sorry I don't think it is that easy. The difference between PUT and PATCH is explained by @wycats in the issue json-api/json-api#211. The PUT verb does update an entire resource all at once and is currently supported by Ghost. PATCH is for updating single attributes and relations which would need implementation. Given that we need to implement relations using links anyway, I think that PATCH support will follow naturally.

[ErisDS]

I think we've reached a basic conclusion here so going to close so we can crack on 😄

[ErisDS]

I think I'm about to piss everyone off by doing a total flip-flop on this 😈

I still believe that there is a fundamental problem with using the 'linked object' format in JSON API. I was convinced that it was a good idea by @manuelmitasch's demonstration of the reduced response size. This is clearly a very strong argument for using linked objects.

However, I do believe that several other issues with regard to the need to pre- and post-process the javascript objects into JSON haven't been properly considered. To reference back to the much earlier discussion @sebgie & I were concerned about the need to pre- and post-process the data. @manuelmitasch's response was as follows:

The pre and post processing mentioned by @sebgie is minimal. To find an author with id 1 it would be something like payload.linked.authors.findBy("id, "1") (using a generic findBy method).

This assumes that the context you are in when dealing with the api response is a JavaScript context. This is true for the admin UI which we are rewriting in Ember (and supposedly Ember works well with the JSON API format) but this is not true for our server-side code which serves a blog - essentially everything in core/server/controllers/frontend.js.

Here, we pass the JSON objects down to the theme to handle with handlebars. Our theme API has no concept of linked objects, it assumes that everything is inline. For example, when in the post context in handlebars, themes expect to be able to work with {{author.name}} and {{tags}}. If we just switch to JSON-API response format, all of this will break. Every one of the 250 themes for Ghost will cease to function, so that's not acceptable.

This means that we have to post-process the data before handing it off to themes. This means that not only do we have to write fancy code to generate the correctly formatted response instead of using the inline JSON format provided by bookshelf (see #2544) but we then have to undo all of that and convert the response back to inline before handing the response to the theme.

Perhaps this might looks something like changing line 32 of frontend.js from:

return api.posts.browse(options);

to

return api.posts.browse(options).inline();

Which we'd have to do in a number of places... but this then means that our data API essentially has 2 entirely different JSON object formats depending on whether you are getting your data directly over HTTP or whether you're using the theme API.

I am totally lost as to what to do at this point. I don't deny that sending less data over HTTP would be preferential, but having to manage and document 2 different JSON formats seems ridiculous. It seems to me that the linked objects JSON format assumes that you have a programmatic context at the end, but what about handlebars? How does ember cope with this given that ember also relies heavily on handlebars?

[hswolff]

I don't think there's a lot of overhead in taking a JSON API format and passing it through an 'inline' function to have it compatible with the needs of Handlebars.

Given a JSON API like this (from the JSON API website):

{
  "links": {
    "posts.author": {
      "href": "http://example.com/people/{posts.author}",
      "type": "people"
    },
    "posts.comments": {
      "href": "http://example.com/comments/{posts.comments}",
      "type": "comments"
    }
  },
  "posts": [{
    "id": "1",
    "title": "Rails is Omakase",
    "links": {
      "author": "9",
      "comments": [ "5", "12", "17", "20" ]
    }
  }]
}

You can simple iterate over the API key and then easily match up linked objects.

var resource = 'posts';
_.each(apiResponse[resource], function(post) {

  _.each(post.links, function(link, linkName) {
    // linkName here is 'author' or 'comments'
    // link being the actual value
    post[linkName] = apiResponse.links[resource + '.' + linkName];
  });

});

Also take note that the bandwidth saving advantages of JSON API don't stop there - it also helps conserve JavaScript memory. When returning a fully inline API response it's as a string, meaning that every duplicate nested resource creates a new JavaScript object.

When using the JSON API method you create one JavaScript object for each linked resource and when nesting them in a merge function it merely assigns references to those objects instead of creating new ones, thus saving the need to create many duplicate objects. This is done due to JavaScript's way of assigning primitives by value, and objects via reference.

but having to manage and document 2 different JSON formats seems ridiculous

Rather than managing 2 different JSON formats it can be taken as a given that every linked JSON API object becomes expanded at the root of every resource node when presented to Handlebars. Referring back to the example above, when presented in Handlebars, every posts object would have a author and comments property with all data attached.

Having a post-process method in controller/frontend.js seems par the course as the role of any controller is to take the data layer and control how it's delivered to the view layer. This seems to be very much inline with that role.

[ErisDS]

You can simple iterate over the API key and then easily match up linked objects.

It's all good saying this is simple and easy, but where do we do it? And why? And how? Doing it in the controller means doing it for every piece of data before it is passed through the filters and to views = code duplication. That might seem reasonably simple and easy right now, but that's only because Ghost is at 0.4 and doesn't have vast amounts of the functionality it intends to have in terms of controllers.

All of our controllers are currently pretty static (not customisable) and in core. What happens when apps are allowed to add controllers - how do we ensure that apps pass all their data through the inlining function at the right point?

And what about dynamic data? What about when we add a {{query}} helper to use the API via themes? The query structure would be the same as the REST API, but the response would be different. How do you reconcile that?

Rather than managing 2 different JSON formats it can be taken as a given that every linked JSON API object becomes expanded at the root of every resource node when presented to Handlebars.

I don't see how 'taken as a given' translates to theme developer documentation, or app developer documentation, especially when considering the context of an app that might operate on both handlebars data (add helpers) and non-handlebars data (anything else) - we would have to fully document both formats and the app would have to duplicate any data handling code to deal with 2 different response formats in different contexts?

Also take note that the bandwidth saving advantages of JSON API don't stop there - it also helps conserve JavaScript memory.

I'm not sure how we conserve much memory if we're both pre and post-processing the objects on the server side anyway? We end up with the api calling a toJSON method on a model which creates the linked object format and passes them to the controller that then converts them to inline objects instead... we're processing everything twice and we still end up with the same number of objects before handing off to the views?

I feel like the use cases at play here are far wider than anyone is considering? JSON API is intended for a REST API - but for our data API, REST is just one little use case.

[hswolff]

It's all good saying this is simple and easy, but where do we do it? And why? And how?

I'd imagine we'd have one utility function in the BaseModel class that would handle all instances of serializing and deserializing the data. One central location which handles output of data for an API call and another that handles when data is requested for rendering in a view.

---

Rather than reply point by point I do want to just make the high level observatoin that ultimately if we use JSON API exactly as specified we will have to support two versions of returned data:

1. One that aligns directly with what JSON API specifies.
2. One that takes what JSON API specifies and inlines all linked resources.

The second version is required when rendering templates as that is what Handlebars requires.

As a suggestion it may be in Ghost's best interest to adopt JSON API's specifications with one alteration: by default inline all linked resources. This would then allow us to have only one way of returning data, reducing confusion and documentation costs, and would allow us to still take advantage of all that JSON API has to offer minus the HTTP payload advantage of using linked resources.

We could even allow a flag from the REST API to return resources in a linked object if an application wants that HTTP payload savings, but by default all resources would be returned inline.

Thoughts?

[wycats]

We're still working towards releasing 1.0, and these updates are really good feedback. I'm going to see whether this shows a need for alternate representation support in JSON API proper.

Thanks for continuing this discussion and noodling on things 😄

[ErisDS]

I think it's becoming more and more apparent that Ghost's primary concern is consistency over performance. I'm still interested in hearing different perspectives and ideas around this though.

I think there is a sensible tradeoff between not including linked/related objects by default (requiring explicit includes parameter), but keeping them inline by default. If we don't include them unless we need them, we reduce the payload.

@hswolff's idea of inlining by default, but having a flag for when we control both ends and are happy to deal with the different format (i.e. in our own client app) makes sense, but I see that as an optimisation we could add at some point in future.

This would mean that in order to deliver Apps/Ember/0.5 we need to implement includes, and make sure that the format returned is consistent and follows JSON-API with the exception of inlining.

As a heads-up, my plan at present is to leave this discussion going until our meeting on Tuesday, bring it to the table then, make a final decision and plow on with getting the API cleaned up.

[sebgie]

I have added an API Overview to our Wiki to help clarify the differences between our APIs and how the modules depend on each other (https://github.com/TryGhost/Ghost/wiki/API-Overview).

I personally don't like the idea of having different formats for internal and external usage. It would add complexity for App developers, new contributors and the need to document all possible variations. Inlining the objects with the ability to expand references seems to me like the best way to prevent confusion.

The possibility of having a smaller payload is great but the fact that we have to support other areas than a restful API as well makes it complex to comply. I have also failed to notice this problem by the outlook of having an easy to implement performance boost.

This would mean that in order to deliver Apps/Ember/0.5 we need to implement includes, and make sure that the format returned is consistent and follows JSON-API with the exception of inlining.

I don't think that supporting include is a prerequisite for delivering Apps/Ember/0.5. We are working with a very small set of objects (default of 15 posts/page) at the moment and the include parameter could be used as performance optimization later as well.

As I wrote earlier, I think that we should use JSON API as our go to reference point, but we are not able to fully commit to the specification atm.

[hswolff]

As I wrote earlier, I think that we should use JSON API as our go to reference point, but we are not able to fully commit to the specification atm.

👍 Agree 100%. I'm fully on board with that plan.

[dgeb]

Hello all! Sorry to arrive to this discussion late. Like @wycats and @steveklabnik, I'm also a collaborator on JSON API and would like to see if it could be a good fit for Ghost.

Thanks to @sebgie for the API overview, complete with diagram. I see that you're not just looking to serve an external RESTful API, but also an internal API to be used by your server-side front end rendering and server-side pluggable apps. Because Ghost's internal and external APIs are serving distinct needs, I think that it's legitimate to consider them separately.

Tradeoffs when using JSON API

JSON API payloads are optimized for transport efficiency and discoverability (via hypermedia links to related resources). This optimization comes at the expense of navigability: the ability to easily traverse an object tree. While an ideally navigable JSON document might feature embedded and duplicated objects that can be traversed without dereferencing ids (e.g. post.comments[1].text), navigability is a secondary concern for JSON API. In contrast, JSON API uses links and linked objects, and some links may even require an href to resolve. These features allow it to deliver compact and even partial payloads to clients.

Because JSON API is not optimized for navigability, it's expected that serialization and deserialization layers will be used on both ends of the API. These layers translate JSON API's transport-optimized payloads to and from usable, navigable sets of data objects on both ends. Because this is the intended usage, many client and server adapters are being written.

JSON API for an External API

The existence of these standard adapters, and the wave that will probably follow once JSON API reaches 1.0, makes a strong case for using JSON API for your external API. Given that Ghost's admin interface is being written in Ember, and Ember Data has an (almost finished) JSON API compatible adapter, you could immediately see the benefit of not having to custom craft a client-side data adapter. By using JSON API along with Ember Data's compatible adapter, you should be able to just define your models in Ember and point your adapter to your server, completely avoiding the headache of writing and maintaining an ajax serialization/deserialization layer.

Similarly, other API users will appreciate your adherence to a spec that will also allow them to avoid hand-rolling client code to unpack your API. Of course, these benefits won't be realized if you customize your external API heavily, requiring tweaks in every client.

JSON API for an Internal API?

I am on much less certain ground making a recommendation for your internal API, given that I'm not familiar with Ghost's internal server-side layers. I can assume that your view templates and pluggable apps need to easily traverse data objects and relationships. It sounds like your templates expect data in a very specific format, and you want to avoid breaking compatibility. Therefore, the strengths of JSON API's compact format would have to be weighed against the need to write deserialization layers specific to your needs.

Furthermore, you may have entirely different concerns for rate limiting, pagination, inclusion of related records, sparse fieldsets, etc. for your internal vs external APIs.

Regardless of the format you choose for your internal API, you'll probably find it necessary to write a thin translation layer to properly serve your RESTful API from your internal resources.

JSON API 1.0

Last but not least, I should point out that JSON API 1.0 is taking a bit longer than planned because we're trying to do more than rubberstamp the current spec.

One of the largest changes we're planning is to establish POST, PUT and/or DELETE equivalents for every PATCH operation in the spec. We've come to realize that PATCH operations are a deal breaker for widespread adoption of JSON API, yet at the same time we want to offer a fully PATCH compliant spec as an alternative usage. JSON API 1.0 will require POST / PUT / DELETE support, but will also provide direction for APIs that also want to provide full PATCH support.

We hope to have JSON API 1.0 RC1 ready for review by the end of the week!

[ErisDS]

Because Ghost's internal and external APIs are serving distinct needs, I think that it's legitimate to consider them separately.

I disagree. Both must serve the same kind of data requests so I don't see that the needs are distinct. A pluggable app may use either the internal or external API and they could quite possibly use both in different parts of the app. This is why I believe having 2 different responses will not be workable.

Furthermore, it doesn't really work with the dogfooding principle, which is very important to us and means that the external API is an HTTP wrapper around the internal API. To try to paint a picture of what it would require to introduce JSON API to Ghost in one form or another:

To serve our current, inline response from both versions of the API:

Data Model -> toJSON() -> Internal API -> data consumed
Data Model -> toJSON() -> Internal API -> External API -> HTTP response -> data consumed

To serve JSON-API responses from both versions of the API:

Data Model -> toJSONAPI() -> Internal API -> toInlineJSON() -> data consumed
Data Model -> toJSONAPI() -> Internal API -> External API -> HTTP response -> ??? -> data consumed

To serve JSON API from just the external API, leaving the internal API inline:

Data Model -> toJSON() -> Internal API -> data consumed
Data Model -> toJSON() -> Internal API -> External API -> toJSONAPI() -> HTTP response -> ??? -> data consumed

???

in the case of the Ember admin UI, it's just an adapter and we are in control
in the case of a Ghost app using the internal API = we can automatically call our own server side adapter
in the case of a Ghost app using the external API = we don't have control
in the case of an external application consuming the external API, we don't have control
in the case of a theme using the external API = we don't have control

The introduction of JSON API reduces consistency across the Ghost landscape and increases complexity. I think, given the kind of rich and flexible ecosystem we are trying to create, having our own API format isn't all that crazy. What we're discussing doing here is making our toJSON() function spit out JSON which is similar to JSON API but with the exception of inline objects rather than linked.

[ErisDS]

This was discussed in today's public meeting and it was agreed to continue to push towards supporting JSON-API but with the exception of keeping our objects inline by default.

The long-term plan is to introduce a way to request linked objects / strict JSON API responses.

The work to change the API is currently covered by #2543 and #2544, although these may be replaced with more specific issues.

[dgeb]

I can understand that inlining (and duplicating) objects makes data navigable by clients of your internal API. However, if any of your use cases involve sending data updates back through your internal API, then you might find it better to deserialize data into a normalized form. In that way, you could ensure that an update to a record would be reflected everywhere that record was referenced by the client of your internal API.

Also, I would assume that in the cases in which you don't have control (i.e. clients of the external API), it would be desirable to conform to a standard rather than requiring that clients write ad hoc adapters.

Anyway, I see that this is closed now, but I'm glad that you're keeping the option to comply fully with JSON API on the table. Thanks for hearing us out :)