RFC: Introduce JSON:API resource format for HAPIv2

[iansolano]

Ref: #4858

What is JSON:API

JSON API is a format that works with HTTP. It delineates how clients should request or edit data from a server, and how the server should respond to said requests. A main goal of the specification is to optimize HTTP requests; both in terms of the number of requests and the size of data packages exchanged between clients and servers.

“JSON API is a wire protocol for incrementally fetching and updating a graph over HTTP” – Yehuda Katz

If you’ve ever argued with your team about the way your JSON responses should be formatted, JSON:API can be your anti-bikeshedding tool.

By following shared conventions, you can increase productivity, take advantage of generalized tooling, and focus on what matters: your application.

Clients built around JSON:API are able to take advantage of its features around efficiently caching responses, sometimes eliminating network requests entirely.

Here’s an example response from a blog that implements JSON:API:

{
  "links": {
    "self": "http://example.com/articles",
    "next": "http://example.com/articles?page[offset]=2",
    "last": "http://example.com/articles?page[offset]=10"
  },
  "data": [{
    "type": "articles",
    "id": "1",
    "attributes": {
      "title": "JSON:API paints my bikeshed!"
    },
    "relationships": {
      "author": {
        "links": {
          "self": "http://example.com/articles/1/relationships/author",
          "related": "http://example.com/articles/1/author"
        },
        "data": { "type": "people", "id": "9" }
      },
      "comments": {
        "links": {
          "self": "http://example.com/articles/1/relationships/comments",
          "related": "http://example.com/articles/1/comments"
        },
        "data": [
          { "type": "comments", "id": "5" },
          { "type": "comments", "id": "12" }
        ]
      }
    },
    "links": {
      "self": "http://example.com/articles/1"
    }
  }],
  "included": [{
    "type": "people",
    "id": "9",
    "attributes": {
      "firstName": "Dan",
      "lastName": "Gebhardt",
      "twitter": "dgeb"
    },
    "links": {
      "self": "http://example.com/people/9"
    }
  }, {
    "type": "comments",
    "id": "5",
    "attributes": {
      "body": "First!"
    },
    "relationships": {
      "author": {
        "data": { "type": "people", "id": "2" }
      }
    },
    "links": {
      "self": "http://example.com/comments/5"
    }
  }, {
    "type": "comments",
    "id": "12",
    "attributes": {
      "body": "I like XML better"
    },
    "relationships": {
      "author": {
        "data": { "type": "people", "id": "9" }
      }
    },
    "links": {
      "self": "http://example.com/comments/12"
    }
  }]
}

The response above contains the first in a collection of “articles”, as well as links to subsequent members in that collection. It also contains resources linked to the article, including its author and comments. Last but not least, links are provided that can be used to fetch or update any of these resources.

JSON:API covers creating and updating resources as well, not just responses.

Benefits From Using JSON API

Now that we have a basic idea of what JSON API is, what are some unique benefits that make it stand out?

Compound Documents

Compound documents is a unique ability in JSON API, allowing servers to send related resources alongside the requested primary resources — if implemented correctly this could decrease the number of necessary HTTP requests. Compound documents works by using the include parameter like as follows:

GET https://api.example.com/posts?include=author
This enables you to include additional resources in an initial request.

Sparse Fieldsets

If you’re using compound documents to include related resources, you could run into an issue of having large responses. Once again, JSON API has a solution.

Another unique aspect of JSON API are sparse fieldsets, which enable clients to only request data from specific fields. It works by adding the field you want to retrieve to the URI parameter with the resource name and the fields you want. This offers additional customization and can decrease bloat. It looks something like:

GET /articles?include=author&fields[articles]=title,body&fields[people]=name HTTP/1.1
Accept: application/vnd.api+json

Sparse fieldsets is a standardized method of allowing clients to specify only the properties they want from an object to be included in the response. Using sparse fieldsets, you get only the fields that you desire, offering unique customization potential that is alluring for lean data sharing environments.

• Additionally, these features around sparse fieldsets and compound documents help to support the coming future around HTTP/2 in which the protocol will favor making more requests, etc.

Optionality

Many of the features in JSONAPI.org are purely optional; you can turn them off or on. The features give the clients the power to determine what resources to accept. Getting clients to agree on how to retrieve and handle data is helpful as it erases redundancy and optimization to reduce bloat.

Optimization Features

JSON API comes equipped with many features to optimize the API return package. Special server side operations in JSON API include sorting, as well as pagination; the ability to limit the number of returned resources to a subset, with first, last, next, and prev links. As well as, standardizing errors response patterns.

Caching

“Because data changes affect fewer resources, there are fewer resources invalidated when data changes”

In JSON API use cases, caching is in essence built into HTTP. Since clients using JSON API access data in the same way, they don’t need to store data in various locations. This design may require a shift in thought, yet if used correctly can bring significant optimization benefits. For example: we can use ETag tokens in requests to match against the token and detect whether the resource has changed.

Normalizing Responses in Clients

Having a standard response format, in this case JSON:API spec format means we can normalize our data in our clients (specifically, our Redux stores)

Normalizing JSON:API data:
https://github.com/yury-dymov/json-api-normalizer

Libraries like this do a number of things but one of the main benefits is it turns JSON:API responses from collections into maps - indexing resources by their ids which allow for constant time lookups.

Considerations When Using JSON API

REST vs RPC

It is simply a fact that not everything in programming fits nicely into set definitions. There will be times where an endpoint should not be / can not be "RESTful", this will require us to break from JSON:API conventions. This is especially true for things like to batch (multiple) resource actions.

Verbosity

JSON:API is intentionally verbose. This isn't a huge deal especially considering we won't have a ton of relationships etc. and is also something we can fine tune excluding things like links and includes

Applicability for Opentrons

This is where things are slightly fuzzier. If we were building a robust web/mobile application(s) I would say this is a much easier choice. Something like JSON:API spec or even better GraphQL would make a ton of sense. Although, this is not the case - some of these api endpoints and subsequent robot interactions don't necessarily fit nicely into this resource based model provided by JSON:API spec. However, the questions to answer is what we expect the future to hold, could this model makes sense for us currently, and how well does this convention support change within our system.

Libraries and FastAPI

My last concern is around FastAPI and choosing an implementation for JSON:API spec in python. There claim to be a couple framework agnostic libraries that do this:

• https://github.com/marshmallow-code/marshmallow-jsonapi
• https://github.com/kalasjocke/hyp
• https://github.com/xamoom/xamoom-janus

Unfortunately, there are no implementations that mention being directly compatible with FastAPI. In the past, I've used JSON:API spec in tandem with a MVC framework which provided a lot of niceties that may be missing from these libraries.

[sfoster1]

I guess my own reaction is - this looks really nice for avoiding bikeshedding; but I don't think its featureset really overlaps with the problems we have specifically in the robot HTTP API, and I'm not sure whether the added complexity and dependencies would be worth it.

[iansolano]

@sfoster1 this is my initial thinking as well. In fact the exercise of writing this RFC swayed me a bit more towards that camp. The libraries that could potentially work with fastapi also don't seem the best. In the past where using JSON:API has really shined for me is, 1) in web/mobile applications, 2) with a library that worked in tandem with an MVC framework because this cuts down tremendously on boilerplate code.

With that in mind, I have 2 follow up questions for folks:

1. Do we see a future where JSON:API is valuable and does that make moving towards it now helpful?
2. If not, how do we want to standardize response formats? Is the plan to write error and response functions that generate custom JSON responses and or classes similar to ConfigureArgsError?

• In ruby land (sorry, i'm not super familiar with python but should be transferable since it's simply OOP), I might chose to abstract the business logic surrounding the various api actions into service objects. I normally use the pattern of then returning a struct which you can then check on the various values based on their keys. Ex. (in ruby it's an OpenStruct, python equivalent. Below is a sudo code)

// service object
if not success
  return OpenStruct.new(success?: false, user: nil, errors: @user.errors)
end

OpenStruct.new(success?: true, user: @user, errors: nil)

then essentially it's a matter of doing this:

// controller
if result.success?
  do successful things
else
  do unsuccessful thing
end

[mcous]

Thanks for this writeup @iansolano! This has been helpful for me to inspect my assumptions about the new HTTP API.

tl;dr

I see value in JSON:API, but I also think it's not quite right for our robot API because:

• It's too heavy-weight for our needs
• Human-readability is important for us and a reason we selected HTTP over continuing with a (standard) RPC format
• It would require added dependencies (or rolling our own) for sane DX and I'm not convinced of the quality / availability of those dependencies for our full stack

I think we should:

• Define response/resource specs inspired by the good parts of JSON:API that are applicable to our needs
  • e.g. response has links, any resource has id and type
• Probably take the JSON:API error format in its entirety

During development of HAPIv2:

• If we find during the pre-release development that we're thinking, "JSON:API" would be much better for this, we can switch!
• But also if the above happens, I think we seriously reconsider GraphQL
• We also may find that what we've developed is (or can be made into) JSON:API with minimal pain, and it that case it might be a good idea

Likes

• Resources require id and type fields
• links objects
  • Single link object format ({ "href": "string", "meta": { ... } })
• JSON:API's error object spec is more comprehensive than RFC 7807
• Resource CRUD specs are sane

Dislikes

• "relationship" model seems pretty heavyweight for our data
  • Mitigation: can we just not use relationships?
  • Mitigation: maybe it would come in handy once we start building HTTP data models for sessions, commands, and protocols
  • I can't think of any potential relationships where it would make sense to modify data relationships independently of the data
• Forcing resource data into an attributes field
  • Mitigation: I guess we could normalize this on the client end?
• Adhering to JSON:API spec with API versioning in headers isn't spec'd (or even agreed upon by the community)
• Would likely require added dependencies for good DX, but I'm not sure the quality of those dependencies

Neutral

• Primary data always in top-level data field
• Errors always in errors field

[iansolano]

@mcous thanks! Yeah, I think I'm pretty much in exact agreement with your assessment.

[sfoster1]

Same.

[amitlissack]

Regarding FastAPI compatibility, i'd check https://pypi.org/project/pydantic-jsonapi/ out. FastAPI's data modeling/validation/schema generation is entirely built upon Pydantic.

[amitlissack]

I'm in agreement with you all. This looks quite complex to me. All the relationships and linked items seem like overkill for our uses.

[mcous]

Not moving forward with JSON:API, but we are taking lots of inspiration for it for our custom consistent response schema