JSON:API v1.1 New Directions

[dgeb]

A couple months ago, @gabesullice and I requested feedback regarding the adoption of v1.1-RC1 of the spec, which was first published in December of 2018. You can read the request and responses in this issue, which also includes my analysis.

According to our own rules, we saw insufficient adoption of RC1 to move forward with finalization of v1.1, and we think we understand why. The main feature introduced in v1.1 (as of RC1) was "profiles", which are based on IETF RFC 6906. By design, profiles are limited in scope and can't be used to extend the spec to include new document elements and processing rules. Many developers seem to have chosen to sit out implementing v1.1 and wait for v1.2 to solve more pressing problems that can only be solved through extensions of the spec. And others have attempted to stretch the limits of profiles beyond their defined scope, since profiles represent the only "official" tool offered to customize the spec. Both scenarios are understandable, but not desirable.

Since then, Gabe and I have had our heads down trying to plot a new course for v1.1, and we're now ready to make a proposal to the community.

The gist of our proposal is that we would like to simplify and descope profiles as they're currently defined in v1.1-RC1 AND simultaneously introduce a complementary means to customize the spec: "extensions". While profiles can only specify meaning for members already reserved for users, extensions can only specify meaning for members reserved for the spec itself. Because any members added by extensions will be namespaced, extensions do not block any future expansion of the spec. Rather, most extensions should be viewed as a way to explore just such a future expansion.

The details of our proposal are discussed below, along with a few other significant changes planned for v1.1.

Simplify Profiles

The first part of our proposal is to scale back the complexity of profiles as they're defined in v1.1-RC1. The following simplifications are either planned or underway:

• Change profiles to be purely opt-in by the server to align with RFC 6906. A client may request one or more profiles (via the Accept header), but the server is under no obligation to honor that request in the response. The server can choose which profiles to apply, and should reflect those profiles in the returned Content-Type. The concept of a profile query param will be removed, since it also violated the spirit of RFC 6906. This change has been PR'd.

• Remove the ability for profiles to assign meaning to reserved query params such as filter and page. RFC 6906 states that "A profile MUST NOT change the semantics of the resource representation when processed without profile knowledge, so that clients both with and without knowledge of a profiled resource can safely use the same representation.". Since these params are intended to provide processing instructions, they are inappropriate for profiles.

• Move profile authoring guidelines out of the base spec. We can make recommendations for best practices in a non-normative way.

• Remove guarantees about profile registration and publication on jsonapi.org.

Introduce Extensions

Just as profiles can be negotiated via the profile media type parameter, extensions will be negotiable via the ext media type parameter. Unlike profiles, a server will be required to understand and apply every extension requested via Content-Type and Accept headers, or else respond with an error. In this way, clients can be guaranteed that their requests will be processed according to the base spec plus any extensions they've applied.

Extensions are strictly additive to the base spec, and can never impose rules about members already reserved for users. Any members introduced via extensions MUST be namespaced with a : prefix (e.g. local:id). Namespaces can only be used by extensions. Besides the fact that members are namespaced, most extensions could be merged with the base spec.

Extensions will become the means to define custom processing directions for query params such as filter and page.

Sharing Official and Community Extensions and Profiles

Extensions and profiles are both specified by URI, so we (the spec editors) have no means or desire to control which ones are used by the community.

However, we do feel a responsibility to provide a means to share extensions and profiles on the official site (jsonapi.org) just as we share implementations of the spec.

We plan to be cautious in particular about which extensions are seen as "official". We may choose not to support any community extensions at first on jsonapi.org as we wade into this new territory. Essentially, we have to be cautious about making the spec confusing through an endless array of conflicting extensions.

Since profiles are applied purely to "user space", we don't see the need to be so cautious about publishing community profiles.

First Official Extensions

Two of the longest awaited additions to the spec, Local identifiers and Operations, are proposed as the basis for the spec's first "official extensions". Content from the original PRs has been simplified, refactored, and reopened as new PRs that are compatible with the new extension proposal.

The "Atomic Operations" extension (#1437) will use the namespace atomic to emphasize the atomicity guarantee that motivates the need for operations. Operations will be requested via an atomic:operations member, and results returned as atomic:results.

The "Local Identitities" extension (#1436) will use the namespace local. Local identity members (local:id) can be used to identify resources that have not yet been assigned an id by the server. This fills a gap in the base spec in which resources without IDs assigned can't yet reference themselves in relationships. This extension will also be critical to enable resources in different operations to reference each other without IDs.

Note that neither of these extensions will be part of the base spec nor do they need to be versioned along with it. Care will be taken to allow side-by-side evolution. It's possible that one or both of these extensions will land in the base spec eventually (without namespaces of course).

Links Upgrade

The other change that we'd like to support for v1.1 has been PR'd and refined by Gabe for a while: Upgrade links to RFC8288-modelled web links. This PR should help improve hypermedia support by allowing params to be optionally added to links to convey more information about them.

Summary

All aspects of this proposal are either currently in development or already PR'd. We now want to reach out to the community for feedback and support in helping us reboot v1.1 of the spec ASAP. Please let us know your thoughts!

[rwjblue]

Any members introduced via extensions MUST be namespaced with a : prefix (e.g. local:id).

This seems reasonable to me, but just to clarify:

• this means the spec itself is prevented from ever using members with a :, correct?
• this doesn't specifically infer where the members with a : are allowed, is it the intention to allow them anywhere (e.g. within all types of top level object types)? Wouldn't there still need to be some restrictions (to avoid collisions with userland attribute names for example)?

[rwjblue]

Please let us know your thoughts!

I'm excited!

[dgeb]

@rwjblue thanks for reviewing :)

this means the spec itself is prevented from ever using members with a :, correct?

That's correct. The spec will reserve : for namespacing extensions.

this doesn't specifically infer where the members with a : are allowed, is it the intention to allow them anywhere (e.g. within all types of top level object types)? Wouldn't there still need to be some restrictions (to avoid collisions with userland attribute names for example)?

That's correct. Extensions should be viewed as potential additions to the spec itself, so they shouldn't take liberties that the spec couldn't. I feel strongly that extensions shouldn't be able to add members to objects that are reserved for users, just as the spec itself could never add a new member to attributes or meta.

[jugaadi]

A recommendation on how JSON:API can leverage HTTP/2 features like server push would be hugely beneficial for the community. This can be absorbed in the specification in the near future.

cc: @gabesullice

[sandstrom]

Seems like a good way forward regarding extensibility!

Thanks for your work on this @dgeb!

[vasilakisfil]

Excellent stuff. I really like the turn around regarding profiles, because I felt there were being misused (and probably Dret as well).

In an ideal world, we should opt for a decoupled IETF RFC that covers the functionality of the extensions, in relation to RFC 6838 (Media Types). I had done some research (introspected.rest) around Media Types in the past and I had seen the same issues: it's important to be able to extend the spec with small well-known (or even non-well known, although it should be avoided) reusable modules (=extensions), in a way to compose the needed functionality for our Hypermedia Interface. This has many advantages, one of them being evolvability. But current Web Infrastructure don't really allow us to do that. There is not even a proper way to negotiate those extensions, starting from the client side (unless you do reactive negotiation). And that's a big problem, that here JSON:API needs to solve by itself coming up with some custom solutions which I don't think it's a good for the community (introspected.rest did the same).

To sum up, my point is that I completely like/need/want/support extensions, and I am looking forward to that, I just think we have to push in general for a common extension mechanism in HTTP through IETF, regardless what's going to happen in JSON:API.

Because in the end, some of those extensions could be designed in a more generic way, outside of JSON:API (although that would be orders of magnitude more challenging).

[remmeier]

thank you for that large & great effort! as soon as people think the specs are more or less stable I could offer one of the first implementations in the context of crnk.io.

[gabesullice]

@jugaadi, entirely agreed! I'm excited for experimentation/modernization like you describe.

[jugaadi]

When does a proposal(spec related changes, extensions, etc) enter FCP(Final Comment Period) and how long does it last?

[Panman82]

So one concern I have with opening up an extension point is that there might be some things better suited in the spec itself. I've seen other ecosystem open up an extension point like this and everything gets pushed out into an extension and it becomes the wild-wild-west. Additionally, it's easier to create an implementation when things are fairly strict. To that, is it expected that implementations build support for extensions/profiles or are they to open up extension points themselves for others to build support for JSON:API extensions/profiles??

Also, when can we expect a new 1.1 draft? 😅

[jugaadi]

Any updates on v1.1? Our last RC for v1.1 was published last year around the same time.

[dgeb]

@gabesullice and I have been working to simplify profiles for v1.1. Please review #1456 as we'd like to merge this soon.

[dgeb]

Also, please review #1457, which introduces extensions for v1.1.

This completes much of the work described in this issue, and should clear the path for a new v1.1 draft soon.

[jelhan]

Profiles and extensions look awesome.

They could also help to model and document domain- and company-specific conventions. So beside the shared official and community extensions and profiles there might be internal ones.

Not sure if such internal extensions and profiles would be powerful enough to cover all cases. For example they can't put restrictions on fields. So a convention like "the value of an attribute that end with At must be an ISO 8601 date time string" can not be modeled using them. But they should be able to cover most conventions.

[aimeos]

According to the spec, profiles and extensions should be requested by "Accept" or "Content-Type" headers but the server may return a very different output depending on the profile or extension. Do (reverse) proxies treat those headers the same as GET parameters and use them to return the appropriate cached content? I don't think so.

If there's a proxy between client and server, it will return the first cached request regardless of the "Accept" or "Content-Type" headers because it will make its decision purely on the GET parameters. Thus, the requested profiles or extensions must be part of the GET parameters if I'm correct.

[aimeos]

One of the most tedious parts of the 1.0 specification is the "included" array being a flat array because you have to loop over all elements to build a map of types and IDs that can be accessed directly when resolving relationships. The original intend was that those included elements are in the correct order but that's irrelevant because the "relationships" in the "data" sections define the order of the related/included items for each data entry.

Now you say:

Extensions are strictly additive to the base spec, and can never impose rules about members already reserved for users.

Thus, it would be only possible to add a new "map:included" element to return a structure like:

"map:included": {
    "<type>": {
        "<id>": {
            "id": "<id>",
            "type": "<type>",
            "attributes": {}
        }
}

If the "included" array from 1.0 spec can't be removed or left empty, this would double the amount of data returned, which is highly undesireable. In my opition, there should be a way to replace or remove existing elements by extensions to solve problem of the 1.x spec until 2.0 fixes the problem.

[geoidesic]

What about m2m relationships? Are there no plans to support this and if not could someone please explain the rationale behind not supporting this?

[jelhan]

One of the most tedious parts of the 1.0 specification is the "included" array being a flat array because you have to loop over all elements to build a map of types and IDs that can be accessed directly when resolving relationships.

@aimeos Could you elaborate on this a little bit. I haven't ever seen it as a limitation in an existing library. Only heard if people try to hack together ad-hoc solutions. Do you have examples for which such a map would be required?

The original intend was that those included elements are in the correct order

Can't follow you here. Do you have a source?

What about m2m relationships? Are there no plans to support this and if not could someone please explain the rationale behind not supporting this?

@geoidesic Many-to-many relationships are supported. Could you link to a more concrete explanation of the limitations you are facing?

[geoidesic]

@jelhan

I mean specifically when POSTing.

Let's say e.g. I have Enquiries and People as an m2m relationship in my database via a link table. Now I'm creating a new Enquiry on the front-end and in the process, I want to also:

A. Link some existing People to that Enquiry (I understand that this can be done)
B. Create some People and link them to the Enquiry (my understanding is that this cannot be done as part of the same request)

It seems the only way to B currently is as two separate requests – first POST to create the People, then add these ids to the People relationship of the enquiry before posting it.

There are two problems with this:

1. It doesn't support an offline / synch architecture. In an offline architecture, each of the two requests would be serialised and put in a queue. Because they are serialised static requests, the Enquiry request can't receive the id's of the created People.
2. Also, having it as two requests prevents one from doing it all as one database transaction, which may be counterproductive in certain instances. E.g. you might not want the Enquiry to be created if there's a validation problem on the related Person.

Maybe I'm approaching the JSONAPI architecture in the wrong way, so please let me know if there's already a pattern for handling this effectively.

The only solution I can think of to these problems within the existing spec is to use client-generated uuids instead of integer primary keys – I'm not wild about the idea because that would represent a big architectural change to the app with all kinds of potential fall-out.

[jelhan]

@geoidesic This should be provided by Atomic Operations extension. See #1437 for details.

[aimeos]

One of the most tedious parts of the 1.0 specification is the "included" array being a flat array because you have to loop over all elements to build a map of types and IDs that can be accessed directly when resolving relationships.

@aimeos Could you elaborate on this a little bit. I haven't ever seen it as a limitation in an existing library. Only heard if people try to hack together ad-hoc solutions. Do you have examples for which such a map would be required?

The original intend was that those included elements are in the correct order

Can't follow you here. Do you have a source?

@jelhan Consider a typical response like this:

"data": [{
  "id": "37", "type": "product",
  "relationships": {
    "price": {
      "data": [{
        "id": "321", "type": "price"
      }]
    }
  }
}, {
  "id": "48", "type": "product",
  "relationships": {
    "price": {
      "data": [{
        "id": "283", "type": "price"
      }]
    }
  }
}],
"included": [{
  "id": "283", "type": "price", "attributes": {}
}, {
  "id": "321",  "type": "price", "attributes": {}
}]

To get the referenced price items in the included array, you have to loop over the complete array once because you can't directly access the right price item. This would be possible if included would be a nested object with the type and id as keys:

"included": {
  "price": {
    "283": {
      "id": "283", "type": "price", "attributes": {}
    },
    "321": {
      "id": "321",  "type": "price", "attributes": {}
    }
  }
}

Then, you could access the referenced price item directly by

response.data.included["price"]["281"]

[dgeb]

Then, you could access the referenced price item directly by response.data.included["price"]["281"]

@aimeos There is no guarantee of this, which is why we didn't pursue this approach in the first place. The base specification requires that only one representation of a resource can be in data and included, and if that resource appears in data, then it must not be duplicated in included. This could happen in your example if resources of type price are both primary and included.

Note that extensions could be used to add new ways to embed, include, and nest resources. However, they should not attempt to redefine existing members in the document structure (like included) nor reserved query params with meaning (like include). So you could create an extension that supports your map:included structure as long as it doesn't attempt to repurpose the already reserved include query param (but it could support map:include).

[dgeb]

They could also help to model and document domain- and company-specific conventions. So beside the shared official and community extensions and profiles there might be internal ones.

@jelhan that's definitely something we've had in mind :)

Not sure if such internal extensions and profiles would be powerful enough to cover all cases. For example they can't put restrictions on fields. So a convention like "the value of an attribute that end with At must be an ISO 8601 date time string" can not be modeled using them. But they should be able to cover most conventions.

The v1.1 draft includes support for the describedby field in links. This field can point to a description document (e.g. JSON Schema or OpenAPI) that absolutely can make requirements on field contents. See https://jsonapi.org/format/1.1/#document-links

An example extension might strictly require usage of this field and enforce field validation according to it.

[aimeos]

Then, you could access the referenced price item directly by response.data.included["price"]["281"]

@aimeos There is no guarantee of this, which is why we didn't pursue this approach in the first place. The base specification requires that only one representation of a resource can be in data and included, and if that resource appears in data, then it must not be duplicated in included. This could happen in your example if resources of type price are both primary and included.

@dgeb In data but not in included isn't a problem in most cases and doesn't affect if the included section is a flat array or structured as nested objects IMHO. If the resource already is in data, there's some extra code required but nested objects improve lookup speed a lot and avoids code for creating that map after each response. There are no guarantees, so the code must always look like

response.data.included && response.data.included["price"] && response.data.included["price"]["281"]

Note that extensions could be used to add new ways to embed, include, and nest resources. However, they should not attempt to redefine existing members in the document structure (like included) nor reserved query params with meaning (like include). So you could create an extension that supports your map:included structure as long as it doesn't attempt to repurpose the already reserved include query param (but it could support map:include).

Yes, in v1.x it's impossible to change that to keep backwards compatibility. Nevertheless, it would be a great improvement for v2 and GraphQL offers more structured data too which makes it much easier for frontend developers.

[jugaadi]

Any updates on publishing v1.1?

[jugaadi]

@dgeb @gabesullice It would be really helpful if you can give us any update regarding v1.1 status.

[dgeb]

@jugaadi We are both independently taking a pass through the v1.1 spec and plan to issue RC2 after making some final (minor) edits.

Now's a good time for others to do the same :)

[aimeos]

According to the spec, profiles and extensions should be requested by "Accept" or "Content-Type" headers but the server may return a very different output depending on the profile or extension. Do (reverse) proxies treat those headers the same as GET parameters and use them to return the appropriate cached content? I don't think so.

If there's a proxy between client and server, it will return the first cached request regardless of the "Accept" or "Content-Type" headers because it will make its decision purely on the GET parameters. Thus, the requested profiles or extensions must be part of the GET parameters if I'm correct.

@dgeb What about profiles and proxies?

[dgeb]

If there's a proxy between client and server, it will return the first cached request regardless of the "Accept" or "Content-Type" headers because it will make its decision purely on the GET parameters. Thus, the requested profiles or extensions must be part of the GET parameters if I'm correct.

If we could not rely on content type negotiation to work with the Accept header, then clients could not specify that they want the json:api media type (application/vnd.api+json). For that matter, they couldn't request application/json vs. application/xml. Certainly some APIs form their urls with these concerns in mind, but this is definitely beyond the scope of this specification.

[jelhan]

I'm very happy to see v1.1 RC2. 🎉

I have a question regarding the official extensions planned to ship with v1.1. In the first post of this issue it was proposed that local identifiers and atomic operations will be released together with v1.1 as official extensions:

Two of the longest awaited additions to the spec, Local identifiers and Operations, are proposed as the basis for the spec's first "official extensions".

Local identifiers were shipped as part of the core specification with v1.1 RC2. The reasons for shipping this as part of the core specification rather than an official extension are explained here.

Does that affect the plan to ship atomic operations as an official extension together with v1.1?

[dgeb]

Does that affect the plan to ship atomic operations as an official extension together with v1.1?

@jelhan Thanks for checking on this. The updated plan still includes shipping atomic operations as an official extension. And now that v1.1 RC2 has shipped, wrapping up #1437 is our highest priority.

[freddrake]

For what it's worth, I'm finding it quite difficult to be interested in 1.1 when it remains difficult to get questions about 1.0 answered. I've tried both the issue tracker and the general forum; while the forum provides responses, they're typically not definitive.

As a result, my own JSON:API support library is pretty solidly stalled, and I see little point in making it public if there's no future for it.

[auvipy]

For what it's worth, I'm finding it quite difficult to be interested in 1.1 when it remains difficult to get questions about 1.0 answered. I've tried both the issue tracker and the general forum; while the forum provides responses, they're typically not definitive.

As a result, my own JSON:API support library is pretty solidly stalled, and I see little point in making it public if there's no future for it.

1.1 is more definitive

[freddrake]

1.1 is more definitive

Agreed that 1.1 is better with regard to clarity. But questions I've had regarding 1.0 have generally not been handled better in 1.1 (see #1496, #1502 for examples).

I'm not suggesting that no answers are forthcoming, either; @gabesullice helped work out clarifications for #1499, for example.

[dgeb]

As with #1403, I'm closing this now that v1.1 RC3 has been released with all the planned updates discussed here. At this point we're just making fine tweaks and waiting for a couple initial implementations to be finished.