Rethink extension negotiation

[gr0uch]

I had a bad feeling about implementing the official extensions, or really any extensions at all to the spec, since extensions may completely break the base specification.

In a comment regarding content negotiation, Roy Fielding writes:

it is a waste of time to negotiate on the basis of obscure parameters while using a meaningless media type like vnd.api+json. Define separate media types (preferably with meaning) if you think the differences will matter to a client. You will note that the examples in the spec are artificial, since there isn't any real-world need to negotiate on parameter names.

"meaningless media type like vnd.api+json"... ouch ;)

[steveklabnik]

Where did fielding make that comment? I didn't even know he was aware.

On May 12, 2015, at 17:20, Dali Zheng notifications@github.com wrote:

I had a bad feeling about implementing the official extensions, or really any extensions at all to the spec, since extensions may completely break the base specification.

In a comment regarding content negotiation, Roy Fielding writes:

it is a waste of time to negotiate on the basis of obscure parameters while using a meaningless media type like vnd.api+json. Define separate media types (preferably with meaning) if you think the differences will matter to a client. You will note that the examples in the spec are artificial, since there isn't any real-world need to negotiate on parameter names.

"meaningless media type like vnd.api+json"... ouch ;)

―
Reply to this email directly or view it on GitHub.

[gr0uch]

I accidentally copied and pasted the wrong link when I posted it, should be fixed in the OP now.

[steveklabnik]

ah interesting. I kinda agree with @royfielding here, though I obviously disagree with 'meaningless'

[dgeb]

Taken from the same comment:

It is assumed that a server capable of meaningful negotiation based on a media type parameter is also going to know what the parameter means (or provide a ranked configuration interface for the resource owner to specify how to handle them). Naturally, this might be media-type specific and is certainly server implementation-specific.

So this is the approach we are taking.

He goes on to state:

In any case, commas are not allowed inside of parameter values unless the value is enclosed in DQUOTEs, so several of those examples are syntactically invalid.

He's correct about this and this is also clearly stated in json-api's format section. Some of the examples in that issue are invalid.

As for the "meaningless" comment: I suspect it was made because the media type is in the vendor tree and not the standards tree. It may have been made without even connecting vnd.api+json to this effort. Regardless, there is clearly meaning to this spec, even if it is not an official standard.

[ethanresnick]

Yup, I've been worried this too. (I asked Roy on twitter about the other issue, which is what prompted his comment.) My tentative feeling now is that:

1. Extensions shouldn't be allowed to break compatibility with the base spec. If they need to do this, they're just their own media type. Calling something an "extension" when it's allowed to totally rewrite the rules is nonsensical.
2. To keep the bulk extension's functionality, we can just redefine it as an optional part of the base spec, rather than an extension. (And perhaps we can break the optional parts out more clearly in the text to make the spec seem less overwhelming.) As with the other optional parts (sorting, include, etc), servers can return 400 if they don't support it. This means that generic clients may have to waste a request to detect feature support, but oh well. Plus, we can always retroactively add an (optional) advertising mechanism post 1.0, via a media-type parameter or otherwise.
3. The extensions that are left are then purely additive to the document (e.g. for request signing, inline documentation, whatever), and are defined in such a way that they're guaranteed to remain additive over time (Additive Extensions Should Remain Additive #603). This dramatically simplifies reasoning about the ext/supported-ext parameters, and makes it possible to do so in a generic way.
4. (Optional) We could then even move client requests for extensions out of the accept header and into a standard query param, which has the advantage of being easier to manipulate. Plus, http-header-based content-negotiation is almost guaranteed to trip up implementors, so it seems like less of it is better.

EDIT: I started writing my comment before @dgeb published his, so sorry about the cross-noise. The above was written mostly in response to @steveklabnik and @daliwali. I agree with @dgeb also that the "meaningless" quote and the stuff about DQUOTE tokens is beside the point. But I still think there's a substantive point here about how we could refine extension negotiation to make it easier to reason about.

[gr0uch]

The way I interpret his comment is that the base spec is meaningless given that parameters may override any part of it. I think this issue is closed prematurely because the point hasn't been discussed: negotiating parameters may be the wrong approach, especially if those parameters alter the format of the base spec.

Also I made a syntactic error by omitting the double quotes, but it doesn't affect the point.

[steveklabnik]

Extensions shouldn't be allowed to break compatibility with the base spec.

Yeah, this is something i worry about.

[dgeb]

Ok, reopening to discuss making extensions additive-only.

[royfielding]

Well, keep in mind that I only looked at this issue — I don't know what your specification says.

JSON is a meaningless data format, and API certainly doesn't add any meaning, and the name doesn't even use the vnd tree properly, so my conclusion is that the media type is a meaningless example. Note that, if you do have a lot of meaning attached to that name, that makes it a poor media type name for something that does have meaning. Use a distinct name, like all the other media types.

[gr0uch]

The name "JSON API" sounds like a generic term for any JSON-over-HTTP implementation, and bound to cause confusion for everyone, including those who have heard of the spec (do they mean their API uses JSON, or that it follows the "JSON API" spec?)

The main use case for this spec seems to be for tightly coupled client-side applications (case in point: Ember Data). I could throw out some name suggestions but it's probably already too late for a name change.

[royfielding]

Alright, this comment is completely out of scope for this issue, but I need to close out my thoughts somewhere.

In short, I see that you already registered the media type name, which means you might as well keep using it. Please understand that, if I were the expert that reviewed the registration, it would have been rejected. Placing a generic, standardizable media type in the vnd tree is just abusing the registration process. Of course, this makes no difference to HTTP or any application thereof, and is still better than using an unregistered type.

What I mean by a meaningless type is what a client might decide by looking at the type name. In this case, all they can really decide is that they should invoke a JSON parser and should expect some common structure according to your spec. That's not much. I can see why a client might want to use such a type in Accept, but not why it would want to negotiate on the basis of parameters of that type. If you expect an extension to only be implemented by some clients in a way that other implementations cannot simply ignore, then you should give it a different media type. If the purpose of this type extension is to supply an extended representation, such as a full traversal of a collection instead of just the top page, then that is a different resource and should have a different URI. Supply a link instead.

Note that this is a very common problem among folks working on RESTful APIs. Half seem to think that all resources are collections and each representation must be complete (e.g., Siren). The other half seems to think that all resources are pages (e.g., Atom) and everything must be a blog. Neither is right. Some resources are collections, some resources are pages, some resources are indexed results, some resources are graph trees, some resources are images. All of them can show the same data in different ways and still be separate resources. A resource's value is as much a reflection of its consistency across representations (sameness over time) as it is the data it provides.

[hhware]

Another off-topic comment: IMHO, the name of the spec is its weak point. It is like calling your child Homo Sapiens -- despite being factually correct, I doubt anybody would do that. Even though some like taxonomy-based names (e.g., Sun DHCP Server, Sun Identity Manager, etc.), I guess their hidden intention is to make it difficult to find information about these things on the web. However, selection of name is parent's privilege, so this discussion is indeed late, and it will become too late in about a week...

[ethanresnick]

The media type name is settled, so let’s get back to talking about extensions. (Also, ping @bintoro, who helped with the initial design.)

I want to put forward what I think the goals for the extension system should be and how I’ve been thinking about the different types of extensions. I still don’t have a preferred negotiation system, but I’m pretty sure the current one isn’t what we want, and I think this is important to resolve pre-1.0.

Given how close we are to launch, I want to share my thoughts ASAP, even though they're half-baked, to see what others think and to prompt discussion. Hopefully you guys will have ideas for how to specify extension constraints and a real negotiation system using these ideas.

Ok, here goes. It seems like the goals are:

1. Extensions should have enough freedom to add missing features to the spec, which is important for spec adoption.
2. But we want to balance that with preserving useful invariants. For example, a guarantee that extensions will compose in predictable ways could dramatically simplify library implementation of extensions and perhaps enable an “extension ecosystem” of sorts, in which users can pick and choose the extensions they want and libraries can facilitate “plugging these in”. Other useful invariants, e.g. about not contradicting base spec semantics, could simplify extension negotiation. For example, it could allow servers to automatically opt clients into various extensions by default, for convenience.
3. Finally, we want to discourage extensions that will confuse users about what json-api is or that will make it more difficult to use off-the-shelf tooling, which is a core part of the spec’s raison d’être. Extensions that redefine the semantics of payloads that are valid under the base spec, or that use different payload structures to more-or-less duplicate base spec features, would fall into this category.

The second and third goals seem to imply that we should put some limits on extensions. Also, given that we can always loosen the constraints on extensions later—but we can’t really tighten them post 1.0—it seems like we should air on the side of over-constraining extensions now rather than under constraining them.

For those constraints, the first and third goals seem to imply that we want, at root, is for the extensions to be additive in some sense. But I see at least three meaningful senses of “additive”:

1. The first type of “additive” extension is one that allows the server to accept requests that, under the base spec, would have been a 4xx error or had undefined behavior. In the newly-accepted requests, the extension defines the semantics of the payload elements. The jsonpatch extension and the bulk extension are both additive in this sense; they turn 400s into 200s, with custom semantics for the newly-allowed requests. Extensions like this are:

   1. Not guaranteed to compose/interoperate with one another;
   2. Not guaranteed to remain additive to the base spec, which could define conflicting semantics for those previously undefined requests.

   Nevertheless, so long as they’re truly opt-in, they can’t cause problems for clients that don’t request them. And they allow a lot of power.

   Note also that, the newly-defined requests these extensions allow might not simply be valid in virtue of new payload structures. These extensions could instead expand the set of accepted requests by coining new resources (in the HTTP sense), e.g. by adding query parameters, or they could describe the behavior of an existing resource for a new method (like PUT).

2. The second type of additive extension is one that doesn’t change the set of requests that the server accepts, and doesn’t change the semantics for any base-spec-defined members in the request/response payload. All the extension does is add data (or perhaps behavior) to requests that already have defined semantics, without interfering with those semantics. An example extension like this would be one that adds a template for writes or other forms of inline documentation. An extension of this type that adds extra behavior might, for example, be one that accepts an extra member on writes to suggest some cache invalidation to consumers that understand the request. These features, which could live under meta but would be less visible there, can be quite useful as extensions. They also have a few nice properties:

   1. They’re composable; multiple extensions of this type can be added to the same request/response independently without problem. As I mentioned above, this could foster an extension ecosystem, in which multiple APIs can share the same extensions (their semantics if not the code), and this could become a breeding ground for new official extensions.
   2. The client can be automatically opted-in to these extensions.
   3. They can be scoped to never conflict with future base spec additions (see Additive Extensions Should Remain Additive #603).

3. Finally, there are extensions that are additive in the sense of adding extra constraints to what constitutes a valid request, but then treating requests that pass these extra constraints according to base spec semantics. A request signing extension (or anything to do with auth/security) would be a good example. Another example would be an extension that mandates certain type values. These extensions:

   1. Can cause problems for generic clients that don’t know how to sign the requests/provide credentials/etc, because they turn 200s into 400s. This is unfortunate but, at least for security related extensions that the client can’t turn off, seems inevitable. Moreover, I don't think this constitutes a base spec break since, by any sensible reading of the current text, the server is already allowed to reject requests that fail to meet its auth or validation constraints.
   2. Can be required by the server. This is quite unlike the other two extension types, and might to have implications for negotiation. Having a required extension is also not something covered by our current extension mechanism.
   3. Like the second type of additive extensions, these extensions are composable so long as the constraints they add aren't contradictory and, for the requests they allow through, they follow the base spec’s semantics.

Ok, I’ll leave it there for now and follow up in a subsequent comment with thoughts about potential designs and the design space I’ve been exploring. I’m confident that we can come up with a pretty simple design with constructs powerful enough to cover all these cases, it’ll just require some thinking. I'm looking forward to hearing ideas others have too!

[cziegenberg]

@ethanresnick:
I also think that extensions should be better defined before 1.0, because that's very important for the acceptance of JSON API and to avoid incompatibilities between extensions and with with future versions of the spec.

In #574 I suggested to use an own namespace for extensions (very similar to your proposal #603) to avoid conflicts - but that's only one problem.

Other thinks to consider (in addition to your points):

• What if extensions use the same name? This could still result in conflicts. Do you offer a global extension registration to avoid this? How to work with own (inofficial, API specific) extensions at the same time?
• Extension could also use own query parameters. How to avoid conflicts there? (Related to Defining Future Query Parameters #602)

[ethanresnick]

Some further observations on the types of extensions:

1. The second type of extension in my taxonomy above perfectly fits the definition of an RFC 6906 profile, so it seems like we should use the existing profile specification to signal these extensions, unless there’s a compelling reason not to. For the third type of additive extension, it seems gray to me whether these fit the definition of a profile, so I’m curious what others think. That said, I’m currently leaning towards thinking that they could be profiles too.
2. However, the first type of “additive extension” seems not to be a profile, because these extensions can accept/respond with representations that would have been invalid under the base spec or that have conflicting semantics. For example, a PATCH update under the jsonpatch extension does not contain a resource object under ”data”, which is what a generic base-spec server would expect. And the response to a bulk POST likewise doesn’t contain a resource object at the “data” key. In both cases, the client or server can’t derive any meaning from the extensions’ newly-defined payloads unless it knows about new processing rules, so it’s not the case that “the original semantics and processing model of the media type still apply, but that an additional processing model can be used to extract additional semantics.”

I think the above suggests that we should have two fundamental types of extensions that the design should revolve around: profiles, which have these nice composability invariants, and what I'll call the "type-1 extensions", for lack of a better term, that have more power.

[ethanresnick]

Some thoughts on constraining "type-1 extensions":

As I mentioned in my initial post, constraining type-1 extensions at all isn't strictly necessary. Even if they blatantly conflict with the base spec, either at the time of their authoring or down the line (if the base spec adds a similar feature), this won't cause problems for clients as long as the client has explicitly opted in to the extension's meaning.

That said, I still think we ought to put some constraints in place to discourage or ensure that "type-1 extensions" don't redefine the semantics of payloads that are already valid under the base spec and to discourage them from using different payload structures to more-or-less duplicate base spec features. (This is goal 3 from my original post.)

Below is one constraint regime I've been thinking about for accomplishing the above, that I'd like feedback on:

1. We commit to only making "type-2" additions to the base spec. This allows us to add new members, like "immutable" and "template" to the base spec down the line, but only to change the structure of existing members through official extensions.
2. Then, all type-1 extensions are required to support a superset of the requests accepted under the base spec. So, a server supporting the bulk extension would still have to accept a request, even if ext=bulk is in use, with only a single resource object in "data", as the base spec allows. (This is only possible once (1) is in place, as otherwise an extension could define a valid request format that becomes non-additive/incompatible with the base spec later.)

Let's look at what these constraints cost us and what they give us:

• For the first constraint, we already can't change the structure of an existing member in the base spec in most cases, as that would break old clients. For example, if we later said in the base spec that the value of meta could be a string "in addition to" it being allowed to be an object, all hell would break loose. So, right now, we only have the freedom to redefine the structure of existing members when a client opts in to that redefintion, which could happen in two ways:

  • On GET requests, the client could supply a query parameter or header that we define.
  • On POST/PATCH/DELETE requests, the client could simply use the extended format. If the server doesn't support the new format, it will respond with an error and the client, which (by virtue of it being able to make the request) was necessarily built after the new format was added, will know that that non-support was a possibility and can carry on appropriately.

  In the the GET case, though, having the client negotiate for an extension doesn't seem (much) more cumbersome than having it provide a query parameter, if done right. So I don't think we're loosing much there. The biggest concession we're making, then, is that clients can't simply use extended formats in their write requests, and we can't simply add extended write formats to the base spec. Instead, we have to define new write formats in extensions. But, given that this is exactly what we're already doing (for the bulk and patch extended write syntaxes), this still doesn't seem like a big cost.

• In exchange for slightly limiting ourselves in the first constraint, we get the second constraint, which:

  • makes it clearer in what sense the extensions are actually "extensions", as opposed to entirely new specs;
  • creates a disincentive for defining extensions that nearly duplicate or redefine base spec functionality, since that base functionality still has to be supported
  • simplifies the client and server implementation of extension handling, since neither end would have to disable/override its logic for base spec requests when an extension is in use, but instead could just augment that logic;
  • likewise simplifies the process of authoring an extension, since the author only has to specify the behavior in the newly-allowed cases. For example, the bulk spec currently doesn't specify what happens if a single resource (i.e. base-spec compliant) request is made. Should it be a 400? Should it proceed as under the base spec? With the second constraint, the request would be handled as under the base spec (again, since bulk is just an extension of that spec).
  • also reduces the amount of header shuffling that clients have to do. For example, if a client app is using the base spec plus some bulk operations, the developer can just set Content-Type: application/vnd.api+json; ext=bulk once in some config somewhere, and not have to worry about whether, for each request his app makes, it's a bulk request or not.

Thoughts?

[ethanresnick]

On JSON-LD and prefixing:

I'd really like it to be possible for a JSON-LD extension on top of JSON-API to be a "type-2" extension and therefore easier to integrate, since I think JSON-LD will only grow in importance. But the only way for a JSON-LD extension to be type two is for the spec to allow type-two extensions to put keys anywhere, at least if they're @-prefixed. But, as we're also going to make type-two additions to the base spec over time, those extension-added keys have to be known to not conflict with future base spec keys, which points the way toward requiring that unofficial type-two extensions use prefixed keys. And, once you have prefixed keys, there's no need for the "extensions" container (which was targeted at type-2 extensions), and you can come up with a unified user-key prefixing strategy to solve the query parameter problem (#602) and get the benefits of a visual indicator for which bits of a response are custom and which bits come from the spec.

[dgeb]

@ethanresnick we don't have a fixed time to tag 1.0 tomorrow. But we will tag 1.0 tomorrow.

I am following your comments from a taxonomical perspective (delineating the types of extensions) but I am unclear how these delineations will coalesce into changes to the base specification. As far as I'm concerned, the most important aspect of any extension is that it is an agreed upon contract between client and server.

I don't favor buckets such as an extensions container at different levels, since extensions can cover so much more than adding members to different objects. I am also unclear on the value of requiring extension-specific namespacing when extensions can also alter non-structural aspects.

I am hopeful that we can keep the rules for extensions rather open-ended and allow different "types" of extensions to evolve organically, using this specification as a baseline.

I will re-read your comments above and additional thoughts in the morning. Thanks for thinking deeply about this!

[orubel]

@royfielding I would also add that people confuse resource with endpoint; in a distributed architecture, binding communication logic to business logic creates an 'architectural cross cutting concern'. This is the way the original API Pattern works as it was intended for centralized architectural... never distributed architectures.

To fix this, you have to abstract the communication logic from the business logic so that you can share and synchronize the communication data (ie I/O state).

https://developers.redhat.com/blog/2017/10/12/new-api-pattern/

[wimleers]

Profiles are now in 1.1, see #1268.

What's next here?

[ethanresnick]

@wimleers Nothing, at least for the time being.

[jelhan]

Closing this as extension and profile negotiation has been settled in v1.1. I think most importantly extensions are now additive only.