Overview: Extending and describing the spec #1207
Comments
An Introduction:Json:api is the language of the sciences: *cites several examples, here.* In short it's very use-case is in the betterment of a greater whole : concision, pieces of ethical conclusion -- atomic life. 'Dissolving over breaking'While it is of an astute nomenclature in addressing it's primary goal (warding) -- I feel it's overtone to be a bit dry: whereas 'dissolving or dissolved' relates to me a more even temperament in moving forward. It was the creator's intention to resolve remedial language, though clear as it should come to mean in time dissolving to me merits a more thorough gaze(perspective) through which to move forward along. 'Profile'I believe in Ethan's (@ethanresnick) concept of profilitic inheritance but would argue semantically addressing it as 'shell (as in electron shell).' This definition addresses it's inert atomic or particular nature in moving forward. In Conclusion:generating instances of the correct perspective on any level or framework are crucial -- sealing how these components look through their years of antiquity (atomically). Additional consignments:names along this(generalized scientific nomenclature) vein ei: References: |
|
Hey Dan! Thanks for getting this conversation started. To dive straight into your questions:
Yes.
I definitely see the appeal of allowing an extension to specify both information in the document body ("structure") and how to use any related functionality ("usage"). These things go together conceptually, and will be implemented them together, so specifying them separately would be awkward. However, I'm not sure that the way to acheive this is to let extensions dictate query parameter usage. Allowing extensions to specify how query parameters work (or anything about the URI) opens back up the possibility of extensions conflicting, both with one another and with other query parameters the server is already using. For example, imagine an API that uses the filter query parameter like so: Of course, as the number of extensions grows over time, the risk of these collision grows too. Because of this risk of collision (and a couple other things), having a specification dictate anything about URLs violates an IETF best practice (RFC 7320), and the logic in that best practice all seems to apply to JSON:API extensions.
The big approach that we haven't discussed much (in the context of extensions), but that is the conventional solution to this problem in the REST school of thought, is links/request templating. Under this approach, the hypothetical pagination extension would define a key where the user of the extension (the server) could place a URL template that the client can use to go to an arbitrary parge. For example, with the profile extensions design from #1195, that might look like this: In the example above, the extension has defined that the Applying this to the filter example above, the extension could define template variables for various filter operators, and the server would be able to provide a template that uses those variables to construct urls that don't conflict with any URLs it's already using. Because most clients are written (or explicitly configured) for one API, they will probably hardcode knowledge of whatever URL construction pattern the server uses, rather than reading it out of the body. Therefore, they won't have to bother making an initial GET request to read the template, nor will they have to know how to interpret url templates(!), but the server will still be able to use the extension without conflict. A truly generic client will need an initial request to read the template, but such a request would be necessary even if the query parameters for a given extension were fixed. (In that case, the initial request would be necessary to determine what extensions are available.) Therefore, this templated approach doesn't impose any extra HTTP overhead. It does require that truly generic clients know how to parse URL templates, but that is a standard and there are good libraries for handling it in every language for which we have a client. As another aside, the example above is kinda ad-hoc. The extension defines a key to contain the link, but this key doesn't tie into JSON:API's Long-term, I also think a hypermedia model probably generalizes better and gets less fragmented than other approaches.
I think this goes back to why someone would define something as an additive extension vs. as a conformant extension in the first place. Consider three of the examples you gave for additive extensions: bulk posting, side posting, and operations. If profile extensions that weren't understood by the recipient produced an error, rather than simply being ignored, than there's no reason any of those couldn't be defined as a profile extension. For bulk posting, you'd have The requests above are all expected to trigger specific, necessary side effects, not just communicate some information or trigger optional side effects, so it's not acceptable for the server to process them if it doesn't understand them completely. That, imo, is the key difference between additive and conformant/profile extensions, as currently defined. The latter don't trigger errors if they're not understood, and instead they are ignored (which is probably to be preferred for interoperability and evolvability whenver possible). Given this, it's important that the recipient of a document knows whether the extensions applied to it are ignorable or not, and I think that would lead to a separate media type parameters for identifying conformant extensions even if we tackled additive extensions at the same. I'll note that the design back in #650, which did try to tackle conformant and additive extensions at the same time, still had two distinct media type parameters for this reason ( (To your additive/subtractive fields example, btw: I actually wouldn't define that as an additive extension; I'd make it a profile extension that just adds one link template to the document, where that template takes two variables: a list of fields to add to the base set and a list of fields to exclude from it.)
I've always thought that additive extensions should happen eventually—although, more recently, I've actually been wondering whether it would make sense to have every would-be additive extension exist simply as an optional part of the base spec. Either way, though, I'm open to holding off on profile extensions if there are additive extension-related features that we think would come out substantially worse/less coherently if we added profile extensions standalone first. That said, my current thinking is still that profile extensions probably are severable without leading to too much undue fragmentation later. |
|
@ethanresnick Hey Ethan - always appreciate your thoughts! Thanks for weighing in. I can't respond to everything at the moment, but will try to zoom in on a central theme right now.
If JSON:API did not already reserve and provide semantics for query parameters, such as
The base spec makes pragmatic concessions to IETF best practices recommendations by narrowly defining usage of specific query parameters. These are the only aspects of the URL that are reserved by the spec. I suppose we shouldn't try to re-litigate this post-1.0. I'm just pointing out that, for better or worse, any best practices violations have already been made by the base spec. I think the bottom line is that I am probably less concerned about managing conflicts between conformant extensions than you are. Given that we can't constrain the creation and use of conformant extensions, any number of them will be created to provide conflicting semantics for a particular aspect of document structure, such as a
I'm definitely not opposed to expanding support for link templating. But I'm not sure it's a panacea for fully spec'ing out query param options. Imagine a complex filtering scheme described by an extension that specifies that the value of a filter be an encoded json object in the form Anyway, I think that we agree more than we disagree, so sorry to focus on what I see as the main disagreement. Maybe we need to bring more use cases and perspectives into this conversation to help us work through this. @wycats ? @steveklabnik ? And I'll try to respond to your other points ASAP. |
@ethanresnick I would not consider allowing extra keys, such as Anyway, I'm not sure if your argument here was to strawman how these extensions could be made conformant instead of additive. I think they're much cleaner and clearer as additive extensions, which would need to be negotiated (at least as long as they remain extensions and not part of a future version of the base spec). |
This seems to me to be a reasonable alternative. Including the spec version as a media type param would be sufficient for clients and servers to acknowledge the potential for additive features. This approach would probably force a decision re: bulk + side posting vs. operations in the base spec. I can understand them coexisting as extensions, but it makes little sense for features with any overlap to coexist. This may force some harder decisions sooner, but that may not be a bad thing.
Perhaps we can inch towards accepting some concepts related to profiles? For instance, we could allow a
I think that, from the spec's perspective, this essentially means that content type negotiation should not fail because of conformant/profile extensions (and I agree). |
|
Hey Dan — sorry it's taken me a couple days to get back to this. There's lots to think about here, and I wanted to do it all justice. Also, a heads up: I've got a crazy week this week, so I probably won't be able to respond again for a few days. (If that means we need to extend the end date of this conversation beyond the 11th a bit, that's obviously fine with me.) Second heads up: I apologize this is so long and perhaps rambly in places. As Pascal said, I didn't have the time to make it shorter; please read charitably :) InteractionsBefore responding to your comments, I also want surface a new issue—one that's only come to mind as we've gone into the details here. That is: if extensions can add query parameters to the url, what happens when multiple extensions do so at the same time? How do their query parameters interact? To me, this is a very different question than managing conflicts between extensions, because, when two extensions conflict—by which I mean that both want to use the same query parameters or document members—the server simply elects not to use one of them (if I'm understanding your proposal correctly). Here, there's not a conflict per se, but there's instead truly undefined behavior. Take a simple case where a pagination and a filtering extension are both in use at the same time. The requested URL will presumably have a pagination parameter and a filtering parameter. Now, assuming each of those extensions is specified separately, how should they interact? Should the server paginate the collection and then filter the requested page? Or, should the server filter the collection and then paginate the filtered results? In this simple case, it's obvious that filtering should be applied first and pagination second. And maybe our rules for extensions could even hard-code this as a special rule. But, in the general case, how to combine two extensions in a request seems like it would be totally undefined, as presumably each extension can't be specified with knowledge of what other extensions it might be applied with. [Note: in the above paragraph and those that follow, I'm assuming that extensions might add parameters to the URL other than just Thinking this all through reminded me of why profile extensions (as specified in #1195) weren't allowed to add query parameters in the first place, which was precisely to avoid these interactions. Looking at #1195 with fresh eyes now, I'm not sure whether it's actually impossible for profile extensions to have analogous interactions, or if it's just very hard, because of the various structural features combined with the normative proscriptions. Regardless, extensions with query parameters seem like at least a huge expansion of the amount of undefined behavior, and I'm not sure that's something we should take on lightly/without more thought. If we do take it on, I think it's a further argument for link templates, because, with a template, at least the server gets to control the interaction. For example, imagine some function from Extension A is in use, by virtue of a query param being in the URL. (This would be a query param that the server picked and specified in some link template in Extension A's body data.) Then, in the server-provided link template pointing to some of Extension B's functionality, the server can simply omit Extension A's parameter if it wouldn't play nicely with Extension B's. By contrast, if both extensions specify "to access [some functionality], add [some query parameter] to the URI", then a client whose on an Extension A URI will end up constructing a URI with both Extension A and Extension B applied when they try to follow the query parameter rule specified by Extension B, and that may not work. In that scenario, when the client lands on the page with Extension A's data, it has essentially navigated into a "dead end", where Extension B may have also added data to the response document, but invoking its functionality doesn't work, because (unbeknownst to it) the client would need to back out of the Extension A url first to avoid the problematic interaction. [Re the potential problems you brought up with link templates, I'll try to respond to those below.] Defining Query Parameter Values
Ah, gotcha. I didn't realize this at first. So, if I'm understanding right, you're talking exclusively about conformant extensions being able to deine the values for But that delineation seems weird to me... A conformant extension could need a whole host of query params to describe it's functionality (e.g, maybe an "actions" extension has an That said, I think the language around some parameters being "reserved by the spec" is actually confusing the issue. As I see it, there are two types of parameters:
Given these two classes above, I imagine you'd want to let conformant extensions define the meaning for any query parameters in the second class, to avoid the fragmentation I brought up above re an extension being able to define a meaning for
Hopefully my division of query parameters above shows that this isn't really true. While the spec has violated IETF best practices for the first class of parameters, it has not violated IETF best practices for the parameters that you're proposing extensions be able to define (i.e., those from the second class). This is simply to say that, even without re-litigating 1.0, we do still have a decision about whether to further violate these best practices—and now in an area with much more potential for conflicts, because we'll have a potentially unlimited number of extensions' specifications defining parameter semantics, rather than only having one specification (the base spec) doing so. Conflicting Extensions
I wanted to acknowledge this, because I think it is gets at a key question: how much does the potential for various types of conflicts actually matter? I agree that I'm probably more concerned about this at the moment than you are, and I'll try to spell out why below with more examples. That said, I think I'm at least somewhat persuadable that these conflicts matter less than I think. Additionally, there might be a solution that you would be happy with and that still makes conflicts impossible, in which case this could be a moot point.
I see how this could work in many cases, but not all. That is why I gave the example of an extension that is created after the API's been launched. In that situation, I don't think there's a good way that the server could have planned for wanting to use that extension, and I don't think it being simply unable to use the new extension is a very good outcome. Another example: the API server starts using Extension A, only to find out later (perhaps because new project requirements are added) that Extension A doesn't meet its needs. It then wants to transition to Extension B. The obvious way to do this would be to support A and B simultaneously, marking A as deprecated and then turning it off altogether once most/all clients are migrated. However, the server won't be able to run A and B side-by-side if they have conflicting query parmeters or document keys. To me, this seems like a very real-world possibility, and one that aliasing and link templates solve nicely; because the server can run Extension B at a different URI or put its data at a different place in the document, the problem goes away. Without this flexibility, I can only think of pretty heavy workarounds—like spinning up a separate instance of the API on a new subdomain and having that instance run Extension B. (Of course, that could have other costs.) Given the above examples, I think a good question to ask is: if we can remove the risk of extensions conflicting, why wouldn't we? I'd genuinely like to hear your concerns spelled out in more detail. Maybe they're are not what I'm imagining them to be, or maybe there's some way to address them that still avoids the conflict problem. Link TemplatesOne concern you brought up is that link templates may be insufficient, and I think there are two possible questions about how they might be "insufficient". The first is: With link templates, can we guarantee that the client can pass to the server all the info it needs to pass in order to invoke an extension's functionality, at a URL where the server can receive it? The answer to this question is clearly yes. Consider a link template like: {
"some-extension": {
"link-template-to-functionality": "http://example.com/?arbitraryParam={data}"
}
}Here, Now, the second question is: can link templates produce pretty enough URLs? (Obviously, a long blob of URL-encoded JSON looks ugly, even though it works.) Here, I think the answer is also yes. Let's use the hypothetical extension you described:
Here's how I imagine that would look: GET /articles
{
"meta": {
"filtering": {
"filterable-fields": ["fieldOne", "fieldTwo"],
"do-filter": "/articles{?filtersByField}",
// alternatively, "do-filter" might look like:
// "do-filter": "/articles{?filtersByFieldProcessed*}"
}
}
}Above, Then, the The first link template assumes the client will expand the template with one variable, filtersByField = { "fieldOne": "JSONforFilter1", "field2": "JSONforFilter2" }That is, the extension's spec would tell the client to use the fields it is filtering on (presumably pulled from Here, the field names and the filter values alternate, separated by commas. (Commas in the JSON are automatically escaped by the template processor.) Honestly, this URL probably looks good enough and is eminently usable. However, if, for some reason, it's really important that the filtersByFieldProcessed = {
"filter[fieldOne]": "JSONforFilter1",
"filter[field2]": "JSONforFilter2"
}Then, run through the second example template (the template in the comment), this data would give us our "ideal" url: Now, you might think: "Doesn't having the extension mandate that the field names be wrapped in Consider our earlier case where a server was using Extension A and wanted to run it alongside Extension B, which it would gradually transition to. Let's assume that extensions A and B are both filtering extensions, and both use this "hack" of putting pre-processed key names into a GET /articles
{
"meta": {
"filtering-a": {
"do-filter": "/articles?{filtersByField*}"
},
"filtering-b": {
"do-filter": "/articles?{filtersByField*}&isB=true"
}
}
}This JSON's pretty simple: the server is simultaneously providing links that old clients can use to apply Extension A, and new clients can use to apply Extension B. The trick is that, when clients apply Extension B, the template tells them to add an [Note: we could come up with some other way to communicate which extension is in use, but those other methods all seem more awkward and less appropriate. For instance, we'd probably end up with a custom header (because Finally, using link templates, the server can even support the (somewhat extreme) case of letting both Extension A and Extension B be applied to the same request. To see this, imagine the client is at a URL whose results are filtered by Extension A. At that URL, the server can provide JSON like: GET /articles?filter[something]=valFromExtA
{
"meta": {
"filtering-b": {
"filter": '/articles?filter[something]=valFromExtA{&filtersByField}'
}
}
}The trick here is that, in its template for applying Extension B, the server has passed back through the parameters applied by (the template for) Extension A. That way, the resulting URL from the new template has both extensions applied. Concretely, the resulting URL is: No doubt that URL looks ugly but, again, this kind of thing (applying two extensions that would otherwise use the same params) is absolutely impossible without templates, and this is about as ugly as it ever looks with templates. This example also goes to what I was saying earlier, about both undefined interactions—when A and B are both applied, should the results be Extension Negotiation
Sorry, I should have been a bit more explicit: when I said that the hypothetical The underlying point I was trying to make, though, is that an extension can have "must ignore" semantics or "must understand" semantics. In the former case, recipients ignore an extension they don't understand; in the latter case, the recipient has to fail the whole request if it doesn't understand an extension. Right now, we've associated "must ignore" semantics with profile extensions and seem to want "must understand" semantics on at least some additive extensions. My convolutedly-framed point about bulk posting etc. was that these extensions don't work well as "must ignore" extensions. That was all a way of saying that "must ignore"/conformant extensions should be negotiated differently than additive/"must understand" extensions, which it sounds like we agree on now. And, if that's the case, it was my way of arguing that we should feel free to introduce the profile parameter for negotiating conformant extensions without worrying that that will lead to an unduly fragmented design if/when we add additive extensions. I think we agree on all this? Misc/Other
I honestly don't know whether optional features or a full-blown additive extension system is a better option, but I'm not sure it's something we benefit from figuring out now. Even if we decide to go down the optional features route for now, I don't think that would foreclose on us later adding additive extensions, and I don't think either route really effects profile extensions.
I love the idea of trying to incrementally add profile-related stuff. I'm not sure about the particular incremental option you proposed, though, only because: if we add a Also, one more thing about query parameters: if we do let conformant extensions specify query parameter usage, I don't think an extension that did so could correctly be called an RFC 6906 profile. (Or, at least the query-parameter-specifying part wouldn't be a profile.) My reasoning for that, basically, is that profiles refer to the content of a representation. That's why the profile parameter modifies media types. Mandating things about query params is far outside the idea of the semantics of a representation, and, given that the author of that RFC has worked a lot on web linking and seems deeply invested in the IETF best practices I described earlier, I doubt he would see describing query params as consistent with that RFC (though I could be wrong). I'm not sure if this matters, but I did want to flag it. Finally, sorry again about the length. Cheers! |
|
I suppose we disagree on a few matters but hopefully can work through them.
A server must only support compatible profiles. If two profiles have conflicting claims on the same query parameters, meta members, attributes, etc., then a server should not support them.
There may be edge cases where the interaction is not obvious, but this is not a fundamental problem for the spec to solve. Again, I believe it's the responsibility of the servers to support compatible profiles and interpret them as well as possible.
Another way to look at this is that anything less than allowing profiles to define all implementation-specific areas of the spec will limit the utility of profiles. If we allow profiles to be negotiated, which on reflection seems desirable and is inline with RFC6906, then it seems important that profiles can signal a required usage of any area of the spec left to implementations.
My strong preference is to fully allow but not require hypermedia concerns throughout the spec. I’ve talked with @wycats about this many times - it has been a constant theme since the beginning. I don’t see that it’s pragmatic or fitting with the spirit of the spec to require the compilation of link templates in order to apply filters to a collection. To require this additional level of indirection has performance and complexity costs that are not appropriate for all clients. With that said, I absolutely support the ability to use link templates to the full extent possible. However, the spec does not even require that
I’m leaning towards keeping the base spec smaller and allowing for separate additive extensions that can be negotiated. In addition, I think it should be possible to negotiate profiles as described in 6906: “Media types defining a 'profile' parameter SHOULD define it as a whitespace-separated list of profile URIs.” As for query parameters: I maintain that the original sin here, if any, is in the base spec for covering their usage. However, a document structure specific solution, such as a top-level
@ethanresnick I'd be glad to meet with you, @wycats, and @steveklabnik to hash out any differences that remain. I'm afraid we're in danger of going around in circles otherwise. Also, I would of course be glad to hear from others in the community. |
|
An end-user's perspective; we're using JSON API via Ember Data, and for our API (which isn't consumed by Javascript clients). I understand that as a spec author there are other considerations, and that a single end-user's use-case shouldn't dictate anything. But in case it's helpful here are some thoughts. There are different ideas around handling of relationships that would be useful for us. Some of them are several years old and I'm worried that JSON-API adoption is impeded by this. For example, the most commented Ember Data issue is waiting for a solution to relations in JSON API. From our perspective a flawed solution today is better than a perfect solution several years from now. The work that has gone into JSON API is substantial, and it's a very well structured standard! I deeply admire your and wycats work on this and other things in the JS community. But there is a tradeoff between progress and rigour. Most widely used standards have flaws, that could have been avoided with more diligence. But would they have been successful if they'd been introduced a year later? So if answering yes to the question:
would speed things up regarding relationships-handling in the spec, then I'm all for it! 😄 |
|
@sandstrom I hear you and largely agree with your sense of urgency. Since my last post, I've been chatting with the other editors privately in an effort to build consensus. And despite the apparent lack of movement on this issue, I believe we've been making good progress! I would say that we're getting very close to agreeing on the fundamentals of both profiles and additive extensions. I'll post an update here soon so that we can continue to involve the community in this discussion. |
|
@dgeb and @ethanresnick, thank you for all your hard work and excellent discussion! Each comment has taught me a little something or made me see things slightly differently :) I'm sorry to post the classic "any updates on this?" comment... but any update on this? 😊 @dgeb I've seen your EmberConf presentation (excellent job BTW) and it seems like there is perhaps consensus being built around this and something official might be imminent? I'm selfishly desperate for something around profile/extension negotiation. We're working very hard to provide JSON API in Drupal by default (within the year if everything goes to plan! 🎊). As I'm sure you can understand, we're very concerned with backwards compatibility and if we put something in as "stable" to later find that profile negotiation works differently than we expect, we'll have a quite difficult time updating all Drupal installations with JSON API enabled in a compatible way. Rather than just asking for your time. I'd love to offer any help that I can. Although I'm not sure how I could help, know that I would be at your service! Perhaps the most minimal/concrete thing that could be agreed upon is simply the media type header? Could a PR be proposed and accepted that reserves that However, just that small step would allow implementors (like us :P) to begin decoupling their server specific features from the base implementation of the spec and permit them to document that any feature defined by the profile extension is "experimental/unstable" until the details are worked out. |
|
@gabesullice Hi Gabe, I'm excited to hear that Drupal's working on implementing JSON:API! As far as updates go, Dan and I have talked extensively about profiles/conformant extensions one-on-one, as he mentioned earlier, but I think we reached a bit of a stalemate. Through those conversations, though, my proposal from #1195 evolved in a few ways. In the latest version:
I think these changes addressed some of Dan's minor concerns, but I know he had some issues (I think with the fundamental approach) that these changes didn't get at. I don't want to speak for him, though so, @dgeb, maybe you can summarize your outstanding objections? |
Oh no! 😱 @ethanresnick, what do you all think of agreeing to agree to just the profile negotiation strategy? I know @dgeb has already proposed a Perhaps this is the fundamental disagreement? If not, I think this would be a great first step that can be defined without actually agreeing to what an extension is permitted to actually extend. Edit: One could even make the parameter |
I don't think there's much disagreement around the profile negotiation mechanism. As I recall, Dan and I both support adding the More fundamentally, though, I'm not sure I understand what locking down just the negotiation strategy would do. You said:
Can you explain that a bit more? |
👍
Sure! First, by "base implementation of the spec," I was referring everything in the spec not left to the implementors discretion. For the most part, this means how we're using the Concretely, I'm thinking about how our "fancy filters" could become a recognized extension in some capacity (perhaps with the registry you've alluded to) and also how we'll need to maintain backwards-compatibility of that extension until Drupal's next major version (this could be years from now). The process of making it a "real" extension will most likely involve at least some revisions to what we have now. Already, if I'm reading your early update correctly, this might mean we'd have to put filter values under So... how would a negotiation strategy help there? My thinking is that with a strategy in place, we'd have a way to "cordon off" the places where our API is still in flux. We could turn off fancy filtering by default and only turn it back on when:
Since Drupal already has a concept of "experimental modules" which don't have the same BC promises as Drupal core itself, the first step would give us some leeway to make breaking changes. By requiring the media type parameter to be in the request, the second step would also allow us to clearly document to client-side consumers that "To enable filtering, you must pass the Then, when/if we do need to make breaking changes, we'll have some understanding in place on both sides of the request that it was experimental and they might need to make minor updates. Then, if we wanted to drop support for the legacy method in Drupal core, we could. And if the broader Drupal community wanted to step in and maintain compatibility, then they'd have something to use to differentiate between old and new clients so that they might transform the incoming request. Maybe this is all a little too clever, I'm not sure, but I'm just putting negotiation out there as as a small positive step forward that could be taken. If nothing else, perhaps this is a good in-the-wild example of something that would benefit from profiles that can inform the overall discussion :) Stepping back from negotiation and into the higher-level topic... I've been starting to discuss the idea of adding support for filtering mechanisms other than fancy filters with the other maintainers. This might mean a simpler key/value filter strategy or a tie-in to Apache SOLR/Elasticsearch, or even "stored queries" where an administrator can preconfigure a query and let the client just pass a few extra parameters. All of these would be entirely valid strategies and I'd like to be able to gracefully handle experimentation and alternatives there. The community is implementing alternatives already, but they're having to do so by digging deep into internal APIs. This kind of thing is really exciting, but it starts to get at your fear about "undefined behavioral interactions." I think it validates that a server may want to have support for multiple extensions that apply to the same query parameter and that the client should be able to indicate which extension it prefers for a particular request. For example, I may want to have an "editor's picks" section of the page with a category filter (this would be the "stored query") and another section of the page that relies on fancy filters exclusively. I'm not sure how I'd prefer to handle that yet, but it's a use-case to think about :) |
Ok, that makes perfect sense, and I it's definitely something I'd like for you to be able to do. @dgeb: what do you think? Can we standardize something minimal enough around negotiation to enable an opt-in? Btw, “Fancy filters" looks like a cool approach. I actually came up with something related for my JSON:API implementation that might interest you. It's reassuring to see that we converged on something so similar! |
|
@gabesullice @ethanresnick thank you for reopening this discussion. Now that I'm back from a couple trips, moving forward on JSON:API 1.1 is one of my highest priorities. And I would very much like to include the concept of profiles in this upcoming version. Thanks Gabe for providing a real-world use case to motivate us. My preference is to lean as much on RFC 6906 as possible. I propose that we follow these rules to negotiate profiles:
In addition, we should recognize that @gabesullice would this meet Drupal's needs? @ethanresnick how does this sound? Are you comfortable moving forward at this level? |
Awesome!
Those rules sound the same as what you've proposed in our prior backchannel threads, though, so I'm still not comfortable with them. But I think I have a plan for how to proceed. First, though, tell me if you agree with this diagnosis of the situation (some of which I'm just writing as a summary for others trying to follow this thread):
Is that all right? If so, is it fair to say that the remaining disagreement is mostly that I think the extra restrictions my proposal places on profiles are useful in various wayss, whereas you think they needlessly complicate the definition or usage of a profile, or still unduly limit what types of usage patterns can be defined in profile? If so, I'd propose we proceed by evaluating the proposed restrictions on profiles one by one, to decide which restrictions add enough value to keep, and which are merely baggage. Does that sound like a good plan of action? The only other area where I think we disagree is how the server should respond when the client requests an unsupported profile. (I'd like to see a 200 response with the |
@dgeb: Yes, these three things would meet the immediate need. However, @ethanresnick just said:
Unfortunately, these are incompatible statements. I think we'd need to resolve them now to nail down the negotiation mechanism. Perhaps the response code is severable, but at first glance, I don't think it would be. Now, thinking through 200 vs. 415: I still need to read 6906 in depth, but from the arguments presented here, a 200 response code would seem insufficient. If the client sends a request that contains information about how the server should process the request and the server ignores that information, then that will cause unexpected behavior and be inconsistent with HTTP semantics. For example, if I send a query to filter out all resources with some value and get a "success" response code, I should be able to assume that the filter was applied. Requiring that I inspect the media type parameter to know if the filter was applied seems odd. With a 415, I could always repeat the same request without the parameter if I can handle that scenario. Another example: if a hypothetical "side-posting" extension exists, I don't think I should get a 204 Created for the primary data alone but still need to inspect the media type parameter to know if the other resources were created. I'd expect the response code to be all or nothing. @ethanresnick, perhaps there's a way around this that you're thinking of? To my untrained eye the 200 approach seems only to work elegantly for extensions that apply to the body of GET requests (which makes sense, as this has been your primary focus in #1195), but doesn't seem to work as well for unsafe methods. |
|
Hi. I'm a Drupal core maintainer, and a colleague of @gabesullice, so I'm coming to this issue from the perspective of adding json:api to Drupal core. Thanks, everyone, for the great work on JSON:API itself and the thoughts here and elsewhere on how to best evolve it.
I agree with @ethanresnick about these being very different use cases. In the case of fancy-filters, if I'm understanding that correctly, all it does is define some semantics for the filter query parameter, and does not affect the semantics of either the request body or the response body at all. Therefore, it doesn't seem to me that changing the Content-Type (even with just a profile parameter) is correct, because clients send Content-Type to define the semantics of the request body and servers send Content-Type to define the semantics of the response body. Is there any existing web standard for how to specify the semantics of the URL's query parameters? |
|
@gabesullice pointed me to this comment from @dgeb:
I think I disagree with this. RFC 6906 makes no mention of URL query parameters, so I see no reason to assume that it covers their processing rules at all. Meanwhile, RFC 7231 says:
|
|
Hey @effulgentsia, nice to see you here! Your comment referenced an issue that was brought up earlier that I hadn't paid much attention to, but it now obviously has a lot of relevance to the whole discussion at its foundation. I (finally) gave 6906 a full read-through and I think found an operative paragraph in section 3.2:
One could argue that the scope of the However, using the |
I was trying to suggest some rules as per your request "Can we standardize something minimal enough around negotiation to enable an opt-in?". I thought you were requesting that we discuss negotiation separately from the scope of profiles. Based on @gabesullice's feedback, it sounds like the negotiation aspects of my proposal would meet the needs of the Drupal team.
I agree with your assertions about our points of agreement and disagreement. However, perhaps an even more fundamental point that's not listed is the notion of negotiation. I believe that clients should be able to form a request in such a way that a profile be followed or else the server can reject the request. This is the use case for including the I agree with @gabesullice's assessment:
I would go further and say that, if a server does not have the option to reject a request based on not understanding / implementing a particular profile, then we are not providing a real means of negotiation. |
|
@dgeb @ethanresnick, do you have thoughts about the I think both could technically work, but |
|
FYI: Per https://www.drupal.org/project/jsonapi/issues/2955020#comment-12550668, we (Drupal/the JSON API Drupal module) cannot keep waiting. On April 15, we will start moving forward again. Ideally, this issue will be fixed, and otherwise, we'll implement what seems to be the most likely outcome of this issue. |
|
@wimleers Thanks for the heads up. I'm hopeful that we can resolve this by then. |
@dgeb Ah, my mistake; I didn't realize your proposal was only for negotiation. So yes, let's try to work out the disagreements there first. There are really two distinct reasons why I’m hesitant about the mix of query parameters in profiles, Representations vs ResourcesThe first has to do with the resource vs. representation distinction at the heart of HTTP. I suspect people following this thread already know about that distinction, but a brief explanation just in case: In HTTP, a resource is some abstract concept (e.g., “today’s weather”), but the user only sees it in the form of concrete representations returned by the server (e.g., an HTML page). The resource is identified by a URI—perhaps The By having one url but distinct representations, everyone talking about the concept of “today’s weather” can link to the same place, and then the viewer can get the representation that suits their needs best. Of course, proactive content-negotiation has downsides and never caught on in a big way, but, when it comes to the question of how to use HTTP “correctly”, these ideas still come up because they’re baked deeply into the protocol. As @effulgentsia pointed out, With all this in mind, it should be clear that profiles (or whatever we call them) that add query parameters to the URL change the underlying resource. From HTTP’s perspective, then, returning 406 on a request for If the server doesn’t understand the filter query param format, that means that the URI (which must, by definition, be trying to identify some resource) refers to a resource that doesn’t exist on the server or that the server can’t find, so the correct response would be 404. (A 400 could also be appropriate by some readings, depending on how/if json:api defined the query parameter that the profile is using.) So, in short, profiles that change query parameters are doing so to request conceptually different content, which means they’re minting new resources, and the existence or not of a resource is totally distinct from the When it comes to RFC 6906, it’s clear that it defines how to communicate extra information about a given resource, in its current representation:
Now, RFC 6906 does leave open an interesting possibility:
I interpret that as saying that it would be fair game for a resource like
Ignorability of ProfilesSo, the resource–representation distinction is why 406 has bothered me for profiles that would change query params. But what about profiles that just add data to the document? For them, a 406 would be semantically valid, if the client only said it could
A document that is “json:api + some profile” is also just a json:api document and can be processed as such. And, in many cases, returning the base document without the extra profile info still allows the client to do something quite useful with it. It’s also never less useful than a 406. A good example might be an RSS reader that has a special UI for podcasts. A podcast feed is, conceptually, a profile of an RSS feed (though, from a technical POV, podcasts predate RFC 6906 and don’t use it, but let’s roll with the example). Moreover, there are various add-on features for podcast feeds (e.g., ways to specify chapter markers, licensing info, update frequency, etc) that some clients understand and some don’t. So, when the user enters a feed URL into the RSS reader, the reader can request it with a listing of all the profiles it supports in the
When you look at the other examples in RFC 6906 (e.g., hCard), and the cases that I’ve emphasized in side threads (e.g., most of these), they’re more similar to the podcast case than the The only cases where, I think, some 4xx response starts to seem appealing is for those “profiles” that do change the query params and mint new resources—precisely because (the representations of) two distinct resources aren’t supposed to interchangeable with one another. But again, I’d suggest 404/400 over 406. (Note: the above is only referring to responses to GET requests; @gabesullice brings up a whole other area when talking about whether the client should be able to require that the server support a given profile when processing client input for an unsafe request [as opposed to the server just ignoring the profile data]. My gut is that such a capability would come with complications, and would put such extensions into @dgeb's "breaking" category from the OP. It's also incompatible with the "ignorability" of profiles. For these reasons, that capability was excluded from #1195, but I admit I haven't thought about it too much.) Query Parameters in ProfilesI realize it probably seems like I’m contradicting my prior posts with what I’ve said above around query parameters. In particular, I said previously that I want to allow query parameters in profiles, while above talking about why they’re outside the scope of RFC 6906. Also, I talked previously about returning a 200 for unrecognized profiles — including those that add query parameters — but above talked about returning a 404. So let me try to unpack those bits:
My apologies for the length of this post. [Insert Pascal quote about no time to make it shorter.] I've been super busy but wanted to keep this convo moving. |
|
I think it's good to consider other RFCs, the 'correct' ways to use HTTP, etc. Well designed protocols are important! But I'm also afraid JSON API will lose adoption if it cannot move forward. For example, XML used to be the preferred wire-format for Ajax/XHR. But while XML was occupied with DTDs, schemas and validation JSON offered a simple protocol that stole the show. Another example is GraphQL, which uses POST instead of GET for most GET-style requests. This deviate from HTTP-principles, but is also a pragmatic solution that many people accept. I'm not trying to criticize your work, I know you've put a tremendous amount of work into this spec, and correctness is important! But there is a tradeoff between progress and rigour. |
|
@sandstrom Your point is well taken. Correctness is important. But so is progress. Hopefully we can achieve both. @ethanresnick Something we have to accept at this point is that the "correct" nature of profile negotiation is not a settled matter by the IETF. Consider this draft proposal that defines "two new HTTP headers that enable User Agents and hosts to indicate and negotiate the profile used to represent a specific resource". Now I'm not proposing that we adopt these proposed headers for profile negotiation, since this draft may not advance, but it's good for us to recognize the fluid nature of the discussion at the IETF. Maybe the application of profiles from 6906 is too contentious and the negotiation mechanism too volatile at this point. Perhaps we risk painting ourselves into a corner. Let's consider an alternative: we don't have to use profiles as defined in 6906 to meet the needs of allowing user defined extensions to json:api. We could instead revive the |
|
@ethanresnick, thanks for the very informative deep dive! (seriously) I think there was a small (irrelevant?) oversight in your post and/or in your reading of @dgeb's 3-point negotiation proposal: in your reply, you made frequent reference to For clarity in this issue, is this a distinction without a difference in your mind? Does it alter your perspective in any way? In order to make progress, we could of course say: "If a server can not apply a requested profile, then it MUST respond with an appropriate 4xx Client Error" |
415 is for when the server doesn't understand the client's Whether we want 415 at all comes back to this point:
But, certainly, we can't only have 415 because we can't use that in response to GET requests.
@dgeb I don't thing changing the parameter name effects the bulk of my comment (specifically, the resource vs. representation section). Boiled down to one sentence, that section is saying that, from HTTP's perspective, the resource
|
|
Here's my concrete proposal for negotiation, as the above is all super theoretical. I don't want to leave the impression that theoretical correctness needs to block progress here; I don't think it does. Covering the theory stuff just seemed like a prerequisite for explaining my concerns with @dgeb's proposal. GET RequestsRequest Response Request Response Unsafe (PATCH/POST/DELETE) requests with a bodyRequest Response Request Response Down the road, if we want clients to be able to require that servers support the extra processing (which I think could get quickly into “breaking” territory from @dgeb’s OP), we can easily extend the scope of extensions to allow the definition of extensions that require a 415 in these cases. |
|
@ethanresnick As for 415 vs. 406 - I agree that 415 is only appropriate for an unsupported I also agree that it's fine for a server to respond with a 404 to an unrecognized query parameter, such as However, the Drupal team is discussing specifying usage of a query parameter already reserved by the spec -
This argument about Accept being about representations, not resource existence, ignores that the base spec already clearly defines how some query params should be used to filter and locate resources. And negotiation is done solely based on |
Cool :)
This one's a little tricky to me, because it's easy to imagine an implementation that doesn't have any support for filtering. In that case (no filtering support), I would expect that server to return 404 for any Given the above, I'd support a general rule that says a server can return 400 over 404 when it knows about the existence of a query param but considers the value provided to be invalid. This rule could then cover the
As I said above above:
In other words, I don't think the base spec actually is negotiating resource existence based on |
Yes, I'm sure we both agree that json:api dictates protocol usage above and beyond resource representation. And in the spec, the only agreement to that contract is through the
I agree with you here as well. The spec simply can have no opinion on this matter. I think this is just a question of precedence. I'm saying that negotiation based on the My thesis is that it's completely consistent with the spec to extend the content type to include profile(s) and/or extensions, and to use those for first order negotiation. I concede that there will always be some who look at any attempt to dictate protocol usage from a media type as impure or improper. But it is consistent with the spec. And there are just so many pragmatic benefits we can bring to users of json:api by giving them a first-order mechanism for negotiating profiles/extensions. |
|
I'd greatly appreciate it if @wycats could weigh in here as the original author of the spec. We are circling around what seems to me to be an essential question about the nature of json:api, so his perspective would be invaluable. |
Maybe it's pedantic, but a plausible interpretation (and the only one consistent with HTTP) is that the agreement is just happening out of band, not through the Accept header itself. Let's put that aside for a second, though. I'm curious about this:
Can you expand on it a bit? As I've been thinking about it:
|
|
All really good conversation.
Just to add some color: We're actually already seeing a need for this on the horizon as I alluded to here. No one has yet responded to an earlier point that I tried to bring up (perhaps it's just a terrible idea 😂 ), but see this following line from 6906:
I'd like to propose that we consider the Request Response I see this as a minor improvement to avoid the conflicts that were worked around in this sentence:
I would very much not like to see this kind of "namespacing" of query param keys. |
|
I can easily believe that there's a reasonable desire to support the kind of granular extensions that @ethanresnick has in mind. That said, there's a much simpler, and still very common use-case that I think @dgeb is trying to target here that I think we should facilitate more quickly, as it has fewer constraints. Specifically: you would like to build a server that has some additional required semantics in the user-controlled portions of the spec (lists of attributes, lists of metadata, naming requirements, support for client-side IDs, for example), and you would like clients to indicate proactively that they know about these requirements and know how to support them. I think of "profiles" as referring to this kind of use-case: no extensions to the base spec in reserved spaces, no allowing behavior that the current spec does not allow. There's nothing stopping people from negotiating this out-of-band, but it seems like a relatively common use-case, and one that we shouldn't have any problem supporting overtly in the way @dgeb envisions. It doesn't stop us from supporting a more composable notion of extensions in the future, nor does it imply that such extensions would be incompatible with these profiles. I think we should get this out the door and continue to ponder the more general question for a future version of the spec. |
|
@gabesullice Just to clarify something: is the Drupal team's use case that one Drupal server/install should be able to support multiple filtering strategies concurrently? Or just that Drupal will ship with/allow multiple strategies, but then the server will have to pick one to use (and the point of the profile parameter or similar is to indicate to the client which the server picked)? I'm guessing it's the former, but want to confirm. |
|
I'd like to point out a practical consideration related to relying on any HTTP header, whether Some CDNs, like Akamai, do not cache a response that includes a This is just one practical example where it's valuable to stay true to HTTP protocol intent about resources vs. representations: a URI should tell you everything you need to know about what resource is wanted. Request headers can tell you about what representation is wanted, but they shouldn't be required for clarifying the resource that's wanted. A change in how to filter a collection is a change of resource, not representation, and should therefore be conveyed in the URL. Therefore, what would you all think of json:api defining a
|
Profiles are a means to describe additional semantics the JSON API media type, without altering the basic semantics. Closes #1207
Profiles are a means to describe additional semantics the JSON API media type, without altering the basic semantics. Closes #1207
Profiles are a means to describe additional semantics the JSON API media type, without altering the basic semantics. Closes #1207
Profiles are a means to describe additional semantics the JSON API media type, without altering the basic semantics. Profiles are identified by URI, which ideally should dereference to a document that describes the semantics of the profile. One or more profiles may be associated with the JSON API media type through the `profile` media type parameter. The application of profiles to a particular document can be specified by clients and servers via the `Content-Type` header. The application of one or more profiles to a response can be requested by a client via the `Accept` header. In order to require the application of one or more profiles, the profile(s) must be specified with the `profile` query parameter. Any structural elements introduced by a profile can be aliased. This allows for better adaptability and composition of profiles. A new profile descriptor object is introduced which allows for declarations of aliases in a particular document. Closes #1207
|
Thanks so much for everyone's feedback here. Please review #1268 so we can try to get a solution landed soon! |
Profiles are a means to describe additional semantics the JSON API media type, without altering the basic semantics. Profiles are identified by URI, which ideally should dereference to a document that describes the semantics of the profile. One or more profiles may be associated with the JSON API media type through the `profile` media type parameter. The application of profiles to a particular document can be specified by clients and servers via the `Content-Type` header. The application of one or more profiles to a response can be requested by a client via the `Accept` header. In order to require the application of one or more profiles, the profile(s) must be specified with the `profile` query parameter. Any structural elements introduced by a profile can be aliased. This allows for better adaptability and composition of profiles. A new profile descriptor object is introduced which allows for declarations of aliases in a particular document. Closes #1207
* Introduce Profiles to v1.1 Profiles are a means to describe additional semantics the JSON API media type, without altering the basic semantics. Profiles are identified by URI, which ideally should dereference to a document that describes the semantics of the profile. One or more profiles may be associated with the JSON API media type through the `profile` media type parameter. The application of profiles to a particular document can be specified by clients and servers via the `Content-Type` header. The application of one or more profiles to a response can be requested by a client via the `Accept` header. In order to require the application of one or more profiles, the profile(s) must be specified with the `profile` query parameter. Any structural elements introduced by a profile can be aliased. This allows for better adaptability and composition of profiles. A new profile descriptor object is introduced which allows for declarations of aliases in a particular document. Closes #1207 * profiles: tweak definitions a bit E.g., clarify that profiles can cover any implementation-specific query params, not just sort/filter/page * profiles: add should about extensibility * profiles: add note about how to serialize space-separated media type param values * profiles: make it a MUST to list the applied profiles in Content-Type * profiles: clarify role of profile in `Accept` per prior convos * profiles: clarify that `Accept` may be totally ignored This is key for servers that don’t want to add `Vary: Accept` for better caching * profiles: add example of using Accept to request profile * profiles: clarify that keywords must be valid member names Should be obvious, but worth saying * profiles: clarify evolution requriments The link to the w3c tag terminology about backwards and forwards compatibility is more precise than “evolve additively”, though it doesn’t change the intended meaning * profiles: add note about 415 handling from 1.0 servers From 1195 * Add top-level links.profile * Fix typo and trailing whitespace * Use the term `profile` in example (instead of `ext`) * Clarify that links can specify URIs, not just URLs * Replace all usages of jsonapi/profiles with links/profile Also add examples of profiles * Profiles: fix broken links * profiles: remove overloaded uses of term "alias" * profiles: editorial/terminology cleanup/tweaks * profiles: define what a profile is * profiles: add must-ignore rule to example timestamps profile Technically, this rule is required to make it legal to add even a new, optional member to the timestamps object. * profiles: create separate authoring section This commit just pulls the existing guidelines about authoring profiles into a dedicated section, without changing any of the substance. I’m pulling this out because there are a few requirements for authoring profiles that I think are missing, and it was getting a bit messy to have the rules for authoring profiles all mixed in with the rules for using them. * profiles: add new minimal extra authoring rules * profiles: editorial tweaks * profiles: define handling of unrecognized values * proflies: document extensibility concerns/requirements * profile: define implicit profile query param value * profiles: clarify shallow definition of elements * profiles: small cleanup * profiles: fix profile param value inference Whoops.
Apologies for the broadness of this issue, which overlaps with many other issues
(see #1195 for perhaps the most up to date history and list of issues). However,
I think it's vital that we agree on some fundamentals before moving forward to
accept means to describe and extend usage of the base specification.
When viewed from the perspective of conformance, there are effectively
three ways to consider expressing additional semantics for the base
specification:
conformant - Semantics are expressed within the areas of the spec
explicitly reserved for implementations.
additive - Semantics are assigned to structural or procedural
elements that are reserved but not specified.
breaking - Structural or procedural elements are assigned semantics that
conflict with the spec.
Each category is discussed below.
Breaking extensions
Although the concept of breaking extensions was considered in 2014-2015, there
has been significant pushback (see #614) against this concept. I believe there's
now fairly unanimous agreement from the editors and the community that creating
a new derivative media type would be preferable to allowing breaking extensions in
the spec.
Additive extensions
Additive extensions would cover allowing additional member(s) in the reserved
areas of the document structure. It's also possible for additive extensions to
dictate how clients and servers should interpret those additional members.
Let's look at a few examples of additive extensions that have been proposed:
Bulk posting - This extension would allow multiple resources to be created
in a single request. It would expect that top-level
databe passed in asan array, and that servers create all resources or none. Created resources
would be returned in an array.
Side posting - This extension is being explored in [WIP] Sideposting draft #1197, which
allows related resources to be created in the
includedarray.Additive / subtractive fields - This extension has been proposed in
Extra Fields #1176, which allows fields to be specified relative to an understanding
of a default set of fields per type.
Operations - I'll be proposing this extension in the next several days.
It will introduce a new
operationsarray that can be used to processmultiple actions, including fetching and mutating resources, serially and
transactionally.
We need to be extremely careful when discussing additive extensions. If
additive extensions are not curated by the editors of the spec itself, it is
inevitable that changes to the spec will break additive extensions.
A key requirement for additive extensions is that it would be technically
possible for them to eventually become part of the base spec, even if we choose
to keep them as extensions indefinitely. Therefore, additive extensions can
not express any demands on non-normative areas of the spec, such as
metavalues.
To avoid fragmentation in implementations of the spec, we should not introduce
extensions with largely overlapping concerns. We should ensure that any additive
extensions can work together.
The usage of additive extensions must be negotiated, probably through a media
type parameter. For an overview of how media type parameter negotiation might
occur, see this deprecated page on extensions.
Conformant mechanisms
Mechanisms to describe spec structure and usage that fully conform to the spec
are tightly constrained. These constraints free us (the editors and the
community) from concerns about managing different implementations, ensuring the
reasoning behind each is sound, and restricting overlap.
Even within the constraints of the spec, there's still a lot of flexibility
available to implementations. There can be benefits to codifying and sharing
the choices made by implementations.
Although we do not need to be concerned about how Organization A is
using the
pageparameter, it would be nice if Organization A could sharetheir methodologies with the broader community so that Organization B might
benefit. And it would be even nicer if tooling were developed that shared these
understandings so that Organization C could benefit without thinking deeply
about the implementation.
This aligns quite well with the concept of a "profile" as described in
RFC 6906:
@ethanresnick has explored these concepts these concepts in detail and has
proposed that these be called "profile extensions", as outlined in
#1195
Ethan's proposal allows profile extensions to be associated with structural
elements in the document. However, I'm also concerned about allowing for
specification of any unreserved areas of the spec, including structural
elements like attributes, relationships, and meta, but also query params like
filter,page, etc. It also seems in keeping with the spirit of a profile toallow the combination of additional semantics in a single profile - including
both usage and structural expectations.
For instance, let's take the
pageexample above. Imagine developers want toshare a common usage of offset / limit pagination that uses the
pageparameter (e.g.
/articles?page[offset]=0&page[limit]=10). And they want toallow servers to specify a
pagemember in the top-levelmetawith thetotalmember. It seems beneficial that these concepts could be wrappedtogether in a single "profile" or "profile extension", so that
implementations can agree on both structure and usage atomically.
Are there other places better suited for the expression of capabilities? For
instance, we have discussed the "home document" proposal, as described in
https://mnot.github.io/I-D/json-home/ Are there other options we have not yet
considered?
I'm opening this issue to discuss a number of matters:
Do we agree on the basic classifications above?
Should negotiation mechanisms differ for additive vs. conformant
extensions (i.e. "profiles")?
What are the appropriate boundaries for a profile extension? Should they
include structure + usage?
Are there other standards, such as home documents and schemas, that should
be considered?
Is consensus strong enough re: additive extensions that we should consider
focusing on the extension mechanism before anything profile-related?
I'd like to start as simply and conservatively as possible. But we also need to
think wholistically. We should establish a path by which every choice made by a
particular implementation of the spec can be described, standardized, and
shared.
This is a great time to express your questions, concerns, and opinions about
extending the specification.
The text was updated successfully, but these errors were encountered: