Support for multiple operations in a single request #795
Comments
Potential Solution: Operations ExtensionOne solution to this problem is to represent each operation as an entity in a request document. Operations could be transmitted in an array, since ordering is often significant. Similarly, responses to each operation could be returned in an array that parallels the operations in a request. This solution is very similar to the experimental JSON Patch extension with the exception that it is additive to the base spec. Operations could include the following members:
Operations would be sent in a top-level Responses could also be returned in a top-level Requests would be sent with the It's important to mention that a server would have complete control over which operations could be performed together at any endpoint. Solutions for use casesCreate two articles, add one existing tag to both, create a new tag and add to bothRequest: Response: Change the title of an article and update its tags: add one existing tag, create another, and remove another (without replacing the whole tag set).Request: Response: |
dgeb
changed the title
|
Leaving notes here for posterity, @dgeb and I discussed the possibility that we should consider supporting a We might also need to support a As an aside, this entire proposal feels like gross RPC, but there really doesn't seem to be a path forward to enable the complex use-cases we are trying to support without this type of extension. |
|
This will probably get messy on the client side when you have relationships multiple layers deep. Doing multiple requests keeps data in a flatter structure, and will keep things simpler, in my opinion. |
You can totally do that. We're discussing an optional/additive extension to the base specification that has been widely requested (see the issues referenced in the OP). If you don't want to use it, this issue doesn't apply to you. |
|
I agree with @tkellen. Yes, using multiple requests keeps things simpler, however sometimes it doesn't make sense to do multiple requests especially if they are to be treated as a single transaction. It would be very difficult to manage a block of separate requests as a single transaction, rollback changes and return error states for the objects. I am currently using included to handle these multi-operational actions which has its down sides and if I am rolling my own it means others are too. That is a good indication that we need a spec. |
|
I really like the proposed solution above. I am excited to see a standard arise out of this. regarding this:
I like the idea of building an app that returns just the status if all my operations were update/delete, and full if any of the operations was an add. So I would like to see a response flag, perhaps as metadata? and this:
Does this mean that the server will return something other than the |
|
Not sure how I feel about this from an implementer's perspective. It adds a lot of complexity just to save on HTTP requests. That said, I'm glad it's being discussed as an optional extension. |
That is one primary goal. Another is to provide a means by which multiple operations can be performed in a single transaction. And still another goal particular to the operations extension is to provide a format compatible with streams (e.g. web sockets). |
No, it means you might have a request like this: {
"operations": [{
"path": "/articles/1/relationships/tags",
"action": "remove",
"data": [
{ "id": "tag-1", "type": "tags" }
]
}, {
"path": "/articles/1/relationships/tags",
"action": "add",
"data": [
{ "id": "tag-3", "type": "tags" }
]
}, {
"path": "/articles/1",
"action": "fetch"
}]
}...with a response like: {
"operations": [{
"status": "204"
}, {
"status": "204"
}, {
"status": "200",
"data": {
"id": "1",
"type": "articles",
"attributes": {
"title": "the first article",
"content": "some content for an article",
},
"relationships": {
"tags": {
"data": [
{ "id": "tag-3", "type": "tags" },
]
}
}
}
}]
} |
|
oh cool, I could see the utility in that. Thanks for the example. |
That sounds like a fair diagnosis to me. That is, I agree that there are some complex cases that require transactional and imperative/RPC-ish semantics. Therefore, we'll need something reasonably like JSON PATCH, and @dgeb's proposal here seems cleaner than the current JSON PATCH extension. However, I wonder if we could cover 80% of the embedding/sideposting use cases with a more declarative, higher-level format and, if so, whether that would make sense. Sideposting ProposalOverviewThe higher-level other option I was thinking about is an extension, negotiated with the TBD extension mechanism, that would be something very similar to what I proposed with an
Use CasesHere's how it would work with the two use cases given above. Creating two articles and adding existing and new tags to each:Request: This is as I proposed in #536, except that I've taken advantage of the idea in the bulk extension to make Change the title of an article and update its tags: add one existing tag, create another, and remove another (without replacing the whole tag set).This use case would need to be handled by operations, if it's to be done in one request. Otherwise, it would take 3 requests (which might be ok). One request would update the article's title; one would create the new tag and add it with the existing one; and one would remove the other tag. The interesting request is the one that creates and adds the new tag simultaneously, which would look (as you'd expect) like this: Specification DetailsFirst, as I mentioned, this proposed extension would also include a way for the client to determine which embedded resources were assigned which ids (if server-side ids are in use). That would work like so: In the request: Then, in the response: The About relationship graphs: the extension would be required to support graphs in which the only links between the resources in the request document are relationships from the primary data to embedded resources. (That is, the embedded resources don't link to one another, and the primary data's resource objects, if there's more than one, don't link to one another. But embedded resources can link to other, pre-existing resources.) This could be extended to say that any links are valid so long as the resulting graph is acyclic, to support the recursively embedded resources that @tkellen asked about on the original embedded resources proposal, in addition to interlinking between primary data resources, etc. As far as supporting other graphs goes, the extension would have feature flags (again, mechanism tbd) indicating the types of graphs it supports. The structure of those flags would be defined over time as we collect more real-world use cases. A request might have a way to specify which type of graph it's sending, in order for the server to use more efficient processing. AnalysisPros, as I see them:
The biggest con I see to this higher-level syntax is that, because we'll probably still need an operations extension too, it's duplicative. However, if a good chunk of APIs could get away without implementing operations, and the high level syntax really has the advantages listed above (and doesn't have any unforeseen problems), then having it as an option might make sense anyway. After all, we shouldn't take the "it's duplicative" argument to its extreme, as that would suggest removing most of the base spec, since all one really needs is operations. Proposal 2: Inline OperationsI think that the ideal request for the second use case would look something like the below: This only uses one request, but it feels less RPC-ish than the straight up operations approach, and it builds on the side posting syntax proposed above. I think of this as the "inline operations" approach because it gets rid of "path" entirely in each operation and thereby tries to blend the operation ideas in with the existing JSON API spec. My hope is that an approach like this could make operations feel a bit more natural, but this is a very new idea, so I'm not sure if it'll actually work. I'm curious what y'all think! |
|
I like this suggestion a lot too. What happens if there are two levels of resources being created? e.g. "Creating two articles and adding new tag and tag category" That is a contrived example, and perhaps the spec should dissuade requests like that? |
|
The original draft of this post also included this use-case, but we didn't fill it out for time reasons:
GET /nodesThis actually starts empty, but the representation below matches this If someone wants to take a stab at representing that in both proposed solutions, that would be great. |
|
@shicholas If I understand your question correctly, that would look like this: POST /articles
{
"data": [{
"type": "articles",
"attributes": { "title": "New Article" },
"relationships": {
"tags": {
"data": [ { "pointer": "/embedded/0" } ]
}
}
}, {
// second article would be just like the first one
}],
"embedded": [{
"type": "tags",
"attributes": {
"name": "new tag!"
},
"relationships": {
"category": {
"data": { "pointer": "/embedded/1" }
}
}
}, {
"type": "category",
"attributes": { "name": "whatever" }
}]
}Per the earlier discussion, though, this example would only be supported if the extension signals it can create any acyclic graph or we make that a base requirement. |
|
@tkellen Re the other use cases:
That would be done very similarly to my above comment, with a single Transforming that structure, once created, though, would require two requests. The first would move PATCH /nodes
{
"data": [{
"type": "nodes",
"id": "nodeB",
"relationships": {
"parent": { "data": null }
}
}, {
"type": "nodes",
"id": "nodeC",
"relationships": {
"parent": { "data": null }
}
}]
}The second request would create the new With inline operations, this might be able to be consolidated into one request, like so: PATCH /nodes
{
"data": [ // same as above ],
"operations": [{
"op": "create",
"data": {
// new node resource object with parent null
}
}]
}However, the fact that the this complex a transformation takes only one or two requests feels like a lucky anomaly. It's enabled by For complex transformations like this to work in the general case, though, we'd probably have to specify that the updates in |
|
Thank you @ethanresnick for answering my question. FWIW, I like your proposal better than the operations approach because I feel it adequately addresses any POST use-case I intend on using with minimal changes to the base spec. I like how it keeps the |
Thanks Nick. I was imagining the response would be the same as the current POST response, except that the newly-created resources from |
|
Still no decision? |
|
it's a tough cookie |
|
+1 just following along! |
|
@travisghansen please use GitHub reactions to +1 or the subscribe button to get notifcations about updates to this issue. +1 comments usually don't bring additonal value to the conversation and cause unnecessary notifications to be sent out to multiple users. Also great to hear that you, too, want to get this long standing use case resolved later! |
|
Sorry if some of what I'm saying has already been discussed here, but I wonder if a couple of simple modifications might help in a number of cases. For example, allowing the I can see a different use case where you want to batch operations from a performance and latency perspective. To me this actually seems like a higher level sort of resource, where you simply post to a generic batch operations endpoint and let the server sort it out. From the client perspective, it might be a |
|
Fwiw, here's how I solved this within the constraints of JSON:API 1.... basically, I took the solution I proposed upthread (#795 (comment)) and jammed all the changes into the various
So, creating two articles with related tags looks like this: POST /articles
{
"data": {
"type": "articles",
"attributes": { "title": "First Article" },
"meta": {
"relationships": {
"tags": {
/* relate it to the tag "new tag!" on creation */
"data": [
{ "pointer": "/meta/included/0" }
]
}
}
}
},
"meta": {
"included": [{
"type": "tags",
"attributes": { "name": "new tag!" },
"meta": {
"relationships": {
/* relate this tag to the "whatever" tag below */
"category": {
"data": { "pointer": "/meta/included/1" }
}
}
}
}, {
"type": "category",
"attributes": { "name": "whatever" }
}, {
/* second article's resource object goes here.
it can have relationships to other resources in its payload in the same way */
}]
}
}Servers still have to decide what linkage graphs to allow (see the 'Analysis' section of the comment I linked above), but that's not too hard and it's sorta inevitable—unless you go for an embedding approach, which trades off reusing a sideposted resource in multiple relationships for a guarantee that the graph is a tree. Implementing such an approach with Finally, fwiw, if an extensions approach like profile extensions is implemented (#957), it could easily be used to specify that {
// map the customly-named keys to extensions with a standardized definition!
// Note that both names must map to the same extension, as splitting this operation up
// over multiple extensions would violate some of the constraints on profile extensions
"aliases": {
"included": "http://example.com/some-url-for-extension-specifying-sideposting",
"relationships": "http://example.com/some-url-for-extension-specifying-sideposting"
},
// rest of the document is identical to what's shown in the no-extensions example
} |
|
How do you handle errors? Is there a transactional boundary across all the things you are creating? |
|
@brainwipe Thats a really good question. Currently, how to handle errors produced by values in Of course, some servers might not support all the relevant extensions, and those severs would simply ignore the data provided by the extensions they don't understand. This would result in the request succeeding, but with only some of the resources/relationships being created. Since that is probably unacceptable, a client would either have to know in advance whether the server it's working with supports the relevant extensions (in today's world, most clients are hard coded to interact with a single sever, so this knowledge would be easy to build in) or the client could use content negotiation to ask the server in advance whether it supports the relevant extensions. |
|
in crnk.io and ngrx-json-api we now implemented the multiple operation support with jsonpatch. so far that works very well. The implementation was quite simple. There was not a need for a new standard. Error handling works well as each request is able to provide its own response (included status code). It can also handle more complex object structures. The objects in question do not necessarily need to be related. The one thing still missing and what would need some specification is how to deal with relationships and server-created IDs when doing POSTs (but usually we do not make use of that in the first place). So from my point of view while it seems easier to attach object somewhere within the relationships or meta data. The jsonpatch/operation approach stays closer to the semantic of jsonapi, making it easier to use and implement with all the feature set of json api (IMHO...). (not to forget that the approach would alsl allow the invocation of non-jsonapi services) |
|
Just so that I understand...
So we're saying that the single POST/PUT/DELETE is a transaction? One fails, they all fail? As for server support, I completely agree that most clients are coded against a single server. For those that rely on discovery, you usually have a "discovery document" on the root of the API (in json.api, that would be a document with meta info only) and that document would be perfect to put the extensions in. As an aside, don't dwell on this bit below... From a domain design point of view, you're giving the client the ability to understand more about the inner workings of the domain and by doing so you're hard coupling the client to the domain. Once you hard couple the domain (not the API, the domain objects themselves) to the client then it becomes increasingly more difficult to change the API. There has to be some coupling but until now json:api has been about resources and their links between them in a generic fashion, but adding in composites you're now saying that the client needs to understand what can be built with what and in what order. I'm just adding this in here for posterity because I can see some domains being leaked through the multi-operations and the API builder should be aware of that. |
|
Just chiming in here, in the spirit of mapping json:api to HTTP semantics, I believe the transaction boundary will always need to be at the request level. Put another way: if one POST/PUT/DELETE action fails, they should all fail. (There is no standard "Partial Success" status code.) I have worked with companies and clients who don't like the idea of maintaining the transaction boundary on the client side, i.e. having the client issue individual requests and then handling rollback in case one fails. In situations like that, we usually go with jsonpatch in the |
Yeah, what if a rollback fails too? |
|
There are many failure scenarios, that being one of them. Another: What if the client loses a path to the server right before it issues a rollback command? |
|
I'm also curious about @brainwipe's question:
I came here because I wanted to be able to update a whole list of I feel like there are probably other use cases here aside from optimization though. Maybe all of them are more for convenience/simplicity than necessity - but those are important too. (The question is, what are they? And should they be addressed by JSON:API?) |
|
@mlts I think you've identified the central question:
Here is the sample application from my jsonapi_suite tutorial: This is a fairly simple application but illustrates a variety of use cases. The form on the right illustrates the need for sideposting. We want to submit the Employee, Positions, and Department all in one request - this way things run in a transaction, and if the transaction fails validation errors are returned. (one final requirement not shown is the ability to disassociate an entity versus destroying it - think removing tags from the employee...we'd want to remove the tag but not destroy it) The sideposting alternatives I've heard either A) have potential to leave orphaned records in the DB B) require the programmer to create a custom aggregate. I dislike both these options because sideposting can be encapsulated by libraries (in my case jsorm). This means developer-facing code is simple and out-of-the-box: // save employee, create a position and associate to existing department
Department.find(1).then({data}) => {
let position = new Position({ title: 'Safety Inspector', department: data });
employee.save({ with: { positions: 'department' } });
});See here for the full app. These common use cases are what I'm looking for - everything else is just gravy. FWIW, for your use case I have a dummy |
|
This is a solution we are using for this: https://github.com/e0ipso/subrequests (nodejs) and https://www.drupal.org/project/subrequests (Drupal). This is an article explaining how it works: https://www.lullabot.com/articles/incredible-decoupled-performance-with-subrequests I hope it helps. |
|
So @richmolj - to compare your use case there against @brainwipe's argument that this allows/encourages domain knowledge to leak into the client: I think it's clear that some domain knowledge-rules are managed in the client here, but only ones that the client cares about, right? In this kind of scenario, the client is not simply a UI for the API, it's an application built on top of a given data source, provided by the API. So while the API has its own domain model and rules, the client application itself may also have some of its own domain modeling and rules that it would like to enforce (in this case, that employees aren't created without positions attached - so the app can always assume that an employee has positions, and therefore not provide an interface for the case of an employee that doesn't have a position). In this context, the API is playing the role of a data source and it's true that a data-source doesn't have to provide features like transactions or multiple simultaneous updates/actions. Lack of those features can be worked around by an application (an interface, or logic can be implemented to handle cases of orphaned records). But they are nice features for a data source to have, which make the [client] application logic and interface simpler and more straight-forward. Whether the scope/intent of JSON:API has room for these features is still debatable, but from a practical and conceptual perspective I think this shows that they would be desirable. |
|
There has been so much evidence (in this thread and others) that sideposting a good idea and a needed feature. @dgeb and I have agreed that it will be coming to JSON:API with 1.1, independent of the broader operations proposal. The only blocker now is some details about how it should work. Those are being discussed in #1197. That should be the canonical issue for now if you want to contribute to the discussion around sideposting. We'll also have an issue, I imagine, for the discussion around operations when @dgeb gets out a new draft of that. For now, though, I'm going to lock this issue. |
and others
It has been widely requested that JSON-API support creating/editing/updating/deleting multiple records in a single request (#202, #205, #753, #536). We have attempted to paint this bikeshed repeatedly over the years. Providing an "official" solution, or solutions, to this problem is one of the primary goals for v1.1 of this specification.
There are several requirements for a solution:
Here are a couple use-cases that test these requirements:
We would like the community to propose ideas for how to solve these use cases (or chime in to support solutions provided in the responses to follow).
Assume the following data is present
The text was updated successfully, but these errors were encountered: