New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Operations to v1.2 #1254

Open
wants to merge 2 commits into
base: gh-pages
from

Conversation

Projects
None yet
@dgeb
Copy link
Member

dgeb commented Feb 9, 2018

We've long discussed the problem of needing to support multiple operations in a
single request (#795 attempted to synthesize requirements from a number of
issues).

I had originally planned to propose "jsonapi-operations" as either an extension
to the base spec that requires negotiation, or as a separate media type
entirely. However, this PR instead introduces the concept directly into the base
spec. The rationale for my change in direction is that, in my experience, the
lack of these facilities in the base spec makes it unsuitable for a wide array
of fairly common scenarios. Some of these scenarios include:

  • Creating multiple resources of the same or different types together and then
    creating relationships between them. For example, creating an invoice
    together with one or more invoice items.

  • Editing memberships of very large relationships, assuming that edits include
    additions and deletions. For example, removing two members of an organization
    at the same time two other members are added.

  • Clearing relationships and explicitly deleting the related resources as well.
    For example, deleting an author and some of the articles associated with that
    author.

  • Sending a general "batch" of changes that must succeed or fail together. For
    example, queueing up an array of operations while offline and then applying
    them together.

I believe this proposal is a complete and thorough solution to these problems
and more.

I really appreciate the work done by others that takes a more incremental
approach to tackling these problems. For instance, @beauby's sideposting draft
(#1197) is a great example of this. There is a nice symmetry with compound
documents that makes it feel like a natural fit for the spec. My primary
concerns with this proposal are:

  • It doesn't allow a request to specify order of changes, which can make
    implementation on the server quite difficult to generalize. There can be
    cross-dependencies and dependencies several layers deep which the server
    must be prepared to unwind and create an execution plan.

  • Discussion regarding extension to accommodate updating and deletion of
    resources and relationships has led to very operation-like proposals such as
    method being added to individual resources.

Given these concerns, I thought it would be worthwhile to consider operations
as the single answer in the base spec.

Note: This PR builds off the Local IDs PR, which are key for linking resources which have not yet been assigned permanent IDs.


Last but not least, I want to say thank you to @ebryn and his company Prototypal for funding much of the time I spent working on the operations proposal. Almost all the time I spend on JSON:API is my own, and it has taken quite a bit over the years, so Erik's assistance was a big boost to helping me push this forward.

With that said, we are all trying to do what's right to move the spec forward, and I'm not trying to make unilateral decisions. I do want to see operations through in one form or another (as part of the base spec or separately), and this is my first attempt. I definitely want to hear what others are thinking.

@gabesullice
Copy link
Contributor

gabesullice left a comment

Very interesting proposal @dgeb!

I've completely read through it once this morning. So I haven't really sat down and thought through it completely yet.

My first reaction is that the "identities" key could use a little more elaboration. I'm least clear about its intended purpose.

For example, in the final example, both an "lid" and "id" are included in the newly created "authors" resource, which I think would serve the same purpose as an identity. I'm probably missing the use case that you're trying to support here and a little handholding in the spec might help elucidate that case.

@courajs
Copy link

courajs left a comment

I love it! I have a few comments, but overall I think this would be a huge win to include in the base spec (though keeping it optional - "a server MAY support usage..." is good).

Every [resource object][resource objects] **MUST** contain an `id` member and a `type` member.
The values of the `id` and `type` members **MUST** be strings.
Every [resource object][resource objects] **MUST** contain an `id` member and a
`type` member. As noted above, the only exception is that the `id` member is not

This comment has been minimized.

@courajs

courajs Feb 10, 2018

I'm not sure this is noted above - I think this is the first mention of it

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

The exception is spelled out above as follows:

A resource object MUST contain at least the following top-level members:

  • id
  • type

Exception: The id member is not required when the resource object originates at
the client and represents a new resource to be created on the server.

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

BTW, I'm open to rewording this in both the 1.0 and 1.1 specs so that the exception clause is integrated in the requirement instead of standing on its own.

Every [resource object][resource objects] **MUST** contain an `id` member and a `type` member.
The values of the `id` and `type` members **MUST** be strings.
Every [resource object][resource objects] **MUST** contain an `id` member and a
`type` member. As noted above, the only exception is that the `id` member is not

This comment has been minimized.

@courajs

courajs Feb 10, 2018

I think this is confusing phrasing. It means we've got a sentence that's false on its own, and needs to be taken together with another sentence. Can we find something that doesn't have to contradict itself later? Maybe something like:

Every resource object MUST contain a type member. Every resource object MUST contain an id member, except when the resource object originates at the client and represents a new resource.

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

It means we've got a sentence that's false on its own, and needs to be taken together with another sentence. Can we find something that doesn't have to contradict itself later?

I agree this is a best practice and will reword. Thanks for the suggestion!

@@ -361,9 +381,19 @@ A "resource identifier object" is an object that identifies an individual
resource.

A "resource identifier object" **MUST** contain `type` and `id` members.
The only exception is that the `id` member is not required when the resource
identifier object contains a `lid` that identifies it as matching a new resource
to be created on the server.

This comment has been minimized.

@courajs

courajs Feb 10, 2018

Similar to the comment on the exception above - can we make every sentence true on its own? Maybe:

A "resource identifier object" MUST contain a type member. A resource identifier object must also contain an id member, except when it contains a lid that identifies it as matching a new resource to be created on the server.

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

Yes, I think that is an improvement.

[resource identifier objects][resource identifier object].

The purpose of the `identities` array is to correlate `lid`s in a request with
`id`s in a response.

This comment has been minimized.

@courajs

courajs Feb 10, 2018

Like @gabesullice, I'm a little fuzzy on why identities is strictly necessary - I think you could construct the same information by crawling the ops list. But, maybe its purpose is just to simplify that task? That would be valid, but it might be nice to have phrasing to that effect. Maybe:

The purpose of the identities array is to make it simple to correlate lids in a request with ids in a response.

requests. The values `"get"`, `"add"`, `"replace"`, and `"remove"` align with
the HTTP methods `GET`, `POST`, `PATCH`, and `DELETE` used for singular
requests. The specific `op` codes are inspired by [RFC
6902](https://tools.ietf.org/html/rfc6902).

This comment has been minimized.

@courajs

courajs Feb 10, 2018

I don't love the choice of replace here. In the referenced JSON-patch spec, replace is only ever applied to a single field, replacing it entirely. In this spec, it's applied to a document, but has the semantics of an update - keys not mentioned in the replace op are not modified. replace seems to imply the whole record would be replaced with only the new values.
Could we change to patch or update? I think otherwise it will cause confusion for people learning the spec.

This comment has been minimized.

@gabesullice

gabesullice Feb 10, 2018

Contributor

I'll second that sentiment. Until @courajs just pointed out the update semantics, I fell into the trap thinking "replace" was like a PUT.

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

I agree that replace could cause confusion and will probably change it to update. I'm trying to avoid usage of HTTP methods, since operations exist at a different layer, so I'm reluctant to use patch.

Without perfect alignment, I should probably do away with the reference to RFC 6902 as well.

An operation that creates a resource **MUST** target a resource collection
either through the request endpoint or the operation's `ref` object. The
operation **MUST** include an `op` code of `"add"`.

This comment has been minimized.

@courajs

courajs Feb 10, 2018

Here, it says the operation must target either through the endpoint or the ref object, but the example does neither. Instead, it targets via properties of the data member.

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

@courajs thanks - good catch!

An operation that updates a resource **MUST** target that resource
either through the request endpoint or the operation's `ref` object. The
operation **MUST** include an `op` code of `"replace"`.

This comment has been minimized.

@courajs

courajs Feb 10, 2018

Same as above comment - this targets via properties of data, but it's stated that ops must use ref.

This comment has been minimized.

@dgeb

dgeb Feb 10, 2018

Author Member

@courajs again - good catch!

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 10, 2018

@gabesullice @courajs thanks for reviewing!

My first reaction is that the "identities" key could use a little more elaboration. I'm least clear about its intended purpose.

The identities key was introduced in #1244 as a means to ensure that every lid that is passed in a request can be aligned with an id in the response, regardless of the shape of that response. This concept was born out of thinking through some exceptions in #1197 in which a response might take a shape that doesn't include all the created resources. However, as you point out, it's not strictly necessary with operations. I'm going to remove the concept from both #1244 and this PR to keep them both as lean as possible. Thanks for questioning this.

Introduce Local IDs to v1.1
A client may include a "local ID" as a `lid` member in a resource object to
uniquely identify the resource within the request document. Every representation
of that resource, whether as a resource object or resource identifier object,
must then include the matching `lid` member and value.

When a server receives a request document that contains resources with local
IDs, the server must include the matching `lid` member and value in every
representation of that resource or resource identifier in the response document.

This addition paves the way for requests of all kinds that may need to establish
linkage between resources that have not yet been assigned a server-generated ID.

@dgeb dgeb force-pushed the operations branch from f85de2d to 6a7ae6f Feb 10, 2018

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 10, 2018

Ok, the identities array has been removed from both here and #1244, and the other suggestions / corrections have been incorporated.

One aspect that perhaps should be fleshed out further with examples is the usage of params in operations. I am reluctant to provide completely redundant examples with the query param examples, but I think some examples could be clarifying.

@dgeb dgeb changed the title Introduce operations to the base 1.1 spec Introduce Operations to v1.1 Feb 10, 2018

Introduce Operations to v1.1
Introduces the concept of operations to the base 1.1 spec.

Operations can be used to perform one or more fetch or mutation operations serially and transactionally.

@dgeb dgeb force-pushed the operations branch from 6a7ae6f to 999e6df Feb 10, 2018

@jaredcnance jaredcnance referenced this pull request Feb 12, 2018

Closed

Operations Support #150

0 of 4 tasks complete
@remmeier

This comment has been minimized.

Copy link
Contributor

remmeier commented Feb 12, 2018

has http://jsonpatch.com/ also been considered as a basis? Everything here looks great and jsonpatch.com would give just a slight different envelope. An additional benefit of this would be the support of other, non-JSONAPI calls.

one minor other thing, "op" could potentially retain a more HTTP-style naming: GET, POST, PATCH, DELETE.

### <a href="#processing-operations" id="processing-operations" class="headerlink"></a> Processing Operations
A server **MUST** perform operations in the order they appear in the

This comment has been minimized.

@gabesullice

gabesullice Feb 12, 2018

Contributor

I keep going back and forth on this. On the one hand, a server can't possibly know the dependencies of every operation. On the other, it seems like it exposes too much of the server implementation to the client.

Take the example of a client which wants to replace a resource. If the operations happen out of order, then the server will generate a collision error (when the add happens before the remove).

If the order of operations given by the client must be followed, then it's the client's responsibility to order these correctly. That makes sense because it would be far too difficult for a server to infer the correct order in every possible scenario.

However, take the example of creating a resource which has a relationship to another resource. There are 3 possible orderings that could be valid or invalid dependent on the storage implementation of the server. (1) If the server stores relationships separately from resources, then both resources need to be saved before the relationship is created. (2) If the server stores relationships on the referencing resource, then the the referenced resource needs to be created first. (3) Finally, if the relationship is stored on the referenced resource, then the opposite is true and the referencing resource must be saved first.

It seems incorrect for this to be the client's responsibility to handle. Generic clients will need to switch behavior based on something not defined in the spec.

After writing that out, I wonder if the way to handle this is with an opt-in dependency tree. That is, allow operations to be identified and also allow operations to specify operations on which they depend. When operations have no dependency or when multiple operations depend on the same operation, the server may perform those operations in an arbitrary order. That would allow a server to make a best-effort at ordering the operations appropriately (e.g. it could order operations according to relationships) but it would also allow the client to express order when it would otherwise be too ambiguous for the server to determine.

This comment has been minimized.

@gabesullice

gabesullice Feb 12, 2018

Contributor

Here's an example of the replace example from above with dependencies on other operations.

PATCH /bulk
{
  "operations": [{
    "op": "add",
    "deps": ["remove_some_resource"],
    "data": {"not": "shown"}},
  {
    "op": "remove",
    "id": "remove_some_resource",
    "ref": {"not": "shown"}}
  ]
}
@richmolj

This comment has been minimized.

Copy link
Contributor

richmolj commented Feb 12, 2018

It's great to finally see this area get momentum @dgeb ❤️ ! I strongly agree with your bullet-points making the case that this should be part of the base spec - the scenarios are too common to overlook. I also think this proposal is more cohesive than what I've seen in the past; very much appreciate the work you have put in here.

I do have two main concerns:

  • A. We're introducing a completely new payload, but all these concepts can be expressed with minor additions to the existing payload.
  • B. Rather than allowing the client to specify the order of operations, it requires the client to do so.

Based on the PR description, I think we agree on A except for order of operations. Is this the case? Here's the "sideposting" doppleganger that would add/remove/update comments from an article, using op and lid keys from this PR:

PATCH /articles/1

{
  data: {
    id: "1" 
    type: "articles",
    relationships: {
      comments: {
        data: [
          { type: "comments", lid: "a", op: "add" },
          { type: "comments", id: 1, op: "remove" },
          { type: "comments", id: 2, op: "update" }
        ] 
      }
    }
  },
  included: [
    {
      type: "comments",
      lid: "a",
      attributes: { text: "I love JSONAPI!" }
    },
    {
      type: "comments",
      id: "2",
      attributes: { text: "An updated comment" }
    }
  ]
}

Note that this also supports creating 2 articles at the same time through sideposting to /bulk.

The benefits of a symmetrical payload seem so great to me, point B above (ordering operations) must be overwhelming if we are to introduce a new payload. But I'm not sure that's the case.

Consider a SQL database - you would, 99% of the time, want to persist the parent (e.g. article) before persisting the child (e.g. comments) (since we need the parent id before we can populate the foreign key). In these cases, forcing the client to specify order of operations seems nothing more than cruft (though I'm certainly open to allowing order of operations for the 1% use case).

There's a more important reason than cruft, though. Let's say I coded all my clients to persist the parent before the children while using a SQL database, then switched to a NoSQL store where this logic no longer applies. I would now want to persist these entities differently - but all my clients have already coded the logic to first persist the parent, then the child. I fear there are a number of examples like this, largely summarized by the statement:

Order of operations is a server implementation detail that is not the responsibility of the client.

Which is similar to the concern and examples raised by @gabesullice here.

I'd love to see order of operations supported optionally instead of required, both because it is almost always not needed and often seems not desirable. One way to support order of operations optionally is to take the above sideposting payload and add another key:

...
relationships: {
  comments: {
    data: [
      { type: "comments", lid: "a", op: "add", order: 1 }
      { type: "comments", lid: "b", op: "add", order: 2 }
  }
}
...

We'd end up mirroring the existing sideloading spec, with the addition of three total keys - lid and op, which are in this PR as well, and an optional order (which could even be an extension rather than base spec at that point).

I have some additional comments on the specifics of this PR but would love to get your thoughts on the above first. Again, thanks for the work here, it is awesome to see JSONAPI attempting to address this problem area.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 12, 2018

has http://jsonpatch.com/ also been considered as a basis? Everything here looks great and jsonpatch.com would give just a slight different envelope. An additional benefit of this would be the support of other, non-JSONAPI calls.

@remmeier yes, this proposal was strongly inspired by JSON Patch. One significant difference is that update is used instead of replace to be consistent with the PATCH (instead of PUT) semantics. Another difference is that ref is used instead of a path pointer to acknowledge that the spec is targeting resources and resource collections, instead of a singular JSON document. Given these differences, I don't feel it's right to try to shoehorn exact compliance with RFC 6902 into this proposal, but it's definitely appropriate to call it out as an inspiration :)

one minor other thing, "op" could potentially retain a more HTTP-style naming: GET, POST, PATCH, DELETE.

Just like in JSON Patch, operations occur at a layer distinct from HTTP, and I didn't want to muddy the waters by pretending otherwise.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 12, 2018

@gabesullice @richmolj thanks for considering this carefully and for your feedback!

There are a few aspects of this proposal about ordering and complexity that I'd like to address:

  • Every operation has complete fidelity with a singular request. Everything possible with singular requests can be completed with operations, and vice versa. The only additional complexity is the usage of lids to bind together inputs and outputs if necessary. And the only additional requirement is for servers to process all the operations in a request transactionally.

  • There is no additional knowledge required by clients of how a server persists resources beyond what is required by the existing spec. For instance, a relationship can't be created between an existing resource and one that has not yet been created yet. If request A can follow request B, then operation A can follow operation B.

  • If order is not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order is important.

  • If order can be elided by the server because of some knowledge it has, then I think it is free to do so. However, there should be no "evidence" in the response, because that breaks the contract with the client. If this allowance requires some tweaking of the language used in this proposal, I'm open to it.


Note: edited for clarity, but then reverted edits after I realized others had commented based on the original text.

@gabesullice

This comment has been minimized.

Copy link
Contributor

gabesullice commented Feb 12, 2018

thanks for considering this carefully and for your feedback!

Thanks for taking the time to address each concern so thoroughly!

Everything possible with singular requests can be completed with operations, and vice versa... There is no additional knowledge required by clients of how a server persists resources beyond what is required by the existing spec.

Thanks for clarifying this @dgeb, that's a subtlety that I didn't pick up on.

Perhaps this is something that could be stated explicitly in the operations introduction or as a note.

Something like, "Individual operations in a request are semantically equivalent to singular requests performed serially with a transactional guarantee."

Perhaps that could be left to the recommendations page of the website too :)

If order can be elided by the server because of some knowledge it has, then I think it is free to do so. However, there should be no "evidence" in the response, because that breaks the contract with the client. If this allowance requires some tweaking of the language used in this proposal, I'm open to it.

Does this suggest that the imperative ought to be SHOULD over MUST? I'm actually unclear on this aspect of specification-writing. As long as there's no "evidence," as you said, does the imperative matter?

@richmolj

This comment has been minimized.

Copy link
Contributor

richmolj commented Feb 12, 2018

Thanks for taking the time to address concerns @dgeb ❤️

Everything possible with singular requests can be completed with operations, and vice versa.

This is why I favor mirroring sideloading payload - why not mimic what we do in singular requests (RESTfully pair a resource with a verb) all the way down, instead of introducing a new payload?

  • There is no additional knowledge required by clients of how a server persists resources
  • If order can be elided by the server because of some knowledge it has, then I think it is free to do so.

I'm not sure how these statements are reconciled with the statement in the PR "A server MUST perform operations in the order they appear in the operations array". Doesn't this mean the client now needs to know the correct order to persist resources? Or is this just a matter of updating the PR language?

If order is not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order is important.

This is often not possible, for instance when things need to run in the same transaction but the order doesn't matter - most nested web forms with validation errors would fall into this category. Time-consuming ETLs is another use case I have personally run into.

In fact, none of the examples given in the PR seem order-dependent (at least not for the >90% use case):

  • Creating an invoice together with one or more invoice items.

Most servers would create the invoice, then create the items, then roll back if anything is invalid.

  • For example, removing two members of an organization at the same time two other members are added.

Note sure why the order would matter for the default use case.

  • Deleting an author and some of the articles associated with that
    author.

Most servers would delete the author, then the articles, within a transaction.

Considering this PR makes a great case for supporting common use cases as part of the base spec, and ordering of operations is not a common use case, wouldn't it make more sense to have "sideposting" in the base spec and operations as an extension? Assuming we don't want to go with the option of simply adding an order key.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 12, 2018

Sorry, I rushed my comment as I was going out the door and realized I should retract this statement:

If order is not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order is important.

This would be better stated:

If order and atomicity are not important, then I would strongly recommend making multiple singular requests in parallel. By using operations, the client is saying that order and / or atomicity are important.

@richmolj Thanks for checking me on this point. I'll respond to your other points soon.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 12, 2018

This is why I favor mirroring sideloading payload - why not mimic what we do in singular requests (RESTfully pair a resource with a verb) all the way down, instead of introducing a new payload?

In the side-posting issue you have proposed modifications to the document structure to support layering capabilities on top of side loading. For instance, you proposed adding op or method to individual resources so that you can, for example, delete some resources as you add others. You have to be extremely careful with HTTP compliance here - only PATCH allows for such flexibility for media types.

I'm not sure how these statements are reconciled with the statement in the PR "A server MUST perform operations in the order they appear in the operations array". Doesn't this mean the client now needs to know the correct order to persist resources? Or is this just a matter of updating the PR language?

I said above that I am open to tweaking the PR language. The important thing (to me) is that the client not be able to distinguish that a server has chosen to parallelize processing some operations.

In the current spec, clients already need to understand order dependencies to some extent because servers are free to impose any order dependencies they want. Some servers will require that invoices be created before invoice items, for instance. The specification of allowed ordering of operations seems orthogonal to this debate and is probably best solved with a capabilities specification (something we should definitely tackle).

I should also point out that sideposting can have related concerns. A server may allow sideposted invoice items together with an invoice as primary data, but not the other way around. Ultimately, as a specification, we have to allow servers to control of what's allowed at each endpoint. Again, I think this can be best communicated with a capabilities specification.

@richmolj

This comment has been minimized.

Copy link
Contributor

richmolj commented Feb 12, 2018

@dgeb I actually think we agree on most things.

The specification of allowed ordering of operations seems orthogonal to this debate and is probably best solved with a capabilities specification (something we should definitely tackle).

I agree! Let's put ordering to the side for now

The important thing (to me) is that the client not be able to distinguish that a server has chosen to parallelize processing some operations.

Agree the client should be agnostic in this regard

only PATCH allows for such flexibility for media types.

While I am not sure I agree with this interpretation, I'm happy to agree on PATCH being the only verb used for operations/sideposting

A server may allow sideposted invoice items together with an invoice as primary data, but not the other way around. Ultimately, as a specification, we have to allow servers to control of what's allowed at each endpoint.

Agree on this was well - just as an invoice can sideload invoice items, but not the other way around, there should be no such expectation for a sidepost.

lid

Yep, sounds like we need this no matter what

Given all the above, it looks like we have two choices to solve this problem:

  • Mirror the existing sideload payload, adding a single key ("sideposting")
  • Introduce a completely new payload ("operations")

Since we seem to agree on so much, I'm wondering how we are reaching different conclusions on this final choice. What am I missing?

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 12, 2018

@richmolj Just want to say that I love your emphasis on consensus <3

I'll get back to you soon with some examples that I hope will clarify any differences that remain.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 13, 2018

@richmolj Ok, below is my attempt to distinguish between sideposting, as I believe you're proposing it, and operations as proposed in this PR. Please feel free to correct any misunderstandings about your proposal.


I'm pretty sure it's an oversimplification to say that sideposting just involves "Mirror the existing sideload payload, adding a single key". I think what you're proposing is allowing an op member to be associated with any resource and resource identifier in a compound document. I'm not sure "sideposting" is an accurate name for your counterproposal - maybe "graph mutations" would be more accurate? Anyway, I'll call it sideposting for the sake of discussion.

I believe that sideposting would involve the following changes to the spec at a minimum:

  • If only PATCH is used for sideposting, then every resource and resource identifier in the compound document request would need to include the op member to understand its role in the request. Or there would need to be some concept of a "default" op for the request.

  • We would need to make this breaking for 1.0 servers which do not understand sideposting. This could be handled via version or feature negotiation or somehow choosing a structural distinciton that would make these requests invalid for 1.0 servers. Keep in mind that 1.0 servers are supposed to simply ignore the op member and wouldn't be expecting anything in included in a request.

  • New rules would be needed for responses to guarantee that every lid sent in a request can be mapped to an id in the response. Something along the lines of the identities array would be needed as I explored for a bit in #1244 (before removing it). Without this guaranteed mapping, clients risk never being able to map local ids to server ids.

In addition, to fulfill some of the other use cases solved by operations, the spec would need to expand in additional ways:

  • In order to allow multiple resources of the same type to be created / updated together, we'd have to expand the definition of primary data to allow it to be an array of resources, and not just a single resource.

And then there are some general implementation complexities:

  • Since the graph of resources could be arbitrarily complex (but ultimately limited by the server), a server would need to generate execution plans for each allowed graph based upon the operation associated with each resource or resource identifier. For instance, the server might need to delete some resources before updating others. Cross references between resources would need to be fully accounted for. Basically, the full graph of primary and included resources would need to be traversed before an execution plan could be generated.

  • Clients would need to track all mutations based upon relationships. If a mutation could not be related in one way or another, they could not be included in the same request. There could be substantial bookkeeping on the client.

Åfter all this, sideposting still wouldn't be able to handle use cases like the following:

  • Create three articles and then fetch the first ten articles sorted by title in a single request.

  • Collect mutations which may or may not be related and perform them together transactionally (e.g. while a client is offline).

To contrast, the operations proposal does introduce a new wrapper object, the operation object, but the rules for processing each operation, and the capabilities allowed for each, are identical to the rules for singular requests. The only additional requirements are accounting for local IDs and transactionality (but these requirements are also present in the sideposting proposal).

In sum, I consider the operations proposal to be the smallest delta from the current spec, while also providing the greatest range of new functionality.

@richmolj

This comment has been minimized.

Copy link
Contributor

richmolj commented Feb 13, 2018

Thanks for taking the time to give a detailed breakdown @dgeb, I really appreciate it ❤️ Though I'll still make the case for sideposting below, I think this will lead to some constructive edits and a healthier PR at a minimum.

Re: Sideposting

We would need to make this breaking for 1.0 servers

As the changes are purely additive (and the server should ignore the additions, as you say), how is this breaking? Seems in-line with a minor version release to me.

every lid sent in a request can be mapped to an id in the response

Because this is a mirrored payload, the lid would exist in the response at the same level (in the relationship resource identifier and next to the server-minted id). This is how clients map lid to server-generated ids.

allow multiple resources of the same type to be created / updated together

I think this is an independent issue. "sideposting" would be compatible with an array of data, for instance, which is something that could be merged independently of this discussion. Relatedly: see the "Operations: B" note below, where I note this might be a better fit for an extension than the base spec.

As a minor note (don't want to get held up on this point), sideposting would support this even without that independent PR like so:

PATCH /bulk

{
  data: {
    type: "bulk_updates"
    relationships: {
      articles: [
        { lid: "a", op: "create", type: "articles" },
        { lid: "b", op: "create", type: "articles" }
    }
  }
  included: [...article data...]
}

server would need to generate execution plans for each allowed graph based upon the operation associated with each resource or resource identifier

I believe that this is a feature, not a bug. Baking this logic into the client means coupling the client to server implementation. Taking the example "For instance, the server might need to delete some resources before updating others." - what if this was true in January, but the reverse is true in December after I have swapped my datastore? I should be able to keep the same API without requiring all clients to be updated.

I think this comes back to the "A server MUST perform operations in the order they appear in the operations array" language in the PR that @gabesullice and I commented on above, and is my chief concern with operations. If we removed the "Must" here it seems both solutions would have that same server-side challenge, if we keep the "Must" it seems like it would lead to client/server coupling.

Clients would need to track all mutations based upon relationships

I'm unsure what you mean by this, could you give an example? How is this different from sideloading, which is also based on available relationships?

Again, I think this might be an independent issue regarding single-request writes of unrelated data.

Re: Operations

Create three articles and then fetch the first ten articles sorted by title in a single request.

Collect mutations which may or may not be related and perform them together transactionally (e.g. while a client is offline).

On the first, I agree, sideposting would not support this. On the second, I somewhat agree but as noted above I think this is a separate PR possibly unrelated to operations or sideposting.

In any case, I'm guessing you separated these out into a separate section because they are less common scenarios. If these two points are what separates the abilities of operations from the abilities of sideposting, it would make sense to

A) Update the PR to explicitly call this out as a requirement we are trying to solve
B) Move operations to an extension and sideposting into the base spec

The reason I arrive at B: per the PR description, this was originally an extension or separate media type, but the scenarios were too common to ignore. I actually strongly agree with this when we are talking about updating primary data alongside relationships in a single request (as "sideposting" solves)...but disagree when we are talking about blending reads and writes into a single request (which "operations" solves, amongst other important but less-common use cases).

As an aside: the benefit of "sideposting" is the intuitive mirroring of "sideloading", which we generally expect from RESTful APIs that are built to transfer the same state representation back and forth to the server. There is no question that "operations" is a more powerful paradigm - but in my opinion it diverges enough from expectations that it makes more sense as an official extension.

in [Errors].
> Note: An empty object (`{}`) is a valid operation object, and thus can be
returned as a result to an operation request.

This comment has been minimized.

@EndangeredMassa

EndangeredMassa Feb 13, 2018

Allowing an empty object to be a valid operations object makes sense in a response, but not a request, right? Should the spec make it clear that this is the case or am I missing something here?

Also, should this be more than a "Note"? This seems like an important part of the spec. The phrase "can be" in this note also seems out of place when we clearly use the imperatives everywhere else. Should it be **MAY** be?

This comment has been minimized.

@dgeb

dgeb Feb 13, 2018

Author Member

This is a note and not a normative portion of the spec because it does not introduce any new requirements or information. In other words, the fact that an empty object can be a valid operation object can be deduced from the description of the operation object. Since it's not normative, we shouldn't use language like **MAY**.

@@ -164,6 +169,8 @@ In addition, a resource object **MAY** contain any of these top-level members:
* `links`: a [links object][links] containing links related to the resource.
* `meta`: a [meta object][meta] containing non-standard meta-information about a
resource that can not be represented as an attribute or relationship.
* `lid`: a local identifier that, together with `type`, uniquely
identifies a resource within the document.

This comment has been minimized.

@EndangeredMassa

EndangeredMassa Feb 13, 2018

Should it be in the scope of the spec to note the value of UUIDs in solving this problem? More specifically, generating a UUID locally and posting that to a server allows you to ensure that the id on the server is the same value without any real chance of conflict.

This comment has been minimized.

@dgeb

dgeb Feb 13, 2018

Author Member

I agree that this would make a good non-normative note 👍 I think we may mention client-generated IDs in a couple places, but mentioning it near lids makes sense as well.

This comment has been minimized.

@wimleers

wimleers Feb 14, 2018

Contributor

Thanks for asking this, @EndangeredMassa, it made me ask a question at #1244: #1244 (comment)

This comment has been minimized.

@krainboltgreene

krainboltgreene Apr 13, 2018

Contributor

IMO this should just be called local, jsonapi doesn't truncate words anywhere else.

> Note: `op` codes are used instead of the HTTP methods prescribed for singular
requests. The values `"get"`, `"add"`, `"update"`, and `"remove"` align with
the HTTP methods `GET`, `POST`, `PATCH`, and `DELETE` used for singular
requests.

This comment has been minimized.

@EndangeredMassa

EndangeredMassa Feb 13, 2018

I understand why you don't want to clash meaning with the http verbs and op values, but the logical alternative to me is "create", "read", "update", "delete" (CRUD). Was this considered when naming these operation types?

This comment has been minimized.

@dgeb

dgeb Feb 13, 2018

Author Member

Haha ... I just had this thought this morning. The original intent was to align with RFC 6902, but since we're now using update and not replace, we may just want to use the more memorable CRUD theme.

@EndangeredMassa

This comment has been minimized.

Copy link

EndangeredMassa commented Feb 13, 2018

The lack of bulk operations (mostly bulk updates) severely hinders adaptability of JSON API in conversations I've had about standardizing API wire formats. I'm excited to see progress made on this front and I very much agree that something like this should go in the base spec.


I like that this doesn't really add much in terms of functionality. It appears to be a small superset of the existing functionality, right? It's essentially a tool for optimizing for less round trips to the server. It has me wondering: if operations can essentially do anything the rest of the spec can do and one of the huge benefits to a standard wire format is that standard tools on each end can understand it, what need do we have for the spec's top-level document structure definition keys (data, included) or the use of HTTP VERBS at all?

For my use of JSON API, it's an explicit goal that I rarely had to read the actual request/response because the tooling on each side does such a good job abstracting that away. That tooling could be simplified if it always used operations, right?

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 13, 2018

@richmolj I'm not sure whether to keep this conversation going here. Maybe a separate issue would be better to allow operations to be refined here? But anyway, I appreciate your responses and will follow up as much as time allows:

As the changes are purely additive (and the server should ignore the additions, as you say), how is this breaking?

We would want 1.0 servers to treat sideposted requests as breaking because they would attempt to understand the payload with a PATCH method as a resource update request, and they would ignore the op code in the resource. In other words, 1.0 servers would misinterpret sideposting because they have no concept of this new feature. We don't want clients to get a false "success".

Because this is a mirrored payload

Are include or sort params allowed to customize the response? Or is it a hard rule that the response directly mirrors the request?

I think this comes back to the "A server MUST perform operations in the order they appear in the operations array" language in the PR that @gabesullice and I commented on above, and is my chief concern with operations. If we removed the "Must" here it seems both solutions would have that same server-side challenge, if we keep the "Must" it seems like it would lead to client/server coupling.

I've responded above that I'm fine to tweak the MUST language: "The important thing (to me) is that the client not be able to distinguish that a server has chosen to parallelize processing some operations." Note that this would allow servers to create an optimized execution plan, but not require it.

I'm unsure what you mean by this, could you give an example?

Clients would need to track dirtying of record graphs instead of tracking a queue of changes (i.e. operations).

I'm guessing you separated these out into a separate section because they are less common scenarios.

The other sections were related to adjustments that would be required to fulfill sideposting as you're describing it. I separated these items out because I have not heard any proposals related to sideposting that attempt to address these scenarios.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 13, 2018

@EndangeredMassa Thanks very much for your feedback!

I like that this doesn't really add much in terms of functionality. It appears to be a small superset of the existing functionality, right?

That's correct.

That tooling could be simplified if it always used operations, right?

Yes, tooling could use operations exclusively without losing any functionality from the current spec.

@richmolj

This comment has been minimized.

Copy link
Contributor

richmolj commented Feb 14, 2018

and they would ignore the op code in the resource

OK, I think I found the source of my confusion 🎉 In trying to be agreeable and accept PATCH, I didn't think it through and realize that this would introduce a breaking change - TBH, the way I currently implement it is to use whichever verb I'd normally use on primary data, meaning op isn't needed for the top-level resource. So I agree PATCH-only sideposting would be a prohibitive breaking change, and this is a great point.

That said, I'm not sure the HTTP spec actually prohibits POST here. Happy to be corrected, but what I read is:

Providing a block of data, such as the result of submitting a form, to a data-handling process
The actual function performed by the POST method is determined by the server

And from the PATCH specification:

A comparison to POST is even more difficult, because POST is used in widely varying ways and can encompass PUT and PATCH-like operations if the server chooses.

So: each resource is paired with a verb. That verb comes from the HTTP method on primary data, and op for relationship data. REST with relationships.

Are include or sort params allowed to customize the response? Or is it a hard rule that the response directly mirrors the request?

While I personally would like to disallow these, it seems fine that clients can supply them as overrides, explicitly opting-out of the mirror.

I'm not sure whether to keep this conversation going here

I certainly don't want to effectively block this PR with noise (and it's regrettable Github doesn't have the equivalent of Slack threads 😞 ). I want this PR to succeed, and I don't see the approaches as mutually exclusive. Just as "operations" can perform the same functions as the base spec, I believe it can perform the same functions as "sideposting" without being considered undue duplication. I view "sideposting" as a simpler, more REST/less RPC alternative that could accommodate all the main goals laid out in the PR description (and the scenarios noted in the original #795), while "operations" is its more-powerful cousin.

If you agree it's possible these payloads could coexist (without breaking changes or violating the HTTP spec, per above), I'd be happy to create a PR based on these comments where we can have further discussion. If you tend to think it's one or the other, I think I've said my piece (though I'd appreciate 2 cents from @beauby or @ethanresnick at some point).

@wimleers
Copy link
Contributor

wimleers left a comment

@richmolj and @dgeb: You both have excellent consensus building skills :)


I have a few key concerns:

  1. "bulk operations", but in reality it is "transactions", because no notion of partial success
  2. tooling consequences of allowing allowed operations and target resources at each endpoint
  3. cacheability consequences of allowing operations requests to only read resources
@@ -190,8 +197,18 @@ Here's how an article (i.e. a resource of type "articles") might appear in a doc

#### <a href="#document-resource-object-identification" id="document-resource-object-identification" class="headerlink"></a> Identification

Every [resource object][resource objects] **MUST** contain an `id` member and a `type` member.
The values of the `id` and `type` members **MUST** be strings.
As noted above, every [resource object][resource objects] **MUST** contain a

This comment has been minimized.

@wimleers

wimleers Feb 14, 2018

Contributor

This portion of the changes actually now lives in #1244, but I think you're keeping it in this PR for now to have a single PR to review everything as one whole?

This comment has been minimized.

@dgeb

dgeb Feb 14, 2018

Author Member

Yeah, I figured portions of this PR would not make sense without basing it on #1244.

For the sake of clarity, requests that include operations are called
"operations requests" while requests that perform a single fetch or mutation are
called "singular requests".

This comment has been minimized.

@wimleers

wimleers Feb 14, 2018

Contributor

❤️ Crystal clear!

`204 No Content` and no response document if no operations are required to
return `data`.
If any operation in a request fails, the server **MUST** respond as described

This comment has been minimized.

@wimleers

wimleers Feb 14, 2018

Contributor

What I'm missing here, is the notion of partial success.

A single operations request could:

  1. create a resource (f.e. create a comment that is a reply to another comment)
  2. fetch resources (f.e. fetch the next page of comments)

Any of those could fail, and in different degrees:

  1. the comment I'm posting could be a reply to a comment that has since been deleted … but I'd still want the other operations to succeed
  2. some of those comments I may not be able to access because they're still in moderation

In either of those cases, I would expect to see the things that did succeed to actually return the expected data … but this pointing to [Errors] (http://jsonapi.org/format/#errors) means that is not possible, because as http://jsonapi.org/format/#document-top-level says:

The members data and errors MUST NOT coexist in the same document.


I suspect the answer is going to be along the lines of "you should use operations requests not for bulk operations, but for transactions, i.e. for operations that should atomically succeed or fail". @dgeb stated something along those lines in #1254 (comment).


Which then leaves me wondering if we're not misnaming this PR and the concepts being added to the spec. Because @EndangeredMassa wrote specifically in #1254 (comment) that "bulk operations hinders adoptability". I suspect many of us see "bulk operations" and conclude that this PR brings that.

He even asked specifically:

That tooling could be simplified if it always used operations, right?

Which @dgeb answered with:

Yes, tooling could use operations exclusively without losing any functionality from the current spec.

But that's not entirely accurate as far as I can tell. Merging many singular requests into a single operations request does lose functionality: any of those singular requests can result in errors without affecting the other singular requests. But if those singular requests are merged into a single operations request, then any single operation failing requires the entire operations request to fail.
Which means that tooling could not exclusively use operations.


I'm wondering whether this should remove all mentions of "bulk" and "operations", and instead use "transactions" in both the spec and as the title of this PR.

Quoting @dgeb from #1254 (comment):

The only additional requirements are accounting for local IDs and transactionality

This comment has been minimized.

@gabesullice

gabesullice Feb 14, 2018

Contributor

Am I summarizing this correctly @wimleers ? You're suggesting:

The PR and new top-level member, operations, should be renamed to better reflect that this proposal is about atomicity, not bulk processing.

Bulk processing, i.e. bundling requests, is actually an anti-pattern where atomicity is not required.

See also, @dgeb in #1254 (comment)

If order is not important, then I would strongly recommend making multiple singular requests in parallel.

This comment has been minimized.

@dgeb

dgeb Feb 14, 2018

Author Member

The transactional nature of this proposal is actually consistent with the current spec:

A request MUST completely succeed or fail (in a single “transaction”). No partial updates are allowed.

This prevents changes to one attribute or relationship from succeeding while others fail. The operations proposal just extends this further.

This comment has been minimized.

@EndangeredMassa

EndangeredMassa Feb 15, 2018

But that's not entirely accurate as far as I can tell. Merging many singular requests into a single operations request does lose functionality: any of those singular requests can result in errors without affecting the other singular requests. But if those singular requests are merged into a single operations request, then any single operation failing requires the entire operations request to fail.
Which means that tooling could not exclusively use operations.

What I meant here was that the tooling could exclusively use operations if it always used one operation per request, not that it could necessarily submit multiple operations all the time.


Requiring this be transactional means that it's serving some uses well, but others not as well. You listed out use cases up top, but I think there are more to consider. I named yours for reference:

Compound Document: Creating multiple resources of the same or different types together and then creating relationships between them. For example, creating an invoice together with one or more invoice items.

Transactionality here seems necessary. If any part of this compound resource fails, you likely want the entire thing to fail.

Add/Remove Relationships: Editing memberships of very large relationships, assuming that edits include additions and deletions. For example, removing two members of an organization at the same time two other members are added.

Transactionality here seems optional. If I've done a lot of relationship remapping and one specific change fails, then I definitely want to know about that. However, I don't think I necessarily want them all to fail.

Cascading Deletes: Clearing relationships and explicitly deleting the related resources as well. For example, deleting an author and some of the articles associated with that author.

Transactionality here seems necessary. I think we'd suggest to API developers to cascade deletes on the server side responsibly, but if an API doesn't do that, you'd want to submit a series of operations like this in a transactional manner.

Offline Sync: Sending a general "batch" of changes that must succeed or fail together. For example, queuing up an array of operations while offline and then applying them together.

Transactionality here seems problematic. It means our offline sync is more likely to fail entirely. I think we'd rather have partial success and show the individual failures to the user. Should a change conflict on a single record fail your entire offline sync that creates/updates/deletes many different records?

New Use Case: Network Performance Sensitivity: A latency-sensitive application optimizing for every drop of network performance wants to submit combine as many operations as it can, whenever it can. (Example: A mobile application running in a poor-connection area wants to update an auto-buy stock record AND get the latest stock prices for your tracked stocks.)

Transactionality here means seems problematic. It means that your unrelated operations can fail each other, which will require follow-up requests, possibly splitting them up into multiple requests to avoid the transactionality of the system. Should my resource update fail my unrelated resource read?


The use cases around related records seem to benefit from transactionality, but the use cases around unrelated records seems to be limited by it. I think that this shows that transactionality should be optional in some way. In my opinion, if transactionality is required, bulk operations won't be as useful.

This comment has been minimized.

@gabesullice

gabesullice Feb 15, 2018

Contributor

@EndangeredMassa thanks for the thorough write up!

My understanding of the proposal thus far is that it's only intended to permit transactionality/atomicity and that the proposal wasn't intending to enable "bulk" operations like your final "network performance" example. While it would be possible, it wouldn't be recommended.

The commit summary which introduced this reads:

Operations can be used to perform one or more fetch or mutation operations serially and transactionally.

I think this is exactly why @wimleers said:

I'm wondering whether this should remove all mentions of "bulk" and "operations", and instead use "transactions" in both the spec and as the title of this PR.

Reducing the perceived scope of this proposal just to transactionality with a few language changes will guide users to its intended usage. Like so:

Introduce Operations to v1.1 Introduce Atomic Operations to v1.1

 A server **MAY** support usage of the `operations` array to complete one or more
-fetch or mutation operations in a single request.
+fetch or mutation operations in a single transaction.
...
-  PATCH /bulk HTTP/1.1
+  PATCH /transaction HTTP/1.1

This comment has been minimized.

@EndangeredMassa

EndangeredMassa Feb 15, 2018

Does the scope creep too much if we remove that requirement? If so, I can understand keeping it for now and proposing a separate thing that could build off of the work done in this proposal. I would then also support making it more explicit that this is for transactions.

This comment has been minimized.

@gabesullice

gabesullice Feb 15, 2018

Contributor

I think it's impossible to remove the transactional requirement in this proposal:

A request MUST completely succeed or fail (in a single “transaction”). No partial updates are allowed. — http://jsonapi.org/format/#crud

Bulk operations with partial success would be a hard break from v1 of the spec.

Which would mean @gabesullice (me), @wimleers, and @EndangeredMassa (you) all "support making it more explicit that this is for transactions."

@dgeb, what are your thoughts? Have we missed anything?

This comment has been minimized.

@dgeb

dgeb Feb 15, 2018

Author Member

Transactionality here seems problematic. It means our offline sync is more likely to fail entirely. I think we'd rather have partial success and show the individual failures to the user.

@EndangeredMassa I agree with your analysis here. The way I presented this use case was a bit sloppy, because it's not how I would handle offline sync either. I would send operations together that map to a logical user action (say, saving a form) in a single request. And I would serially send requests until the offline queue was cleared, so that client code could reason about failure cases.

There is another level this proposal could go, in which we allow an array of operations to be grouped together into a transaction, and we send an array of transactions. However, this could be handled at a later date in a separate non-breaking proposal that relies on a top-level transactions array. Something like this may seem like an ultimately efficient and flexible solution, but there are also a lot of non-obvious decisions re: responses that represent mixed results. I'd rather not muddy the waters of the operations proposal at this time by expanding the scope this much.

This comment has been minimized.

@dgeb

dgeb Feb 15, 2018

Author Member

Bulk operations with partial success would be a hard break from v1 of the spec.

@gabesullice I think we could technically create new rules for operations requests distinct from the rules for singular requests, since operations are a new feature. However, I am not in favor of introducing non-atomic requests now (see my response to @EndangeredMassa).

> Note: The server can decide which operations and target resources are allowed
at each endpoint. Servers may use resource-specific endpoints like `/articles`
and `/authors` or generic endpoints for all resources like `/operations` or
`/bulk`.

This comment has been minimized.

@wimleers

wimleers Feb 14, 2018

Contributor

How can the tooling know which target resources are allowed at each endpoint? Doesn't this prevent generic tooling from being written?

Like the one @EndangeredMassa referred to in #1254 (comment):

That tooling could be simplified if it always used operations, right?

This comment has been minimized.

@dgeb

dgeb Feb 14, 2018

Author Member

I consider the enumeration of capabilities at an endpoint to be orthogonal to this PR and the current spec. But I do think that as a community we should consider a capabilities spec!

{
"operations": [{
"op": "get",

This comment has been minimized.

@wimleers

wimleers Feb 14, 2018

Contributor

I'm wondering if it should be forbidden to do operations requests that only fetch resources, because this severely hurts cacheability. Arguably, that's a choice of the client. But if the client uses tooling that uses operations requests for everything, that means they never are able to reuse cached responses!

IOW: should operations requests have at least one mutation? Allowing reading data after a mutation makes sense: to update the client's data store to the post-mutation state.

This comment has been minimized.

@dgeb

dgeb Feb 14, 2018

Author Member

IOW: should operations requests have at least one mutation?

I think we can make this a recommendation but not a formal requirement.

This comment has been minimized.

@gabesullice

gabesullice Feb 14, 2018

Contributor

I suspect that many client implementations will be tempted to use operations for all requests in pursuit some perceived simplification (they will not need to manage specific resource URLs, for example).

However, PATCH requests are extremely difficult to cache by nature. API-providers may be reluctant to implement this version of the spec if they are concerned that they will no longer be able to cache most reads at the edge or if it might require re-tooling for things like rate-limiting and/or API gateway quotas.

A statement like:

Client implementations SHOULD use singular requests where atomicity of the operations is not required.

Note: Clients should prefer multiple singular requests sent asynchronously to operations requests when their operations do not need to succeed or fail as a single transaction.

Would send a clear signal that there will be both a performance enhancement for the client (in theory, unrelated parallel PATCHes should be faster than the same ones performed serially). It might also put API providers' minds at ease.

For reference, RFC5789 Patch Method for HTTP specification:

A response to [PATCH] is only cacheable if it
contains explicit freshness information (such as an Expires header or
"Cache-Control: max-age" directive) as well as the Content-Location
header matching the Request-URI, indicating that the PATCH response
body is a resource representation. A cached PATCH response can only
be used to respond to subsequent GET and HEAD requests; it MUST NOT
be used to respond to other methods (in particular, PATCH).

This comment has been minimized.

@dgeb

dgeb Feb 14, 2018

Author Member

@gabesullice I agree that we should make a pretty clear statement about caching + PATCH and recommend support for singular requests in addition to operations. Thanks for your suggestions!

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Feb 14, 2018

If you agree it's possible these payloads could coexist (without breaking changes or violating the HTTP spec, per above), I'd be happy to create a PR based on these comments where we can have further discussion. If you tend to think it's one or the other, I think I've said my piece (though I'd appreciate 2 cents from @beauby or @ethanresnick at some point).

@richmolj thank you for the discussion! I believe we have come to an understanding about each other's proposals and helped refine both in the process. I would say that both are defined clearly enough through these discussions (in my mind at least) to make a decision about how and whether to support one or both. I won't discourage you from creating a separate issue to refine your proposal further, but I don't think it's strictly necessary for my benefit after our discussions here.

We have a few hard decisions to make, discussed in #1207 and elsewhere, about extending the spec. We also have an obligation to make hard decisions that prevent bike shedding and lay out reasonable and clear guidance, or else we risk this spec fading in relevance. Hopefully @wycats, @steveklabnik, @ethanresnick, and I can make these tough decisions soon.

endpoint if that endpoint represents multiple resources. `ref` **MUST** be
expressed as an object that **MAY** contain any of the following members:
- `type`: resource `type`
- `id` or `lid`: resource identifier or local identifier

This comment has been minimized.

@jaredcnance

jaredcnance Mar 2, 2018

Contributor

It may be useful to provide an example when you might include a lid member on a ref. Personally, I'm struggling to see a use case for it. From what I can tell the only time I need a lid is when I am creating a new resource and need to reference that resource somewhere else in the same request. The only place I can imagine needing to reference it somewhere else would be as a relationship to another resource in a resource identifier object. Other possible scenarios this allows:

Add followed by a get.
This is not necessary since a is returned by the first operation

0: add a
1: get a

Add followed by an update or remove
I can't see a purpose for this.

0: add a
1: update a
2: remove a
- `sort`
- `filter`
- `page`
- `fields`

This comment has been minimized.

@jaredcnance

jaredcnance Mar 31, 2018

Contributor

Initially, I would assume that each of the corresponding value types should be string to keep in line with the same form they would take in a query string? Or is the type open to implementation specific interpretation? For example, an implementation may currently define a filter strategy that allows values to be prefixed with a comparison identifier (e.g. filter[name]=like:bikeshed). I would expect params.filter to work the same way, but if the spec is ambiguous about the type the value may be, I could see this enabling implementations to provide more powerful filter actions:

{  
   "params": {  
      "filter": {
         "name": { "like":"bikeshed" }
      }
   }
}

or even more complex expression trees:

{
   "params": {
      "filter": [
         {
            "name": [
               {
                  "or": [
                     { "like":"paints" },
                     { "like":"bikeshed" }
                  ]
               }
            ]
         }
      ]
   }
}
@ef4

This comment has been minimized.

Copy link

ef4 commented Apr 12, 2018

With the introduction of a first-class operations concept in the spec, we may want to eliminate the two places that word was already used more colloquially: "..pagination operations..." and "...filtering operations...", since those are not talking about the operations feature at all.

@mydea

This comment has been minimized.

Copy link

mydea commented Aug 28, 2018

What's the status on this? We are very excited about this and would love to see it landing!

@pcrglennon pcrglennon referenced this pull request Aug 30, 2018

Open

[WIP] Sideposting draft #1197

6 of 17 tasks complete

@arjun289 arjun289 referenced this pull request Sep 12, 2018

Merged

Order transition address to delivery #186

0 of 6 tasks complete

@dgeb dgeb changed the title Introduce Operations to v1.1 Introduce Operations to v1.2 Dec 3, 2018

@mydea

This comment has been minimized.

Copy link

mydea commented Dec 3, 2018

Really sad to not see this in 1.1 :( What's the reason for this? What is open to get this (Operations) finalized? And what is the (rough) timetable for 1.2? This has been open for a pretty long time now, so it would be really great to finally have these capabilities in JSON:API.

@dgeb

This comment has been minimized.

Copy link
Member Author

dgeb commented Dec 3, 2018

Really sad to not see this in 1.1 :( What's the reason for this? What is open to get this (Operations) finalized? And what is the (rough) timetable for 1.2? This has been open for a pretty long time now, so it would be really great to finally have these capabilities in JSON:API.

@mydea from my perspective, operations fills one of the most pressing gaps in the spec, so I understand your feelings. Simply put, we decided to ship 1.1 with a smaller set of features rather than delay it further, since profiles will bring a lot of value on their own. We will resume discussing 1.2 this week with the understanding that operations are now the highest priority. I think it's possible that we could ship a 1.2 RC within a few months of 1.1 final.

@donnysim

This comment has been minimized.

Copy link

donnysim commented Dec 7, 2018

I've skimmed through this a bit, and wonder how would this look with something like "belongs to" relation? If the client is creating a Place and in the same window/form he can create a City (through a modal, like a + button near city select etc.), so Place belongs to City and both should be persisted on save. In this scenario City must be created before Place, but if this is handled on the client side, then the client needs to know about what kind of relationship this is (and all the others) and make implementations/execution plan based on it. From developers standpoint, this would be a "disgusting" API to work with? What would be the go to in this scenario? From my point of view, the server should be the one to build the execution plan, otherwise this leaves API's frustrating to use if anything.

@marijnz0r

This comment has been minimized.

Copy link

marijnz0r commented Feb 20, 2019

Hi @dgeb,

Could you provide us with an update regarding this pull request and/or version 1.2?

Kind regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment