Introduce Local IDs to v1.1

[dgeb]

A client may include a "local ID" as a lid member in a resource object to
uniquely identify the resource within the request document. Every representation
of that resource, whether as a resource object or resource identifier object,
must then include the matching lid member and value.

When a server receives a request document that contains resources with local
IDs, the server must include the matching lid member and value in every
representation of that resource or resource identifier in the response document.

This addition paves the way for requests of all kinds that may need to establish
linkage between resources that have not yet been assigned a server-generated ID.

---

Note: This proposal obviously overlaps with a portion of #1197, but I'd like to more fully integrate the concept of local ids in the spec without restricting them to a section on "side posting".

Note: I'm proposing the name lid after considering negative feedback to my key proposal in #1197. I favor a name that references either "local" or "document" instead of "temporary", because this field does not really seem to have a temporal component to it. I also don't like squishing multiple words into one (like tempid) but an abbreviation like lid is different and consistent with id.

[ethanresnick]

👍

Thanks for writing this up. lid as a name seems fine.

I just want to note the connection between the approach here and #1245: adding lid to existing resource identifier objects only works if we have a feature negotiation strategy in place; otherwise, the server will ignore the lid while continuing to process the request as valid, which (I think...) could produce the wrong result in an unacceptable way. That's why the proposal in #1197 defined an entirely new resource identifier format, i.e., something like:

{ "lid": "...." /* not id key here; maybe no type either, but that doesn't matter */ }

This makes lids basically an optional feature that can hard fail if not supported, analogous to how the base spec handles unsupported optional features like ?sort and ?include.

Again, a new format like that may not be necessary; I'm just highlighting the dependency on #1245 in its absence.

[sandstrom]

@dgeb Very much in favour! 🏅

This should enable implementors such as Ember Data to solve some issues around nested resources. May also be useful as an alternative to JSON pointers.

[dgeb]

I've updated this PR to introduce the concept of a top-level identities array, which allows correlation between lids in a request with ids in a response.

By including an identifier in this array, a client can guarantee that the server will return it, regardless of the shape of the data in the response.

[sandstrom]

@dgeb Awesome improvement! 😄

Easy to parse for clients (that need to pair up lids with ids), and conceptually simple (i.e. easy for people to understand).

[dgeb]

While the identities array is necessary to guarantee that clients can always receive server-generated IDs in #1197 (side-posting), it is not strictly necessary in #1254 (operations). Thus, I've removed this commit for now to keep this PR as lean as possible and will rebase #1254 to use this.

I've also reworded some of the language in this PR as per suggestions from @courajs in #1254.

I think I will let the optional identities array stand on its own as a separate PR if there is strong support for it, but I think it may cause more confusion than good until we resolve more fundamental issues.

[valscion]

This [resource identifier objects] fails to link to the correct section. Should this instead be

[resource identifier objects][resource identifier object]

[DLiblik]

missing word: "...of that resource, including within other..."

[wimleers]

I don't think that word is missing? Because:

including other ROs and RIOs must also contain […]

("including within other ROs and RIOs must also contain" does not sound right to me)

[DLiblik]

The full sentence:

If a resource object contains a lid member, then every representation of that resource, including other resource objects and [resource identifier objects], MUST also contain the lid member with the same value.

I am suggesting clarifying that the other resources objects are referring to the subject of the sentence, not that they themselves are other versions of it (which is prohibited by JSON API).

Here's another go, even more specific:

If a resource object contains a lid member, then every representation of that resource, including referential representations contained within other resource objects and [resource identifier objects], MUST also contain the lid member with the same value.

[wimleers]

Just pointing out a few potential clarifications.

P.S.: after reading "lid member" many times now, I can't help but point out something funny: "member" in Dutch is … "lid" 😄

[wimleers]

I don't think that word is missing? Because:

including other ROs and RIOs must also contain […]

("including within other ROs and RIOs must also contain" does not sound right to me)

[wimleers]

Should this then state explicitly that lids can only exist in request documents, not in response documents?

[sandstrom]

@wimleers lid can exist both in the request and the response object (that's sort of their purpose)

[wimleers]

Right, to communicate to the client what the server-generated id for a client-specified lid is. Of course. Silly me!

[wimleers]

There is no actual change here, just reflowing. This change can/should hence be reverted AFAICT.

[wimleers]

There is no actual change here, just reflowing. This change can/should hence be reverted AFAICT.

[wimleers]

After reading @EndangeredMassa's review at #1254 (review), I started wondering why we even need lids? Why can't we just ask that clients generate UUIDs and specify those as the id for each new resource to be created?

I'm assuming it's for two reasons:

1. we don't trust that every client generates UUIDs correctly (most importantly: sufficiently random)
2. we don't want to burden every client with generating UUIDs correctly

[dgeb]

@wimleers Thanks for reviewing! And yes, you are correct that the entire need for lids goes away if you can depend on client-generated UUIDs. I think we should encourage their use more, although we can't force the issue too much because of the two reasons you gave.

[DLiblik]

@dgeb @wimleers Hang on - this statement is not true:

...the entire need for lid goes away if you can depend on client-generated UUIDS.

The broader domain problem being solved by lid is linkage on two levels:

1. Linkage between resources in a request where some resources do not yet have a persisted ID assigned.
2. Linkage between resources in a request that do not yet have a persisted ID assigned and those same resources in the response where they do have a persisted ID assigned.

By far the most common driver of this is that the server does not allow for client ID generation, not
that the client can't do a good job of random ID generation.
This happens for lots of server-side reasons, some of the more common ones being:

• incrementing ID (in enterprise systems, this is still the dominant case and is REQUIRED in many systems
  for real-world concerns like financial audits, where, for example, invoice numbers that are sequential
  allow for detection of fraud via missing invoice numbers, etc.)
• dependent/derived ID's where parts of the information used in ID generation does not exist on the client
• ID-space distribution (a.k.a. hashing) where the ID is transformed to evenly distribute over a target space
• ... and so on

You might argue the validity of any of those points from a design perspective, but for now I just
want to be able to wrap those systems in a JSON API interface that is efficient for the client!

[dgeb]

@DLiblik I wholeheartedly agree that the spec should continue to be compatible with systems which do not support client-generated UUIDs. I believe the conditional if in my statement keeps it correct, but sorry if it was confusing or misleading.

[DLiblik]

10-4 @dgeb - and thanks for putting this all together. Btw, we've being doing this "our own way" since 2015 (we tucked a {meta: _localId: "123" } on our objects and otherwise followed all the same rules) and I can tell you that this approach does solve the problems it sets out to address! I'm very happy to see it get formalized into the spec.

[krainboltgreene]

Bit late in, but as I pointed out in another PR this is the first specification name that is truncated. Any reason we shouldn't just call this a local or some other real word?

[sandstrom]

@dgeb I know you're busy with many other things, but just curious if there are any outstanding issues with this PR or #1268? 😄

[dgeb]

@sandstrom there are a few minor details to iron out in #1268. I hope we can land that very soon, and then follow up with local IDs and operations.

[sandstrom]

@dgeb Just curious, now that profiles (#1268) is merged, are there any outstanding issues here?

[sandstrom]

Looks good to me!

[jesseclark]

I'm concerned about the parameter name 'lid'. It doesn't convey any information about what the param is actually used for. Could we consider changing it to something more meaningful? I don't see any params that are two combined words in the spec anywhere so not sure if that is allowed or handled with hyphens or underscores but what about 'linkage_id', 'local_id', or something else that ties in with the description of what the purpose of the parameter is?

[sandstrom]

@jesseclark I understand your point, but I actually prefer a succinct name in this case. It's clearly explained in the spec, so it's easy to look up (id itself is an abbreviation of identity/identification).

Somewhat similarly, the spec also contain meta (short for metadata) and prev (for previous).

[jesseclark]

@sandstrom id has been so long in usage in computing that it is fairly universally understood at this point, lid not so much. If someone is looking at a payload that contains lid they almost have to go look it up, while a name like linkage_id or link_id would at least give some clue to the purpose and seeing that param with the same values in different sections of a json document one could infer that the field linked those sections together. lid increases cognitive dissonance in the spec.

Also, meta and prev are also in somewhat common usage in programming. Both are also abbreviations for one word while lid combines two abbreviations. Would mda or pre have been acceptable alternatives to meta or prev?

[krainboltgreene]

lid is going to be impossible to Google.

…

On Fri, Sep 21, 2018, 9:36 AM Jesse Clark ***@***.***> wrote: @sandstrom <https://github.com/sandstrom> id has been so long in usage in computing that it is fairly universally understood at this point, 'lid' not so much. If someone is looking at a payload that contains 'lid' they almost have to go look it up, while a name like 'linkageid' would at least give some clue to the purpose and seeing that param with the same values in different sections of a json document one could infer that the field linked those sections together. 'lid' increases cognitive dissonance. Also, 'meta' and 'prev' are also in somewhat common usage in programming. Both are also abbreviations for one word while 'lid' crushes together two abbreviations. Would 'mda' or 'pre' have been acceptable alternatives to 'meta' or 'prev'? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1244 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUb2VvojqRRUW0IDmIE3lS2MM3s1SVfks5udRVxgaJpZM4Q74_a> .

[jbescoyez]

@jesseclark @krainboltgreene Stop bike shedding please. We really need local ids land to JSONAPI. IMHO It has been enough time discussing that. lt is just fine regarding the constraints the jsonapi team has set. So, let it go and keep going forward.

[krainboltgreene]

This is a specification. Understandability and searchability are not "bikeshed" topics. They're paramount. On Sep 21, 2018 4:26 PM, "Jean-Baptiste Escoyez" <notifications@github.com> wrote: @jesseclark <https://github.com/jesseclark> @kraigboltgreene Stop bike shedding please. We really need local ids land to JSONAPI. IMHO It has been enough time discussing that. lid is just fine regarding the constraints the jsonapi team has set. So, let’s does it go and keep forward. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1244 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUb2Rx4ggxAfdLRnKGlAUdEUBt4KDlOks5udXW-gaJpZM4Q74_a> .

[jbescoyez]

@krainboltgreene You are right. Sorry for the negative comment above. It does not bring anything good to the conversation. I'm just a bit concerned to re-discuss something that has been already discussed a lot in #1197 and in this PR.

[turtleman]

The discussion seems to have stalled to the naming of the local id attribute the last time. Multiple commentators are saying that lid is not a good one. I'll throw one suggestion to the mix and propose that what if the name is changed from lid to local-id and that's it? This proposal would be good to go then?

[sandstrom]

Are there any news on this?

Local IDs will help solve some concrete problems around duplicate data with nested records. For example, this is solved in Ember Data under JSON API ([see thread[(https://github.com/emberjs/data/pull/4441#issuecomment-524501809)) using a lid, even though support isn't officially added in JSON API yet.

If the spec could introduce this much-discussed feature I think many would find it useable.

[dgeb]

@sandstrom please see #1435 for a discussion of extensions as well as the newly proposed local identities extension in #1436.

I'm going to keep this PR open until we make a final decision about including local identities as an extension vs. directly in the base spec. Either way, we know they're an important addition.

[dgeb]

After exploring the concept of a local identities extension (#1436), @gabesullice and I have come back around to the original approach taken in this PR. We are planning to introduce a lid field that can be used to identify resources uniquely by type, locally within a document, when an id field is not present.

This fills a gap in the base spec in which resources can not reference themselves in relationships when ids are server-generated. For example:

 POST /people HTTP/1.1
 Content-Type: application/vnd.api+json"
 Accept: application/vnd.api+json

 {
   "data": {
     "lid": "a",
     "type": "person",
     "attributes": {
       "firstName": "John",
       "lastName": "Doe"
     },
     "relationships": {
       "bestFriend": {
         "data": {
           "lid": "a",
           "type": "person"
         }
       }
     }
   }
 }

This will also prove useful for extensions, such as atomic operations (#1437), and undoubtedly many others.

I should also note that, although the updated spec makes no allowance for lid members to be returned in responses (in which every resource should have an id), it is quite possible for extensions to be written that place a requirement on clients and servers to include a map between lids and ids if that's useful to the extension.

Please review the updated language in this PR. We'd like to merge this into the 1.1 spec and get a new draft out soon.

[DLiblik]

@dgeb I think this is a great outcome. We've been using this approach in practice for several years now in a proprietary MERGE extension we put in place in a number of systems now to handle batch (atomic) operations (slightly different syntax, but same idea: a local ID where a global one is not available).

It's worked very well and we have yet to hit a transactional situation that broke down (to be clear, I'm not suggesting any changes to use MERGE as the verb - just indicating our particular flavor of rubber bands and popsicle sticks).

For any transactions that assign a global ID to the object during the server transaction we also return both local and (assigned) global ID fields so that the client can tell which witch is which when reintegrating into client-side state.

[sandstrom]

@dgeb Looks great! 💯

[jbescoyez]

We are also using this approach for atomic operations. We even have extracted it so that it is reusable throughout our Ember app. I'm excited to see this landing in the 1.1 specs so that it can be implemented in "official" JSONAPI libs. 🎉

[gabesullice]

I think we've settled on a limited scope, well defined solution. This fills a gap in the base spec, while providing the necessary foundation for new extensions (like atomic operations in #1437). Thanks for the help, everyone!