Support External Ids in Dgraph

[akshaydeo]

Earlier dgraph had support to xid or custom uid (I have used till 0.8.3)
Now it seems that both of these are removed. This may result in a lot of problems

1. As far as I remember Dgraph does not do dedup
2. Earlier I used to use assigned Uid (which was a func() on xid), and keep on mutating the nodes without worrying about anything
3. Now it seems like I have to first query the node, get the Uid and then start operating onto it

Most of the devs would want this as the core feature of the dgraph.

Related slack conv: https://dgraph.slack.com/archives/C13LH03RR/p1518791448000040

[cyphercider]

What I have done to solve this problem is create a helper function getExtIdToUidMap that takes an array of ext ids and returns me the array of uids so that I can operate on them. Agreed that one database call is better than two, but at least a helper can encapsulate the query -> mutate process.

I think stopping support for XID and treating it like other edges allowed the Dgraph team to simplify the code base for v1.

Another interesting problem affecting external IDs is related to the Dgraph team's suggestion to have many small predicates rather than a single large predicate (like "external id"). So the best practice solution may be to have different XID predicates for different data types... like person.xid, place.xid, company.xid.

[malzzz]

Indeed, this was giving us headaches for a bit as well. What we ended up doing was to create a central "gateway" microservice that handles database writes, with an internal cache of uid mappings, as well as a Redis layer behind that.

[akshaydeo]

Yeah @antikantian this was the first thing I had thought of. Using the inbuilt badger for maintaining a map of xid to uid. But this feature seems like a candidate for a product. This feature was already there which was removed.

[malzzz]

@akshaydeo, I totally agree. External id support (or, I guess re-support in this case) would make things a lot easier.

[hoonmin]

+1 for this. I'm actually using dgraph with Redis for UID mapping. XID-to-UID map is a fundamental function for normal applications. I think one of the best dgraph feature is its simplicity. But when it comes to UID map, we need another big storage like Redis which might make people to look for alternatives.

[akshaydeo]

@pawanrawal @manishrjain can anyone from dgraph help us on this?

[akshaydeo]

I have an update. Seeing the least enthusiasm from the maintainers of the project 😞, I started digging into the source code.

• One of the main reason, of removing this support seems like to avoid the collision of the UIDs
• In the current setup, the current leader does the UID allocation.
• The lease is shared with all the nodes for faster verifications etc.

Hacky approach

• There is a function called VerifyUid which does the verification of the UID passed with the quad against the available lease.
  Refer: https://sourcegraph.com/github.com/dgraph-io/dgraph/-/blob/query/mutation.go#L96:6
• I just returned nil via the function.

Pros

• Passing all the test cases of the project 🍾
• Passing all the test cases I have in my project 👍
• If you don't pass your own UID entire flow works the same.

Cons 🚑

• This approach won't handle collisions of the UID. As far as my tests go, it just overwrites which was desired behavior for my app
• I have tried my best to check all the flows that might break, but could not find any, but it's still a hacky approach.

@pawanrawal @manishrjain Could you guys let us know if there could be any issues you could think of in this approach, until this is officially supported by dgraph?

[jimanvlad]

Glad someone is prepared to get their hands dirty :) Could you please expand a bit more on “This approach won't handle collisions of the UID. As far as my tests go, it just overwrites which was desired behavior for my app”? Are you saying that this is not functionality to deal with the ‘if node exists’ vs. ‘If node doesn’t exist’ case, but simply ensured that there can’t ever be 2 nodes with the same xid?

On Fri, 23 Mar 2018 at 12:08, Akshay Deo ***@***.***> wrote: I have an update. Seeing the least enthusiasm from the maintainers of the project 😞, I started digging into the source code. - One of the main reason, of removing this support seems like to avoid the collision of the UIDs - In the current setup, the current leader does the UID allocation. - The lease is shared with all the nodes for faster verifications etc. Hacky approach - There is a function called VerifyUid which does the verification of the UID passed with the quad against the available lease. Refer: https://sourcegraph.com/github.com/dgraph-io/dgraph/-/blob/query/mutation.go#L96:6 - I just returned nil via the function. Pros - Passing all the test cases of the project 🍾 - Passing all the test cases I have in my project 👍 - If you don't pass your own UID entire flow works the same. Cons 🚑 - This approach won't handle collisions of the UID. As far as my tests go, it just overwrites which was desired behavior for my app - I have tried my best to check all the flows that might break, but could not find any, but it's still a hacky approach. @pawanrawal <https://github.com/pawanrawal> @manishrjain <https://github.com/manishrjain> Could you guys let us know if there could — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACKSZl8IhJO_ru2FfWWjvoK1Pkjjqwnxks5thOW2gaJpZM4SIfoF> .

-- Thanks, Vlad

[akshaydeo]

Yes. So here is the example:

1. I perform mutate by following NQuads

{Subject: "0x1", Predicate: "Name", Object: "Jarvis"}
{Subject: "0x1", Predicate: "Color", Object: "White"}

Premise is Node with UID 0x1 is not in the system

And fetch the record by UID 0x1 would return this node with predicates Name and Color with Values Jarvis And White

2. If I perform one more mutation after this as follows

{Subject: "0x1", Predicate: "Name", Object: "Jarvis2"}

And fetch the record by UID 0x1 would return this node with predicates Name and Color with Values Jarvis2 And White

[jimanvlad]

But you can already perform mutations that write to a UID and ensure that there’s no duplication. Can the UID in your example be replaced with an XID? For example, can it be something like “account_193748”?

On Fri, 23 Mar 2018 at 12:39, Akshay Deo ***@***.***> wrote: Yes. So here is the example: 1. I perform mutate by following NQuads {Subject: "0x1", Predicate: "Name", Object: "Jarvis"} {Subject: "0x1", Predicate: "Color", Object: "White"} Premise is Node with UID 0x1 is not in the system And fetch the record by UID 0x1 would return this node with predicates Name and Color with Values Jarvis And White 1. If I perform one more mutation after this as follows {Subject: "0x1", Predicate: "Name", Object: "Jarvis2"} And fetch the record by UID 0x1 would return this node with predicates Name and Color with Values *Jarvis2* And White — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2134 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACKSZjl1pw2AUy8qnn6ApvSeCdQdSZ8Aks5thOzkgaJpZM4SIfoF> .

-- Thanks, Vlad

[akshaydeo]

Yes yes, but with a twist.
So In dgraph, uid is the ultimate identifier, which has to be a hex integer.

But there is a workaround which I have been using since I have started using dgraph (0.7.x)

1. Have a functional mapping of uid and xid. I do something like

func uidFromxid(xid string) uint64 {
	f := fnv.New64a()
	f.Write([]byte(xid))
	return f.Sum64()
}

And pass this as uid in the above example.

This way the mapping becomes programmatical

[jimanvlad]

Ah, so all you're doing is disabling the verification of the UID. Your function to translate an XID into a UID will lead to the data being stored with UIDs that are not sequential, but all over the place. So then, as your question stated above, you need to understand the unintended consequences of this.

[liqweed]

At least superficially, it sounds like it would fit our scenario has well. Could that validation be made optional?

[akshaydeo]

I can quickly send a pull request, which can skip the verification based on a param sent with dgraph server?

Would that be a right way to introduce that?